swMATH ID: 28473
Software Authors: S. Lu, Z. Li, F. Qin, L. Tan, P. Zhou, Y. Zhou
Description: Bugbench: Benchmarks for Evaluating Bug Detection Tools. Benchmarking provides an effective way to evaluate different tools. Unfortunately, so far there is no good benchmark suite to systematically evaluate software bug detection tools. As a result, it is difficult to quantitatively compare the strengths and limitations of existing or newly proposed bug detection tools. In this paper, we share our experience of building a bug benchmark suite called BugBench. Specifically, we first summarize the general guidelines on the criteria for selecting representative bug benchmarks, and the metrics for evaluating a bug detection tool. Second, we present a set of buggy applications collected by us, with various types of software bugs. Third, we conduct a preliminary study on the application and bug characteristics in the context of software bug detection. Finally, we evaluate several existing bug detection tools including Purify, Valgrind, and CCured to validate the selection of our benchmarks.
Homepage: http://pages.cs.wisc.edu/~shanlu/paper/63-lu.pdf
Source Code:  https://github.com/lihebi/bugbench
Related Software: JUnit; GitHub; Travis CI; Maven; Defects4J; Bears; Codeflaws; GHTorrent; JGit; jtravis; Bugs.jar; ASTOR; Nopol; QuixBugs; jGenProg; Docker; nose; Error Prone; TestNG; Gradle
Cited in: 1 Document

Cited in 0 Serials

Citations by Year