×

Automation of broad sanity test generation. (English. Russian original) Zbl 1191.68199

Program. Comput. Softw. 34, No. 6, 351-363 (2008); translation from Programmirovanie 34, No. 6 (2008).
Summary: The technology for the broad generation of sanity tests for complex software developed in the Institute for System Programming (Russian Academy of Sciences) is presented. This technology is called Azov; it is based on using a database containing structured information about the interface operations of the system under test and on a procedure for enriching this information by refining constraints imposed on parameter types and results of operations. Results of a practical application of this technology prove its high efficiency in generating sanity tests for systems with a large number of functions.

MSC:

68N99 Theory of software
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] IEEE 1003.1-2004. Information Technology–Portable Operating System Interface (POSIX), New York: IEEE, 2004.
[2] http://doc.trolltech.com/4.2/index.html .
[3] http://www.linuxbase.org .
[4] ISO/IEC 9899-1999. Programming Languages-C, Geneva: ISO, 1999.
[5] http://www.pathname.com/fhs/ .
[6] XLib–C Language X Interface. X Consortium Standard, http://refspecs.freestandards.org/X11/xlib.pdf .
[7] http://www.opengl.org .
[8] http://www.gtk.org .
[9] http://doc.trolltech.com/3.3/index.html .
[10] http://www.linux-foundation.org/navigator/commons/welcome.php .
[11] Colbourn, C.J., Combinatorial Aspects of Covering Arrays, Le Mathemetiche (Catania), 2004, vol. 58, pp. 121–167. · Zbl 1195.05017
[12] Hartman, A. and Raskin, L., Problems and Algorithms for Covering Arrays, Discrete Math., 2004, vol. 284, nos. 1–3, pp. 149–156. · Zbl 1044.05029 · doi:10.1016/j.disc.2003.11.029
[13] http://www.pairwise.org/tools.asp .
[14] http://aetgweb.argreenhous.com/ .
[15] Cohen, D.M., Dalal, S.R., Kajla, A., and Patton, G.C., The Automatic Efficient Test Generator (AETG), System. Proc. of the 5th Int. Symposium on Software Reliability Engineering (ISSRE), Monterey, Calif., 1994.
[16] http://www.testcover.com/ .
[17] Sherwood, G., Effective Testing of Factor Combinations, Proc. of the 3rd Int. Conf. on Software Testing, Analysis, and Review, Washington, 1994.
[18] http://www.satisfice.com/tools.shtml .
[19] http://burtleburtle.net/bob/math/jenny.html .
[20] http://alphaworks.ibm.com/tech/whitch .
[21] http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi .
[22] http://www.turbodata.ca/ .
[23] http://www.sqlmanager.net/products .
[24] http://www.sqledit.com/dg/ .
[25] http://www.datatect.com/ .
[26] http://www.forsql.com/ .
[27] Chays, D., Deng, Y., Frankl, P.G., Dan, S., Vokolos, F.I., and Weyuker, E.J., AGENDA: A Test Generator for Relational Database Applications, Tech. Report of the Polytechnic University, Brooklin, New York TR-CIS-2002-04, 2004.
[28] Binning, C., Kossmann, D., and Lo, E., Testing Database Applications, Proc. of the ACM SIGMOD Int. Conference, Chicago, ACM, 2006, pp. 739–741.
[29] Bruno, N. and Chaudhuri, S., Flexible Database Generators, Proc. of the 31st Int. Conf. on very Large Databases, Trondheim, Norway, 2005, pp. 1097–1107.
[30] Barbosa, D. and Mendelzon, A., Declarative Generation of Synthetic XML Data, Software: Practice & Experience, 2006, vol. 36, no. 10, pp. 1051–1079. · Zbl 05133879 · doi:10.1002/spe.724
[31] Lämmel, R. and Schulte, W., Controllable Combinatorial Coverage in Grammar-Based Testing, Proc. of TESTCOM’2006, Lect. Notes Comput. Sci., 2006, vol. 3964, pp. 19–38. · Zbl 1185.68256 · doi:10.1007/11754008_2
[32] http://www.alphaworks.imb.com/tech/xmlgenerator .
[33] http://xml-xig.sourceforge.net/ .
[34] http://iwm.uni-koblenz.de/datagen/ .
[35] Purdom, P., A Sentence Generator for Testing Parsers, BIT, 1972, vol. 12, no. 3, pp. 366–375. · Zbl 0242.68015 · doi:10.1007/BF01932308
[36] Celentano, A., Crespi Rghezzi, S., Della Vigna, P., Ghezzi, C., Granata, G., and Savoretti, F., Compiler Testing Using a Sentence Generator, Software: Practice & Experience, 1980, vol. 10, pp. 897–918. · doi:10.1002/spe.4380101104
[37] Maurer, P., Generating Test Data with Enhanced Context-Free Grammars, IEEE Software, 1990, vol. 7, no. 4, pp. 50–56. · Zbl 05101959 · doi:10.1109/52.56422
[38] Zelenov, S.V. and Zelenova, S.A., Generation of Positive and Negative Tests for Parsers, Programmirovanie, 2005, vol. 31, no. 6, pp. 25–40 [Programming Comput. Software (Engl. Transl.), 2005, vol. 31, pp. 310–320]. · Zbl 1115.68376
[39] http://www..mmsindia.com/JSynTest.html .
[40] Zelenov, S.V., Zelenova, S.A., Kossatchev, A.S., and Petrenko, A.K., Test Generation for Compilers and Other Formal Text Processors, Programmirovanie, 2003, vol. 29, no. 2, pp. 59–69 [Programming Comput. Software (Engl. Transl.), 2003, vol. 29, no. 2, pp. 104–111]. · Zbl 1063.68042
[41] Demakov, A.V., Zelenov, S.V., and Zelenova, S.A., Generation of Complex Structure Test Data with Account of Context Constraints, Trudy ISP RAN, 2006, vol. 9, pp. 83–96. · Zbl 1191.68206
[42] Gotlieb, A., Botella, B., and Rueher, M., Automatic Test Data Generation Using Constraint Solving Techniques, ACM SIGSOFT Software Eng. Notes, 1998, vol. 23, no. 2, pp. 53–62. · doi:10.1145/271775.271790
[43] DeMillo, R.A. and Offut, A.J., Constraint-Based Automatic Test Data Generation, IEEE Trans. Software Eng., 1991, vol. 17, no. 9, pp. 900–910. · Zbl 05114378 · doi:10.1109/32.92910
[44] Korel, B., Automated Software Test Data Generation, IEEE Trans. Software Eng., 1990, vol. 16, no. 8, pp. 870–879. · Zbl 05114234 · doi:10.1109/32.57624
[45] Gupta, N., Mathur, A.P., and Soffa, M.L., Automated Test Data Generation Using an Iterative Relaxation Method, ACM SIGSOFT Software Eng. Notes, 1998, vol. 23, no. 6, pp. 231–244. · doi:10.1145/291252.288321
[46] http://www.parasoft.com/jsp/products.jsp .
[47] http://www.t-vec.com/solutions/tvec.php .
[48] http://www.suresofttech.com/eng/main/product/api.asp .
[49] Marinov, D. and Khurshid, S., TestEra: A Novel Framework for Automated Testing of Java Programs, Proc. of the 16th IEEE Int. Conf. on Automated Software Engineering, 2001, pp. 22–31.
[50] Boyapati, C., Khurshid, S., and Marinov, D., Korat: Automated Testing Based on Java Predicates, Proc. of Int. Symposium on Software Testing and Analysis, Rome, 2002, pp. 123–133.
[51] Korel, B., A Dynamic Approach of Automated Test Data Generation, Proc. of Conf. on Software Maintenance, San Diego, 1990, pp. 311–317.
[52] Pargas, R.P., Harrold, M.J., and Peck, R., Test-Data Generation Using Genetic Algorithms, Software Testing, Verification &amp; Reliability, 1999, vol. 9, no. 4, pp. 263–282. · doi:10.1002/(SICI)1099-1689(199912)9:4<263::AID-STVR190>3.0.CO;2-Y
[53] Seesing, A. and Gross, H.-G. A Genetic Programming Approach to Automated Test Generation for Object-Oriented Software, Int. Trans. Syst. Sci. Appl., 2006, vol. 1, no. 2, pp. 127–134.
[54] Korel, B., Automated Test Data Generation for Programs with Procedures, Proc. of ISSSTA, 1996, pp. 209–215.
[55] Ferguson, R. and Korel, B., The Chaining Approach for Software Test Data Generation, ACM Trans. Software Eng. Methodology, 1996, vol. 5, no. 1, pp. 63–86. · Zbl 01936327 · doi:10.1145/226155.226158
[56] Csallner, C. and Smaragdakis, Y., JCrasher and Automatic Robustness Tester for Java, Software: Practice &amp; Experience, 2004, vol. 34, no. 11, pp. 1025–1050. · Zbl 02184345 · doi:10.1002/spe.602
[57] Csallner, C. and Smaragdakis, Y., Check ’n’ Crash: Combining Static Checking and Testing, Proc. of the 27th Int. Conf. on Software Engineering (ICSE), ACM, 2005, pp. 422–431.
[58] Csallner, C. and Smaragdakis, Y., DSD-Crasher: A Hybrid Analysis Tool fo Bug Finding, Proc. of the ACM SIGSOFT Int. Symposium on Software Testing and Analysis (ISSTA), ACM, 2006, pp. 245–254.
[59] Smaragdakis, Y. and Csallner, C., Combining Static and Dynamic Reasoning for Bug Detection, Proc. of TAP2007, Lect. Notes. Comput. Sci., 2007, vol. 4454, pp. 1–16. · Zbl 05207429 · doi:10.1007/978-3-540-73770-4_1
[60] Flanagan, C., Leino, K.R.M., Lillibridge, M., Nelson, G., Saxe, J.B., and Stata, R., Exrended Static Checking for Java, Proc. of the ACM SIGPLAN Conf. on Programming Language Design and Implementation, 2002, pp. 193–205.
[61] Ernst, M.D., Cockrell, J., Griswold, W.G., and Notkin, D., Dynamically Discovering Likely Program Invariants to Support Program Evolution, IEEE Trans. Software Eng., 2001, vol. 27, no. 2, pp. 99–123. · doi:10.1109/32.908957
[62] Meyer, B., Ciupa, I., Leitner, A., and Liu, L., Automatic Testing of Object-Oriented Software, Proc. of the 33rd. Conf. on Current Trends in the Theory and Practice of Computer Science (SOFSEM), Springer, 2007.
[63] Pacheco, C. and Ernst, M.D., Eclat: Automatic Generation and Classification of Test Inputs, Proc. of ECOOP, 2005, pp. 504–527.
[64] Pacheco, C., Lahiri, S.K., Ernst, M.D., and Ball, T., Feedback-Directed Random Test Generation, Proc. of ICSE, 2007, pp. 75–84.
[65] Xie, T., Marinov, D., and Notkin, D., Rostra: A Framework for Detecting Redundant Object-Oriented Unit Tests, Proc. of the 19th IEEE Int. Conf. on Automated Software Engineering (ASE 2004), Linz, Austria, 2004, pp. 196–205.
[66] Xie, T., Marinov, D., Schulte, W., and Notkin, D., Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution, Proc. of the 11th Int. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2005), Edinburgh, 2005, pp. 365–381. · Zbl 1087.68601
[67] Artho, C., Barringer, H., Goldberg, A., Havelund, K., Khurshid, S., Lowry, M., Pasareanu, C., Rosu, G., Sen, K., Visser, W., and Washington, R., Combining Test Case Generation and Runtime Verification, Theor. Comput. Sci., 2005, vol. 366, nos. 2–3, pp. 209–234. · Zbl 1080.68062 · doi:10.1016/j.tcs.2004.11.007
[68] Sen, K. and Agha, G., CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools, Proc. of Computer Aided Verification, 2006, pp. 419–423.
[69] Godefroid, P., Klarlund, N., and Sen, K., DART: Directed Automated Random Testing, Proc. of the ACM SIGPLAN Conf. on Programming Language Design and Implementation, Chicago, 2005, pp. 213–223.
[70] Tillmann, N. and Schulte, W., Parameterized Unit Tests with Unit Meister, ACM SIGSOFT Software Eng. Notes, 2005, vol. 30, no. 5, pp. 241–244. · doi:10.1145/1095430.1081745
[71] Yorsh, G., Ball, T., and Sagiv, M., Testing, Abstraction, Theorem Proving: Better Together, Proc. of the Int. Symposium on Software Testing and Analysis, Portland, Maine, 2006, ACM, 2006, pp. 145–156.
[72] Brucker, A.D. and Wolf, B., Interactive Testing with HOL-TestGen, Proc. of FATES, 2006, Lect. Notes Comput. Sci., 2006, vol. 3997, pp. 87–102.
[73] Harman, A., Model-Based Test Generation Tools, 2002, http://www.agedis.de/documents/Model-BasedTestGene rationTools.pdf .
[74] Model-Based Testing of Reactive Systems: Advanced Lectures, Broy, M., Jonsson, B., Katoen, J.-P., Leucker, M., and Pretschner, A., Eds., Lect. Notes Comput. Sci., 2005, vol. 3472.
[75] Kuliamin, V.V., Petrenko, A.K., Kossatchev, A.S., and Bourdonov, I.B., The UniTesK Approach to Designing Test Suites, Programmirovanie, 2003, no. 6, pp. 25–43 [Programming Comput. Software (Engl. Transl.), 2003, vol. 29, no. 6, pp. 310–322].
[76] Ammann, P. and Black, P.E., Abstracting Formal Specifications to Generate Software Tests via Model Checking, NIST-IR 6405 (extended version), 1999.
[77] Gargantini, A. and Heitmeyer, C., Using Model Checking to Generate Test Form Requirements Specifications, Proc. of the Joint 7th European Software Engineering Conference and the 7th ACM SIGSOFT Int. Symposium on Foundations of Software Engineering (ESEC/FSE99), Toulouse, 1999.
[78] Devaraj, G., Heimdahl, M.P.E., and Liang, D., Coverage-Directed Test Generation with Model Checkers: Challenges and Opportunities, Proc. of the 29th Annual Int. Computer Software and Applications Conference (COMPSAC’05), 2005, vol. 1, pp. 455–462.
[79] Beyer, D., Henzinger, T.A., Jhala, R., and Majumdar, R., The Software Model Checker Blast: Applications to Software Engineering, Int. J. Software Tools Technol. Transfer, 2007, vol. 9, nos. 5–6, pp. 505–525. · Zbl 05536156 · doi:10.1007/s10009-007-0044-z
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.