A rate function approach to computerized adaptive testing for cognitive diagnosis. (English) Zbl 1322.62335

Liu, Jingchen; Ying, Zhiliang; Zhang, Stephanie

Psychometrika 80, No. 2, 468-490 (2015).

Summary: Computerized adaptive testing (CAT) is a sequential experiment design scheme that tailors the selection of experiments to each subject. Such a scheme measures subjects’ attributes (unknown parameters) more accurately than the regular prefixed design. In this paper, we consider CAT for diagnostic classification models, for which attribute estimation corresponds to a classification problem. After a review of existing methods, we propose an alternative criterion based on the asymptotic decay rate of the misclassification probabilities. The new criterion is then developed into new CAT algorithms, which are shown to achieve the asymptotically optimal misclassification rate. Simulation studies are conducted to compare the new approach with existing methods, demonstrating its effectiveness, even for moderate length tests.

MSC:

62P15

Applications of statistics to psychology

Keywords:

computerized adaptive testing; cognitive diagnosis; large deviation; classification

References:

[1]	Chang, H.-H.; Ying, Z., A global information approach to computerized adaptive testing, Applied Psychological Measurement, 20, 213-229, (1996)
[2]	Cheng, Y., When cognitive diagnosis meets computerized adaptive testing CD-CAT, Psychometrika, 74, 619-632, (2009) · Zbl 1272.62116
[3]	Chiu, C.; Douglas, J.; Li, X., Cluster analysis for cognitive diagnosis: theory and applications, Psychometrika, 74, 633-665, (2009) · Zbl 1179.62087
[4]	Cox, D., & Hinkley, D. (2000). Theoretical statistics. London: Chapman & Hall. · Zbl 0334.62003
[5]	Torre, J.; Douglas, J., Higher order latent trait models for cognitive diagnosis, Psychometrika, 69, 333-353, (2004) · Zbl 1306.62527
[6]	DiBello, L.V.; Stout, W.F.; Roussos, L.A.; Nichols, P.D. (ed.); Chipman, S.F. (ed.); Brennan, R.L. (ed.), Unified cognitive psychometric diagnostic assessment likelihood-based classification techniques, 361-390, (1995), Hillsdale
[7]	Edelsbrunner, H.; Grayson, D.R., Edgewise subdivision of a simplex, Discrete & Computational Geometry, 24, 707-719, (2000) · Zbl 0968.51016
[8]	Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: blending theory with practicality. Unpublished doctoral dissertation, University of Illinois, Urbana-Champaign.
[9]	Junker, B.; Lissitz, R.W. (ed.), Using on-line tutoring records to predict end-of-year exam scores: experience with the assistments project and MCAS 8th grade mathematics, (2007), Maple Grove
[10]	Junker, B.; Sijtsma, K., Cognitive assessment models with few assumptions, and connections with nonparametric item response theory, Applied Psychological Measurement, 25, 258-272, (2001)
[11]	Leighton, J.P.; Gierl, M.J.; Hunka, S.M., The attribute hierarchy model for cognitive assessment: a variation on tatsuoka’s rule-space approach, Journal of Educational Measurement, 41, 205-237, (2004)
[12]	Lord, F.M., Robbins-monro procedures for tailored testing, Educational and Psychological Measurement, 31, 3-31, (1971)
[13]	Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale: Erlbaum.
[14]	Owen, R.J., Bayesian sequential procedure for quantal response in context of adaptive mental testing, Journal of the American Statistical Association, 70, 351-356, (1975) · Zbl 0324.62061
[15]	Rupp, A.A., Templin, J., & Henson, R.A. (2010). Diagnostic measurement: theory, methods, and applications. New York: Guilford Press.
[16]	Serfling, R.J. (1980). Approximation theorems of mathematical statistics. New York: Wiley-Interscience (W. Shewhart & S. Wilks (Eds.)). · Zbl 0538.62002
[17]	Tatsuoka, K.K., Rule space: an approach for dealing with misconceptions based on item response theory, Journal of Educational Measurement, 20, 345-354, (1983)
[18]	Tatsuoka, K.K., A probabilistic model for diagnosing misconceptions in the pattern classification approach, Journal of Educational Statistics, 12, 55-73, (1985)
[19]	Tatsuoka, K. (1991). Boolean algebra applied to determination of the universal set of misconception states (ONR-Technical Report No. RR-91-44). Princeton: Educational Testing Services.
[20]	Tatsuoka, C. (1996). Sequential classification on partially ordered sets. Doctoral dissertation, Cornell University.
[21]	Tatsuoka, C., Data-analytic methods for latent partially ordered classification models, Applied Statistics, 51, 337-350, (2002) · Zbl 1111.62381
[22]	Tatsuoka, K.K. (2009). Cognitive assessment: an introduction to the rule space method. New York: Routledge.
[23]	Tatsuoka, C.; Ferguson, T., Sequential classification on partially ordered sets, Journal of the Royal Statistical Society, Series B, Statistical Methodology, 65, 143-157, (2003) · Zbl 1063.62113
[24]	Templin, J.; Henson, R.A., Measurement of psychological disorders using cognitive diagnosis models, Psychological Methods, 11, 287-305, (2006)
[25]	Templin, J., He, X., Roussos, L.A., & Stout, W.F. (2003). The pseudo-item method: a simple technique for analysis of polytomous data with the fusion model (External Diagnostic Research Group Technical Report).
[26]	Thissen, D.; Mislevy, R.J.; Wainer, H. (ed.); etal., Testing algorithms, 101-133, (2000), Mahwah
[27]	Linden, W.J., Bayesian item selection criteria for adaptive testing, Psychometrika, 63, 201-216, (1998) · Zbl 1008.62709
[28]	von Davier, M. (2005). A general diagnosis model applied to language testing data (Research report). Princeton: Educational Testing Service.
[29]	Xu, X., Chang, H.-H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of the American Educational Research Association, Chicago, April 2003.