zbMATH — the first resource for mathematics

Consensus theory for mixed response formats. (English) Zbl 1416.62646
Summary: Measuring shared beliefs, expert consensus, or the details of a crime in eyewitness testimony represents a psychometric challenge. In expert interviews, for example, the correct responses representing the expert consensus (i.e., the answer key) are initially unknown and experts may differ in their contribution to this consensus. I propose the variable-response model, an extension of latent-trait models. The model allows the estimation of the answer key and the latent trait for continuous, categorical, or mixed responses. I describe some minimal requirements for the addition of new response formats to the model. I further propose a Markov chain Monte Carlo algorithm to estimate the model parameters. The results of a simulation study demonstrate that the algorithm accurately recovers the data-generating parameters. I also present an application of the variable-response model to the empirical data of a geography test. In this application, the parameter estimates correspond well with the true answer key.

62P15 Applications of statistics to psychology
62D05 Sampling theory, sample surveys
Full Text: DOI
[1] Akaike, H., Information theory and an extension of the maximum likelihood principle, (Petrov, B. N.; Csaki, F., Proceeding of the second international symposium on information theory, (1973), Akademiai Kiado Budapest), 267-281 · Zbl 0283.62006
[2] Anders, R.; Batchelder, W. H., Cultural consensus theory for multiple consensus truths, Journal of Mathematical Psychology, 56, 452-469, (2012)
[3] Anders, R.; Batchelder, W. H., Cultural consensus theory for the ordinal data case, Psychometrika, 80, 151-181, (2015) · Zbl 1314.62267
[4] Anders, R.; Oravecz, Z.; Batchelder, W. H., Cultural consensus theory for continuous responses: A latent appraisal model for information pooling, Journal of Mathematical Psychology, 61, 1-13, (2014) · Zbl 1309.91108
[5] Aßfalg, A.; Erdfelder, E., CAML—maximum likelihood consensus analysis, Behavior Research Methods, 44, 189-201, (2012)
[6] Barton, M. A.; Lord, F. M., An upper asymptote for the three-parameter logistic item-response model, ETS Research Report Series, 1981, 1-8, (1981)
[7] Batchelder, W. H.; Anders, R., Cultural consensus theory: comparing different concepts of cultural truth, Journal of Mathematical Psychology, 56, 316-332, (2012) · Zbl 1282.91293
[8] Batchelder, W. H.; Romney, A. K., Test theory without an answer key, Psychometrika, 53, 71-92, (1988) · Zbl 0718.62260
[9] Batchelder, W. H.; Strashny, A.; Romney, A. K., Cultural consensus theory: aggregating continuous responses in a finite interval, (Chai, S. K.; Salerno, J. J.; Mabry, P. L., Advances in social computing, SBP 2010, Lecture notes in computer science, (2010), Springer Heidelberg, Germany), 98-107
[10] Birnbaum, A., Some latent train models and their use in inferring an examinee’s ability, (Lord, F. M.; Novick, & M. R., Statistical theories of mental test scores, (1968), Addison-Wesley Reading, MA), 395-479
[11] Comrey, A. L., The minimum residual method of factor analysis, Psychological Reports, 11, 15-18, (1962)
[12] Fox, J. P., Bayesian item response modeling: theory and applications, (2010), Springer Science & Business Media New York, NY · Zbl 1271.62012
[13] Gelman, A.; Hwang, J.; Vehtari, A., Understanding predictive information criteria for Bayesian models, Statistics and Computing, 24, 997-1016, (2014) · Zbl 1332.62090
[14] Gelman, A.; Roberts, G. O.; Gilks, W. R., Efficient metropolis jumping rules, Bayesian Statistics, 5, 599-607, (1996)
[15] Hastings, W. K., Monte Carlo sampling methods using Markov chains and their applications, Biometrika, 57, 97-109, (1970) · Zbl 0219.65008
[16] Karabatsos, G.; Batchelder, W. H., Markov chain estimation for test theory without an answer key, Psychometrika, 68, 373-389, (2003) · Zbl 1306.62447
[17] Metropolis, N.; Rosenbluth, A. W.; Rosenbluth, M. N.; Teller, A. H.; Teller, E., Equation of state calculations by fast computing machines, The Journal of Chemical Physics, 21, 1087-1092, (1953)
[18] Moustaki, I.; Knott, M., Generalized latent trait models, Psychometrika, 65, 391-411, (2000) · Zbl 1291.62236
[19] Moustaki, I.; Papageorgiou, I., Latent class models for mixed variables with applications in archaeometry, Computational Statistics & Data Analysis, 48, 659-675, (2005) · Zbl 1430.62254
[20] Patz, R. J.; Junker, B. W., A straightforward approach to Markov chain Monte Carlo methods for item response models, Journal of Educational and Behavioral Statistics, 24, 146-178, (1999)
[21] R Core Team, R: A language and environment for statistical computing, (2018), R Foundation for Statistical Computing Vienna, Austria
[22] Rasch, G., On general laws and the meaning of measurement in psychology, (Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, vol. 4, (1961), University of California Press Berkeley, CA), 321-333 · Zbl 0107.36805
[23] Reckase, M. D., A linear logistic multidimensional model for dichotomous item response data, (Van Der Linden, W. J.; Hambleton, R. K., Handbook of modern item response theory, (1997), Springer New York), 271-286
[24] Romney, A. K.; Weller, S. C.; Batchelder, W. H., Culture as consensus: A theory of culture and informant accuracy, American Anthropologist, 88, 313-338, (1986)
[25] Sammel, M. D.; Ryan, L. M.; Legler, J. M., Latent variable models for mixed discrete and continuous outcomes, Journal of the Royal Statistical Society. Series B. Statistical Methodology, 59, 667-678, (1997) · Zbl 0889.62043
[26] Schwarz, G., Estimating the dimension of a model, The Annals of Statistics, 6, 461-464, (1978) · Zbl 0379.62005
[27] Sheng, Y., An empirical investigation of Bayesian hierarchical modeling with unidimensional IRT models, Behaviormetrika, 40, 19-40, (2013)
[28] Spiegelhalter, D. J.; Best, N. G.; Carlin, B. P.; Van Der Linde, A., Bayesian measures of model complexity and fit, Journal of the Royal Statistical Society. Series B. Statistical Methodology, 64, 583-639, (2002) · Zbl 1067.62010
[29] Watanabe, S., Equations of states in singular statistical estimation, Neural Networks, 23, 20-34, (2010) · Zbl 1396.68106
[30] Waubert de Puiseau, B.; Aßfalg, A.; Erdfelder, E.; Bernstein, D. M., Extracting the truth from conflicting eyewitness reports: A formal modeling approach, Journal of Experimental Psychology: Applied, 18, 390-403, (2012)
[31] Waubert de Puiseau, B.; Greving, S.; Aßfalg, A.; Musch, J., On the importance of considering heterogeneity in witnesses’ competence levels when reconstructing crimes from multiple witness testimonies, Psychological Research, 81, 947-960, (2017)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.