Empirical Bayes and item-clustering effects in a latent variable hierarchical model: A case study from the National Assessment of Educational Progress. (English) Zbl 1073.62512

Summary: Empirical Bayes regression procedures are often used in educational and psychological testing as extensions to latent variables models. The National Assessment of Educational Progress (NAEP) is an important national survey using such procedures. The NAEP applies empirical Bayes methods to models from item response theory to calibrate student responses to questions of varying difficulty. Due partially to the limited computing technology that existed when NAEP was first conceived, NAEP analyses are carried out using a two-stage estimation procedure that ignores uncertainty about some model parameters. Furthermore, the item response theory model that NAEP uses ignores the effect of item clustering created by the design of a test form. Using Markov chain Monte Carlo, we simultaneously estimate all parameters of an expanded model that considers item clustering to investigate the impact of item clustering and ignoring uncertainty about model parameters on an important outcome measure that NAEP report. Ignoring these two effects causes substantial underestimation of standard errors and induces a modest bias in location estimates.


62C12 Empirical decision procedures; empirical Bayes procedures
62P25 Applications of statistics to social sciences
65C40 Numerical analysis or methods applied to Markov chains
Full Text: DOI