×

Students’ complex problem solving profiles. (English) Zbl 1458.62268

Summary: Complex problem solving (CPS) is an up-and-coming twenty-first century skill that requires test-takers to solve dynamically changing problems, often assessed using computer-based tests. The log data that users produce when interacting with a computer-based test provide valuable information about each individual behavioral action they undertake, but such data are rather difficult to handle from a statistical point of view. This paper addresses this issue by building upon recent research focused on decoding log data and aims to identify homogeneous student profiles with regard to their ability to solve CPS tasks. Therefore, we estimated a discrete two-tier item response theory model, which allowed us to profile units (i.e., students) while taking into account the multidimensionality of the data and the explanatory effect of individual characteristics. The results indicate that: (1) CPS can be thought of as a three-dimensional latent variable; (2) there are ten latent classes of students with homogenous profiles regarding the CPS dimensions; (3) students in the higher latent classes generally demonstrate higher cognitive and non-cognitive performances; (4) some of the latent classes seem to profit from learning-by-doing within tasks, whereas others seem to exhibit the reverse behavior; (5) cognitive and non-cognitive skills, as well as gender and to some extent age, contribute to distinguishing among the latent classes.

MSC:

62P15 Applications of statistics to psychology
62D05 Sampling theory, sample surveys
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Adams, RJ; Wilson, MR; Wang, WC, The multidimensional random coefficients multinomial logit, Applied Psychological Measurement, 21, 1-24 (1997)
[2] Akaike, H.; Petrov, BN; Csaki, F., Information theory and an extension of the maximum likelihood principle, Second international symposium of information theory, 267-281 (1973), Budapest: Akademiai Kiado, Budapest · Zbl 0283.62006
[3] Bacci, S.; Bartolucci, F., Two-tier latent class IRT models in R, The R Journal, 8, 139-166 (2016)
[4] Bacci, S.; Bartolucci, F.; Gnaldi, M., A class of multidimensional latent class IRT models for ordinal polytomous item responses, Communications in Statistics-Theory and Methods, 43, 787-800 (2014) · Zbl 1462.62400
[5] Bartolucci, F., A class of multidimensional IRT models for testing unidimensionality and clustering items, Psychometrika, 72, 141-157 (2007) · Zbl 1286.62099
[6] Bartolucci, F., & Bacci, S. (2016). MLCIRTwithin: Latent class item response theory (LC-IRT) models under within-item multidimensionality. R package version 2.10. https://cran.r-project.org/web/packages/MLCIRTwithin. Cited July 27th 2018.
[7] Bartolucci, F.; Bacci, S.; Gnaldi, M., Statistical analysis of questionnaires: A unified approach based on R and stata (2015), Boca Raton, FL: Chapman & Hall, CRC Press, Boca Raton, FL
[8] Birnbaum, A.; Lord, FM; Novick, MR, Some latent trait models and their use in inferring an examinee’s ability, Statistical theories of mental test scores, 395-479 (1968), Reading, MA: Addison-Wesley, Reading, MA · Zbl 0186.53701
[9] Bock, RD; Gibbons, R.; Muraki, E., Full-information item factor analysis, Applied Psychological Measurement, 12, 261-280 (1988)
[10] Bolt, DM; Cohen, AS; Wollack, JA, Item parameter estimation under conditions of test speededness: Application of a mixture Rasch model with ordinal constraints, Journal of Educational Measurement, 39, 331-348 (2002)
[11] Bonifay, WE; Reise, SP; Revicki, DA, An illustration of the two-tier item factor analysis model, Handbook of item response theory modeling, 207-225 (2015), Abingdon-on-Thames: Routledge, Abingdon-on-Thames
[12] Bradlow, ET; Wainer, H.; Wang, X., A Bayesian random effects model for testlets, Psychometrika, 64, 2, 153-168 (1999) · Zbl 1365.62451
[13] Buchner, A.; Frensch, PA; Funke, J., Basic topics and approaches to the study of complex problem solving, Complex problem solving: The European perspective, 25-47 (1995), Hillsdale, NJ: Lawrence Erlbaum, Hillsdale, NJ
[14] Cai, L., A two-tier full-information item factor analysis model with applications, Psychometrika, 75, 581-612 (2010) · Zbl 1208.62183
[15] Cai, L.; Yang, JS; Hansen, M., Generalized full-information item bifactor analysis, Psychological Methods, 16, 221-248 (2011)
[16] Chalmers, RP, mirt: A multidimensional item response theory package for the R environment, Journal of Statistical Software, 48, 1-29 (2012)
[17] Choi, I.; Wilson, M., Multidimensional classification of examinees using the mixture random weights linear logistic test model, Educational and Psychological Measurement, 75, 78-101 (2015)
[18] Cohen, J., A coefficient of agreement for nominal scales, Educational and Psychological Measurement, 20, 37-46 (1960)
[19] Cohen, J., Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit, Psychological Bulletin, 70, 213-220 (1968)
[20] Danner, D.; Hagemann, D.; Holt, DV; Hager, M.; Schankin, A.; Wüstenberg, S.; Funke, J., Measuring performance in dynamic decision making. Reliability and validity of the Tailorshop simulation, Journal of Individual Differences, 32, 225-233 (2011)
[21] Dayton, CM; Macready, GB, Concomitant-variable latent-class models, Journal of the American Statistical Association, 83, 173-178 (1988)
[22] de la Torre, J.; Minchen, N., Cognitively diagnostic assessments and the cognitive diagnosis model framework, Psicologia Educativa, 20, 89-97 (2014)
[23] DeMars, CE, Application of the bi-factor multidimensional item response theory model to Testlet-based tests, Journal of Educational Measurement, 43, 145-168 (2006)
[24] Demetriou, A.; Platsidou, M.; Efklides, A.; Metallidou, Y.; Shayer, M., The development of quantitative-relational abilities from childhood to adolescence: Structure, scaling, and individual differences, Learning and Instruction, 1, 19-43 (1991)
[25] Dempster, AP; Laird, NM; Rubin, DB, Maximum likelihood from incomplete data via the EM algorithm (with discussion), Journal of the Royal Statistical Society, Series B, 39, 1-38 (1977) · Zbl 0364.62022
[26] Formann, AK, Mixture analysis of multivariate categorical data with covariates and missing entries, Computational Statistics and Data Analysis, 51, 5236-5246 (2007) · Zbl 1445.62266
[27] Frischkorn, G.; Greiff, S.; Wüstenberg, S., The development of complex problem solving: A latent growth curve analysis, Journal of Educational Psychology, 106, 1004-1020 (2014)
[28] Funke, J.; Stern, E.; Guthke, J., Neue Verfahren zur Erfassung intelligenten Umgangs mit komplexen und dynamischen Anforderungen [new methods to assess intelligent dealing with complex and dynamic requirements], Perspektiven der Intelligenzforschung [Perspectives of intelligence research], 89-107 (2001), Lengerich D: Pabst Science Publishers, Lengerich D
[29] Funke, J.; Frensch, PA; Jonassen, DH, Complex problem solving: The European perspective—10 years after, Learning to solve complex scientific problems, 25-47 (2007), New York, NY: Lawrence Erlbaum, New York, NY
[30] Gibbons, RD; Bock, RD; Hedeker, D.; Weiss, DJ; Segawa, E.; Bhaumik, DK; Kupfer, DJ; Frank, E.; Grochocinski, VJ; Stover, A., Full-information item bifactor analysis of graded response data, Applied Psychological Measurement, 31, 4-19 (2007)
[31] Gibbons, RD; Hedeker, DR, Full-information item bi-factor analysis, Psychometrika, 57, 423-436 (1992) · Zbl 0760.62097
[32] Gnaldi, M. (2018). Indicators definition: Use of extended IRT models for composite indicator development. In Encyclopedia Wiley statistics reference. John Wiley & Sons. 10.1002/9781118445112.stat08112.
[33] Gnaldi, M.; Bacci, S.; Bartolucci, F., A multilevel finite mixture item response model to cluster examinees and schools, Advances in Data Analysis and Classification, 10, 53-70 (2015) · Zbl 1414.62491
[34] Gnaldi, M.; Bacci, S.; Greiff, S.; Kunze, T.; Petrucci, A.; Verde, R., Profiles of students on account of complex problem solving (CPS) strategies exploited via log-data, Statistics and data science: New challenges, new generations, 505-512 (2017), Firenze: FUP, Firenze
[35] Goodman, LA, Exploratory latent structure analysis using both identifiable and unidentifiable models, Biometrika, 61, 215-231 (1974) · Zbl 0281.62057
[36] Greiff, S.; Holt, DV; Funke, J., Perspectives on problem solving in educational assessment: Analytical, interactive, and collaborative problem solving, Journal of Problem Solving, 5, 71-91 (2013)
[37] Greiff, S.; Neubert, JC, On the relation of complex problem solving, personality, fluid intelligence, and academic achievement, Learning and Individual Differences, 36, 37-48 (2014)
[38] Greiff, S.; Wüstenberg, S.; Avvisati, F., Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving, Computers and Education, 91, 92-105 (2015)
[39] Greiff, S.; Wüstenberg, S.; Funke, J., Dynamic problem solving. A new measurement perspective, Applied Psychological Measurement, 36, 189-213 (2012)
[40] Greiff, S.; Wüstenberg, S.; Molnar, G.; Fischer, A.; Funke, J.; Csapo, B., Complex problem solving in educational contexts—Something beyond g: Concept, assessment, measurement invariance, and construct validity, Journal of Educational Psychology, 105, 364-379 (2013)
[41] Hambleton, RK; Swaminathan, H., Item response theory: Principles and applications (1985), Boston: Kluwer Nijhoff, Boston
[42] Jiao, H.; Lissitz, R.; Macready, GB; Wang, S.; Liang, S., Exploring levels of performance using the mixture Rasch model for standard setting, Psychological Test and Assessment Modeling, 53, 499-522 (2011)
[43] Kretzschmar, A.; Neubert, JC; Wüstenberg, S.; Greiff, S., Construct validity of complex problem solving: A comprehensive view on different facets of intelligence and school grades, Intelligence, 54, 55-69 (2016)
[44] Kroner, S.; Plass, JL; Leutner, D., Intelligence assessment with computer simulations, Intelligence, 33, 347-368 (2005)
[45] Lazarsfeld, PF; Henry, NW, Latent structure analysis (1968), Boston: Houghton Mifflin, Boston · Zbl 0182.52201
[46] Lotz, C.; Scherer, R.; Greiff, S.; Sparfeldt, JR, Intelligence in action. Effective strategic behaviors while solving complex problems, Intelligence, 64, 98-112 (2017)
[47] Maij-de Meij, AM; Kelderman, H.; van der Flier, H., Fitting a mixture item response theory model to personality questionnaire data: Characterizing latent classes and investigating possibilities for improving prediction, Applied Psychological Measurement, 32, 611-631 (2008)
[48] McLachlan, G.; Peel, D., Finite mixture models (2000), New York: Wiley, New York · Zbl 0963.62061
[49] Mislevy, RJ; Verhelst, N., Modeling item responses when different subjects employ different solution strategies, Psychometrika, 55, 195-215 (1990)
[50] Nair, KU; Ramnarayan, S., Individual differences in need for cognition and complex problem solving, Journal of Research in Personality, 34, 305-328 (2000)
[51] Education for life and work: Developing transferable knowledge and skills in the 21st century (2012), Washington, DC: The National Academies Press, Washington, DC
[52] PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems (2014), Paris: OECD Publishing, Paris
[53] Reckase, MD, Multidimensional item response theory (2009), New York: Springer, New York · Zbl 1291.62023
[54] Reise, SP, The rediscovery of bifactor measurement models, Multivariate Behavioral Research, 47, 667-696 (2012)
[55] Rost, J., Rasch models in latent classes: An integration of two approaches to item analysis, Applied Psychological Measurement, 14, 3, 271-282 (1990)
[56] Rost, J., A logistic mixture distribution model for polychotomous item responses, The British Journal of Mathematical and Statistical Psychology, 44, 75-92 (1991)
[57] Rost, J.; von Davier, M.; Fischer, GH; Molenaar, IW, Mixture distribution Rasch models, Rasch models. Foundations, recent developments and applications, 257-268 (1995), New York: Springer, New York · Zbl 0825.62926
[58] Rudolph, J.; Greiff, S.; Strobel, A.; Preckel, F., Understanding the link between need for cognition and complex problem solving, Contemporary Educational Psychology, 55, 53-62 (2018)
[59] Schwarz, G., Estimating the dimension of a model, Annals of Statistics, 6, 461-464 (1978) · Zbl 0379.62005
[60] Schweizer, F.; Wüstenberg, S.; Greiff, S., Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity, Learning and Individual Differences, 24, 42-52 (2013)
[61] Stadler, M.; Becker, N.; Godker, M.; Leutner, D.; Greiff, S., Complex problem solving and intelligence: A meta-analysis, Intelligence, 53, 92-101 (2015)
[62] Vainikainen, M., Finnish primary school pupils’ performance in learning to learn assessments: A longitudinal perspective on educational equity (2014), Helsinki: University of Helsinki, Helsinki
[63] Van der Linden, W.; Hambleton, RK, Handbook of modern item response theory (1997), New York: Springer, New York · Zbl 0872.62099
[64] Vollmeyer, R.; Rheinberg, F., Motivation and metacognition when learning a complex system, European Journal of Psychology of Education, 14, 541-554 (1999)
[65] von Davier, M., A general diagnostic model applied to language testing data, British Journal of Mathematical and Statistical Psychology, 61, 2, 287-307 (2008)
[66] Wang, W-C; Wilson, MR; Adams, RJ; Wilson, M.; Draney, K.; Eglehard, G., Rasch models for multidimensionality between items and within items, Objective measurement: Theory into practice, 139-155 (1997), Greenwich: Ablex Publishing, Greenwich
[67] Willemsen, MC; Johnson, EJ; Mecklenbeck, M.; Kühberger, A.; Ranyard, R., Visiting the decision factory: Observing cognition with MouselabWEB and other information acquisition methods, A handbook of process tracing methods for decision making, 21-42 (2011), New York, NY: Taylor & Francis, New York, NY
[68] Wüstenberg, S.; Greiff, S.; Funke, J., Complex problem solving. More than reasoning?, Intelligence, 40, 1-14 (2012)
[69] Wüstenberg, S.; Greiff, S.; Molnar, G.; Funke, J., Cross-national gender differences in complex problem solving and their determinants, Learning and Individual Differences, 29, 18-29 (2014)
[70] Wüstenberg, S.; Greiff, S.; Vainikainen, MP; Murphy, K., Individual differences in complex problem solving skills. How they evolve and what they imply, Journal of Educational Psychology, 108, 1028-1044 (2016)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.