Estimating mode effects from a sequential mixed-mode experiment using structural moment models. (English) Zbl 1498.62317

Summary: Until recently, the survey mode of the household panel study Understanding Society was mainly face-to-face interview, but it has now adopted a mixed-mode design where individuals can self-complete the questionnaire via the web. As mode is known to affect survey data, a randomized mixed-mode experiment was implemented during the first year of the two-year Wave 8 fieldwork period to assess the impact of this change. The experiment involved a sequential design that permits the identification of mode effects in the presence of nonignorable nonrandom mode selection. While previous studies have used instrumental variables regression to estimate the effects of mode on the means of the survey variables, we describe a more general methodology based on novel structural moment models that characterizes the overall effect of mode on a survey by its effects on the moments of the survey variables’ joint distribution. We adapt our estimation procedure to account for nonresponse and complex sampling designs and to include suitable auxiliary data to improve inference and relax key assumptions. Finally, we demonstrate how to estimate the effects of mode on the parameter estimates of generalized linear models and other exponential family models when both outcomes and predictors are subject to mode effects. This methodology is used to investigate the impact of the move to web mode on Wave 8 of Understanding Society.


62P25 Applications of statistics to social sciences
62D20 Causal inference from observational studies
Full Text: DOI


[1] ANGRIST, J. D., IMBENS, G. W. and RUBIN, D. R. (1996). Identification of causal effects using instrumental variables (with discussion). J. Amer. Statist. Assoc. 91 444-472. · Zbl 0897.62130
[2] BUELENS, B. and VAN DEN BRAKEL, J. A. (2017). Comparing two inferential approaches to handling measurement error in mixed-mode surveys. J. Off. Stat. 33 513-531.
[3] CARPENTER, H. (2018). UK Household Longitudinal Study: Wave 8 Technical Report. Kantar Public. Available at https://doc.ukdataservice.ac.uk/doc/6614/mrdoc/pdf/6614_wave8_technical_report.pdf.
[4] CLARKE, P. S. and BAO, Y. (2022a). Supplement to “Estimating mode effects from a sequential mixed-mode experiment using structural moment models. (Part A).” https://doi.org/10.1214/21-AOAS1557SUPPA
[5] CLARKE, P. S. and BAO, Y. (2022b). Supplement to “Estimating mode effects from a sequential mixed-mode experiment using structural moment models (Part B).” https://doi.org/10.1214/21-AOAS1557SUPPB
[6] CLARKE, P. S., PALMER, T. M. and WINDMEIJER, F. (2015). Estimating structural mean models with multiple instrumental variables using the generalised method of moments. Statist. Sci. 30 96-117. · Zbl 1332.62408 · doi:10.1214/14-STS503
[7] CLARKE, P. S. and WINDMEIJER, F. (2010). Identification of causal effects on binary outcomes using structural mean models. Biostatistics 11 756-770. · Zbl 1437.62423
[8] D’ARDENNE, J., COLLINS, D., GRAY, M., JESSOP, C. and PILLEY, S. (2017). Assessing the risk of mode effects: Review of proposed survey questions for waves 7-10 of Understanding Society. Understanding Society Working Paper Series 2017-04.
[9] FIELD, C. A. and WELSH, A. H. (2007). Bootstrapping clustered data. J. R. Stat. Soc. Ser. B. Stat. Methodol. 69 369-390. · Zbl 07555357 · doi:10.1111/j.1467-9868.2007.00593.x
[10] Hansen, L. P. (1982). Large sample properties of generalized method of moments estimators. Econometrica 50 1029-1054. · Zbl 0502.62098 · doi:10.2307/1912775
[11] HERNÁN, M. A. and ROBINS, J. M. (2020). Causal Inference: What If. Champman & Hall/CRC, Boca Raton, FL.
[12] IMBENS, G. W. and RUBIN, D. B. (1997). Estimating outcome distributions for compliers in instrumental variables models 64 555-574. · Zbl 0887.90041 · doi:10.2307/2971731
[13] ISER (2018). Understanding Society: Waves 1-8, 2009-2017 and Harmonised BHPS: Waves 1-18, 1991-2009. [data collection], 11th ed. University of Essex, Institute for Social and Economic Research. UK Data Service. SN: 6614. · doi:10.5255/UKDA-SN-6614-13
[14] JÄCKLE, A., GAIA, A. and BENZEVAL, M. (2017). Mixing modes and measurement models in longitudinal studies. CLOSER Resource Report. CLOSER , London, UK.
[15] JÄCKLE, A., ROBERTS, C. and LYNN, P. (2010). Assessing the effect of data collection mode on measurement. Int. Stat. Rev. 78 3-20.
[16] KOLNENIKOV, S. and KENNEDY, C. (2014). Evaluating three approaches to statistically adjust for mode effects. Journal of Survey Statistics & Methodology 2 126-58.
[17] LUGTIG, P., LENSVELT-MULDERS, G. J. L. M., FRERICHS, R. and GREVEN, A. (2011). Estimating nonresponse bias and mode effects in a mixed-mode survey. Int. J. Mark. Res. 53 669-686.
[18] PARK, S., KIM, J. K. and PARK, S. (2016). An imputation approach for handling mixed-mode surveys. Ann. Appl. Stat. 10 1063-1085. · Zbl 1398.62382 · doi:10.1214/16-AOAS930
[19] Robins, J. M. (1994). Correcting for non-compliance in randomized trials using structural nested mean models. Comm. Statist. Theory Methods 23 2379-2412. · Zbl 0825.62203 · doi:10.1080/03610929408831393
[20] STOCK, J. H. and YOGO, M. (2005). Testing for weak instruments in linear IV regression. In Identification and Inference for Econometric Models (D. W. K. Andrews and J. H. Stock, eds.) 80-108. Cambridge Univ. Press, Cambridge. · Zbl 1121.62066 · doi:10.1017/CBO9780511614491.006
[21] Tsiatis, A. A. (2006). Semiparametric Theory and Missing Data. Springer Series in Statistics. Springer, New York. · Zbl 1105.62002
[22] VANNIEUWENHUYZE, J. T. A. (2015). Mode effects on variances, covariances, standard deviations, and correlations. Journal of Survey Statistics & Methodology 3 1-21.
[23] VANNIEUWENHUYZE, J. T. A. and LOOSVELDT, G. (2013). Evaluating relative mode effects in mixed-mode surveys: Three methods to disentangle selection and measurement effects. Sociol. Methods Res. 42 82-104. · doi:10.1177/0049124112464868
[24] VANNIEUWENHUYZE, J., LOOSVELDT, G. and MOLENBERGHS, G. (2010). A method for evaluating mode effects in mixed-mode surveys. Public Opin. Q. 74 1027-1045.
[25] VANNIEUWENHUYZE, J. T. A., LOOSVELDT, G. and MOLENBERGHS, G. (2014). Evaluating mode effects in mixed-mode survey data using covariate adjustment models. J. Off. Stat. 30 1-21.
[26] VANSTEELANDT, S. and JOFFE, M. (2014). Structural nested models and G-estimation: The partially realized promise. Statist. Sci. 29 707-731. · Zbl 1331.62208 · doi:10.1214/14-STS493
[27] Wooldridge, J. M. (2010). Econometric Analysis of Cross Section and Panel Data, 2nd ed. MIT Press, Cambridge, MA · Zbl 1327.62009
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.