Personalized policy learning using longitudinal mobile health data. (English) Zbl 1457.62347

Hu, Xinyu; Qian, Min; Cheng, Bin; Cheung, Ying Kuen

J. Am. Stat. Assoc. 116, No. 533, 410-420 (2021).

Summary: Personalized policy represents a paradigm shift one decision rule for all users to an individualized decision rule for each user. Developing personalized policy in mobile health applications imposes challenges. First, for lack of adherence, data from each user are limited. Second, unmeasured contextual factors can potentially impact on decision making. Aiming to optimize immediate rewards, we propose using a generalized linear mixed modeling framework where population features and individual features are modeled as fixed and random effects, respectively, and synthesized to form the personalized policy. The group lasso type penalty is imposed to avoid overfitting of individual deviations from the population model. We examine the conditions under which the proposed method work in the presence of time-varying endogenous covariates, and provide conditional optimality and marginal consistency results of the expected immediate outcome under the estimated policies. We apply our method to develop personalized push (“prompt”) schedules in 294 app users, with the goal to maximize the prompt response rate given past app usage and other contextual factors. The proposed method compares favorably to existing estimation methods including using the R function “glmer” in a simulation study.

MSC:

62P10	Applications of statistics to biology and medical sciences; meta analysis
62J05	Linear regression; mixed models
62C05	General considerations in statistical decision theory

Keywords:

contextual bandits; endogenous variables; generalized linear mixed model; individualized decision rule; push notifications

Software:

glmer; tn; R; lme4

References:

[1]	Bates, D., Mächler, M., Bolker, B., and Walker, S. (2014), “Fitting Linear Mixed-Effects Models Using lme4,” arXiv no. 1406.5823.
[2]	Bohmer, M.; Hecht, B.; Schoning, J.; Kruger, A.; Bauer, G., Falling Asleep With Angry Birds, Facebook and Kindle—A Large Scale Study on Mobile Application Usage, 47-56 (2011)
[3]	Boruvka, A.; Almirall, D.; Witkiewitz, K.; Murphy, S. A., “Assessing Time-Varying Causal Effect Moderation in Mobile Health, Journal of the American Statistical Association, 113, 1112-1121 (2018)
[4]	Cheung, K.; Ling, W.; Karr, C. J.; Weingardt, K.; Schueller, S. M.; Mohr, D. C., “Evaluation of a Recommender App for Apps for the Treatment of Depression and Anxiety: An Analysis of Longitudinal User Engagement, Journal of the American Medical Informatics Association, 25, 955-962 (2018)
[5]	Cheung, Y. K.; Chakraborty, B.; Davidson, K. W., “Sequential Multiple Assignment Randomized Trial (SMART) With Adaptive Randomization for Quality Improvement in Depression Treatment Program, Biometrics, 71, 450-459 (2015) · Zbl 1390.62244
[6]	Cho, H.; Wang, P.; Qu, A., “Personalize Treatment for Longitudinal Data Using Unspecified Random-Effects Model, Statistica Sinica, 27, 187-206 (2017) · Zbl 1468.62322
[7]	Christensen, H.; Griffiths, K.; Farrer, L., “Adherence in Internet Interventions for Anxiety and Depression, Journal of Medical Internet Research, 11, e13 (2009)
[8]	Depp, C. A.; Mausbach, B.; Granholm, E.; Cardenas, V.; Ben-Zeev, D.; Patterson, T. L.; Lebowitz, B. D.; Jeste, D. V., “Mobile Interventions for Severe Mental Illness: Design and Preliminary Data From Three Approaches, The Journal of Nervous and Mental Disease, 198, 715 (2010)
[9]	Diggle, P.; Heagerty, P.; Liang, K.-Y.; Zeger, S., Analysis of Longitudinal Data (2002), Oxford: Oxford University Press, Oxford
[10]	Ertefaie, A.; Strawderman, R. L., “Constructing Dynamic Treatment Regimes Over Indefinite Time Horizons, Biometrika, 105, 963-977 (2018) · Zbl 1506.62432
[11]	Helander, E.; Kaipainen, K.; Korhonen, I.; Wansink, B., “Factors Related to Sustained Use of a Free Mobile App for Dietary Self-Monitoring With Photography and Peer Feedback: Retrospective Cohort Study, Journal of Medical Internet Research, 16, e109 (2014)
[12]	Heron, K. E.; Smyth, J. M., “Ecological Momentary Interventions: Incorporating Mobile Technology Into Psychosocial and Health Behaviour Treatments, British Journal of Health Psychology, 15, 1-39 (2010)
[13]	Hui, F. K.; Müller, S.; Welsh, A., “Joint Selection in Mixed Models Using Regularized PQL, Journal of the American Statistical Association, 112, 1323-1333 (2017)
[14]	Kravitz, R.; Duan, N., “Design and Implementation of N-of-1 Trials: A User’s Guide (2014)
[15]	Laber, E.; Linn, K.; Stefanski, L., “Interactive Model Building for Q-Learning, Biometrika, 101, 831-847 (2014) · Zbl 1306.62235
[16]	Lei, H., Tewari, A., and Murphy, S. (2017), “An Actor-Critic Contextual Bandit Algorithm for Personalized Interventions Using Mobile Devices,” arXiv no. 1706.09090.
[17]	Lillie, E.; Patay, B.; Diamant, J.; Issell, B.; Topol, E.; Schork, N., “The n-of-1 Clinical Trial: The Ultimate Strategy for Individualizing Medicine, Personalized Medicine, 8, 161-173 (2011)
[18]	Luckett, D. J.; Laber, E. B.; Kahkoska, A. R.; Maahs, D. M.; Mayer-Davis, E.; Kosorok, M. R., “Estimating Dynamic Treatment Regimes in Mobile Health Using V-Learning, Journal of the American Statistical Association, 115, 692-706 (2020) · Zbl 1445.62279
[19]	Mohr, D. C.; Cheung, K.; Schueller, S. M.; Brown, C. H.; Duan, N., “Continuous Evaluation of Evolving Behavioral Intervention Technologies, American Journal of Preventive Medicine, 45, 517-523 (2013)
[20]	Mohr, D. C.; Tomasino, K. N.; Lattie, E. G.; Palac, H. L.; Kwasny, M. J.; Weingardt, K.; Karr, C. J.; Kaiser, S. M.; Rossom, R. C.; Bardsley, L. R.; Caccamo, L., “IntelliCare: An Eclectic, Skills-Based App Suite for the Treatment of Depression and Anxiety, Journal of Medical Internet Research, 19, e10 (2017)
[21]	Murphy, S. A., “Optimal Dynamic Treatment Regimes, Journal of the Royal Statistical Society, Series B, 65, 331-355 (2003) · Zbl 1065.62006
[22]	Nash, S. G., “A Survey of Truncated-Newton Methods, Journal of Computational and Applied Mathematics, 124, 45-59 (2000) · Zbl 0969.65054
[23]	Ohrnberger, J.; Fichera, E.; Sutton, M., “The Rise of Consumer Health Wearables: Promises and Barriers, Journal of the Economics of Ageing, 9, 52-62 (2017)
[24]	Pepe, M. S., and Anderson, G. L. (1994), “A Cautionary Note on Inference for Marginal Regression Models With Longitudinal Data and General Correlated Response Data,” Communications in Statistics—Simulation and Computation, 23, 939-951. DOI: .
[25]	Qian, M.; Murphy, S. A., “Performance Guarantees for Individualized Treatment Rules, Annals of Statistics, 39, 1180-1210 (2011) · Zbl 1216.62178
[26]	Qian, T., Klasnja, P., and Murphy, S. A. (2019), “Linear Mixed Models Under Endogeneity: Modeling Sequential Treatment Effects With Application to a Mobile Health Study,” arXiv no. 1902.10861. · Zbl 07292522
[27]	Riley, W. T.; Rivera, D. E.; Atienza, A. A.; Nilsen, W.; Allison, S. M.; Mermelstein, R., “Health Behavior Models in the Age of Mobile Interventions: Are Our Theories up to the Task?, Translational Behavioral Medicine, 1, 53-71 (2011)
[28]	Song, R.; Wang, W.; Zeng, D.; Kosorok, M. R., “Penalized Q-Learning for Dynamic Treatment Regimes, Statistical Sinica, 25, 901-920 (2015) · Zbl 1415.62054
[29]	Steihaug, T., “The Conjugate Gradient Method and Trust Regions in Large Scale Optimization, SIAM Journal on Numerical Analysis, 20, 626-637 (1983) · Zbl 0518.65042
[30]	Tewari, A.; Murphy, S.; Rehg, J.; Murphy, S.; Kumar, S., Mobile Health, From Ads to Interventions: Contextual Bandits in Mobile Health, 495-517 (2017), Cham: Springer, Cham
[31]	Yuan, M.; Lin, Y., “Model Selection and Estimation in Regression With Grouped Variables, Journal of the Royal Statistical Society, Series B, 68, 49-67 (2006) · Zbl 1141.62030
[32]	Zhang, B.; Tsiatis, A. A.; Laber, E. B.; Davidian, M., “A Robust Method for Estimating Optimal Treatment Regimes, Biometrics, 68, 1010-1018 (2012) · Zbl 1258.62116
[33]	Zhao, Y.; Zeng, D.; Laber, E. B.; Kosorok, M. R., “New Statistical Learning Methods for Estimating Optimal Dynamic Treatment Regimes, Journal of the American Statistical Association, 110, 583-598 (2015) · Zbl 1373.62557
[34]	Zhao, Y.; Zeng, D.; Rush, A. J.; Kosorok, M. R., “Estimating Individualized Treatment Rules Using Outcome Weighted Learning, Journal of the American Statistical Association, 107, 1106-1118 (2012) · Zbl 1443.62396