Distributed multinomial regression. (English) Zbl 1454.62036

Taddy, Matt

Ann. Appl. Stat. 9, No. 3, 1394-1414 (2015).

Summary: This article introduces a model-based approach to distributed computing for multinomial logistic (softmax) regression. We treat counts for each response category as independent Poisson regressions via plug-in estimates for fixed effects shared across categories. The work is driven by the high-dimensional-response multinomial models that are used in analysis of a large number of random counts. Our motivating applications are in text analysis, where documents are tokenized and the token counts are modeled as arising from a multinomial dependent upon document attributes. We estimate such models for a publicly available data set of reviews from Yelp, with text regressed onto a large set of explanatory variables (user, business, and rating information). The fitted models serve as a basis for exploring the connection between words and variables of interest, for reducing dimension into supervised factor scores, and for prediction. We argue that the approach herein provides an attractive option for social scientists and other text analysts who wish to bring familiar regression tools to bear on text data.

MSC:

62-08	Computational methods for problems pertaining to statistics
62J12	Generalized linear models (logistic models)

Keywords:

distributed computing; logistic regression; Lasso; text analysis; multinomial inverse regression; computational social science

Software:

MapReduce; R; MASS (R); glmnet

References:

[1]	Birch, M. W. (1963). Maximum likelihood in three-way contingency tables. J. Roy. Statist. Soc. Ser. B 25 220-233. · Zbl 0121.14001
[2]	Cook, R. D. (2007). Fisher lecture: Dimension reduction in regression. Statist. Sci. 22 1-26. · Zbl 1246.62148 · doi:10.1214/088342306000000682
[3]	Dean, J. and Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Comm. ACM 51 107-113.
[4]	Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33 1-22.
[5]	Gopalan, P., Hofman, J. M. and Blei, D. M. (2013). Scalable recommendation with Poisson factorization. Available at . arXiv:1311.1704
[6]	Hodges, J. L. Jr. and Le Cam, L. (1960). The Poisson approximation to the Poisson binomial distribution. Ann. Math. Statist. 31 737-740. · Zbl 0100.14301 · doi:10.1214/aoms/1177705799
[7]	Hurvich, C. M. and Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika 76 297-307. · Zbl 0669.62085 · doi:10.1093/biomet/76.2.297
[8]	McDonald, D. R. (1980). On the Poisson approximation to the multinomial distribution. Canad. J. Statist. 8 115-118. · Zbl 0452.62016 · doi:10.2307/3314676
[9]	Palmgren, J. (1981). The Fisher information matrix for log linear models arguing conditionally on observed explanatory variables. Biometrika 68 563-566. · Zbl 0477.62039
[10]	Taddy, M. (2013a). Multinomial inverse regression for text analysis. J. Amer. Statist. Assoc. 108 755-770. · Zbl 06224965 · doi:10.1080/01621459.2012.734168
[11]	Taddy, M. (2013b). Rejoinder: Efficiency and structure in MNIR. J. Amer. Statist. Assoc. 108 772-774. · doi:10.1080/01621459.2013.821408
[12]	Taddy, M. (2013c). Measuring political sentiment on Twitter: Factor optimal design for multinomial inverse regression. Technometrics 55 415-425. · doi:10.1080/00401706.2013.778791
[13]	Taddy, M. (2014). One-step estimator paths for concave regularization. Available at . arXiv:1308.5623
[14]	Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Roy. Statist. Soc. Ser. B 58 267-288. · Zbl 0850.62538
[15]	Venables, W. N. and Ripley, B. D. (2002). Modern Applied Statistics with S , 4th ed. Springer, New York. · Zbl 1006.62003 · doi:10.1007/b97626