×

PCA and SVD with nonnegative loadings. (English) Zbl 1173.68676

Summary: Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) are widely used in multivariate statistical analysis for data reduction. The work considers exponential, logit, and multinomial parameterization of the eigenvectors’ elements that always yields nonnegative loadings of shares for variable aggregation. In contrast to regular PCA and SVD, matrix decomposition by the positive shares shows explicitly which variables and with which percent are composed into each group, so what is each variable contribution to data approximation. The least squares objective of matrix fit is reduced to Rayleigh quotient for variational description of the eigenvalues. Eigenvectors with the nonlinear parameterization can be found in Newton-Raphson optimizing procedure. Numerical examples compare the classical and nonnegative loadings results, with interpretation by the Perron-Frobenius theory for each subset of variables identified by sparse loading vectors.

MSC:

68T10 Pattern recognition, speech recognition

Software:

ElemStatLearn; PRMLT
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Pearson, K., On lines and planes of closest fit to systems of points in space, Philos. Mag., 2, 559-572 (1901) · JFM 32.0246.07
[2] Hotelling, H., Analysis of a complex of statistical variables into principal components, J. Educ. Psychol., 24, 417-441 (1933)
[3] Hotelling, H., Simplified calculation of principal components, Psychometrika, 1, 27-35 (1936) · JFM 62.1352.04
[4] Eckart, C.; Young, G., The approximation of one matrix by another of lower rank, Psychometrika, 1, 211-218 (1936) · JFM 62.1075.02
[5] Lynn, H. S.; McCulloch, C. E., Using principal component analysis and correspondence analysis for estimation in latent variable models, J. Am. Stat. Assoc., 95, 561-572 (2000) · Zbl 0996.62060
[6] Lipovetsky, S.; Tishler, A.; Conklin, M., Multivariate least squares and its relation to other multivariate techniques, Appl. Stochastic Models Bus. Ind., 18, 347-356 (2002) · Zbl 1039.62049
[7] Weingessel, A.; Hornik, K., A robust subspace algorithm for principal component analysis, Int. J. Neural Syst., 13, 307-313 (2003)
[8] Pavoine, S.; Dufour, A. B.; Chessel, D., From dissimilarities among species to dissimilarities among communities: a double principal coordinate analysis, J. Theoret. Biol., 228, 523-537 (2004) · Zbl 1439.92198
[9] Levi, Y.; Last, I.; Jortner, J., Dynamics of fission and Coulomb explosion of multicharged large finite systems, Mol. Phys., 104, 1227-1237 (2006)
[10] Rousson, V.; Gasser, T., Simple component analysis, Appl. Statist., 53, 539-555 (2004) · Zbl 1111.62310
[11] Lipovetsky, S., Multimode data analysis for decision making, (Tzeng, G. H.; Wang, H. F.; Wen, U. P.; Yu, P. L., Multiple Criteria Decision Making (1994), Springer: Springer New York), 275-282 · Zbl 0817.90049
[12] deLathauwer, L.; deMoor, B.; Vandewalle, J., A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl., 21, 1253-1278 (2000) · Zbl 0962.15005
[13] Drineas, P.; Frieze, A.; Kannan, R.; Vempala, S.; Vinay, V., Clustering large graphs via the singular value decomposition, Mach. Learn., 56, 9-33 (2004) · Zbl 1089.68090
[14] Kazantsev, S. G.; Bukhgeim, A. A., Singular value decomposition for the 2D fan-beam Radon transform of tensor fields, J. Inverse Ill-Posed Probl., 12, 245-278 (2004) · Zbl 1058.65143
[15] Hochstenbach, M. E., Harmonic and refined extraction methods for the singular value problem, with application in least squares problems, BIT Numer. Math., 44, 721-754 (2004) · Zbl 1079.65047
[16] Carroll, J. D.; Green, P. E.; Schaffer, C. M., Interpoint distance comparisons in correspondence analysis, J. Mark. Res., 23, 271-280 (1986)
[17] Shugan, S. M., Estimating brand positioning maps using supermarket scanning data, J. Mark. Res., 24, 1-18 (1987)
[18] Kaciak, E.; Louviere, J., Multiple correspondence analysis of multiple choice experiment data, J. Mark. Res., 27, 455-465 (1990)
[19] Steenkamp, J. B.; Trijp, H. V.; Berge, J. T., Perceptual mapping based on idiosyncratic sets, J. Mark. Res., 31, 15-27 (1994)
[20] Sinha, I.; DeSarbo, W. S., An integrated approach toward the spatial modeling of perceived customer value, J. Mark. Res., 35, 236-249 (1998)
[21] Hastie, T.; Tibshirani, R.; Friedman, J., The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2001), Springer: Springer New York · Zbl 0973.62007
[22] Bishop, C. M., Pattern Recognition and Machine Learning (2006), Springer: Springer New York · Zbl 1107.68072
[23] Elden, L., Matrix Methods in Data Mining and Pattern Recognition (2007), SIAM: SIAM Philadelphia · Zbl 1120.68092
[24] Hoerl, A. E.; Kennard, R. W., Ridge regression: biased estimation for nonorthogonal problems, Technometrics, 12, 55-67 (1970), (Reprinted in: Technometrics 42 (2000) 80-86) · Zbl 0202.17205
[25] Lipovetsky, S., Two-parameter ridge regression and its convergence to the eventual pairwise model, Math. Comput. Modelling, 44, 304-318 (2006) · Zbl 1133.62340
[26] Tibshirani, R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. B, 58, 267-288 (1996) · Zbl 0850.62538
[27] Frank, I.; Friedman, J., A statistical view of some chemometrics regression tools, Technometrics, 35, 109-148 (1993) · Zbl 0775.62288
[28] Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Stat., 32, 407-489 (2004) · Zbl 1091.62054
[29] Lipovetsky, S., Optimal \(L_p\)-metric for minimizing powered deviations in regression, J. Mod. Appl. Stat. Methods, 6, 219-227 (2007)
[30] Jolliffe, I. T.; Trendafilov, N.; Uddin, M., A modified principal component technique based on the lasso, J. Comput. Graphical Stat., 12, 531-547 (2003)
[31] Zou, H.; Hastie, T.; Tibshirani, R., Sparse principal component analysis, J. Comput. Graphical Stat., 15, 265-286 (2006)
[32] Hoyer, P. O., Non-negative matrix factorization with sparseness constraints, J. Mach. Learn. Res., 5, 1457-1469 (2004) · Zbl 1222.68218
[33] Berry, M. W.; Browne, M., Email surveillance using non-negative matrix factorization, Comput. Math. Organ. Theory, 11, 249-264 (2005) · Zbl 1086.68502
[34] Pauca, V. P.; Piper, J.; Plemmons, R. J., Nonnegative matrix factorization for spectral data analysis, Linear Algebra Appl., 416, 29-47 (2006) · Zbl 1096.65033
[35] Fogel, P.; Young, S. S.; Hawkins, D. M.; Ledirac, N., robust non-negative matrix factorization analysis of microarray data, Bioinformatics, 23, 44-49 (2007)
[36] Lipovetsky, S.; Conklin, M., Singular value decomposition in additive, multiplicative, and logistic forms, Pattern Recognition, 38, 1099-1110 (2005) · Zbl 1069.68591
[37] Bellman, R., Introduction to Matrix Analysis (1960), McGraw-Hill: McGraw-Hill New York · Zbl 0124.01001
[38] Marcus, M.; Minc, H., A Survey of Matrix Theory and Matrix Inequalities (1964), Allyn & Bacon: Allyn & Bacon Boston · Zbl 0126.02404
[39] (Arminger, G.; Clogg, C. C.; Sobel, M. E., Handbook of Statistical Modeling for the Social and Behavioral Sciences (1995), Plenum Press: Plenum Press New York, London) · Zbl 0837.62102
[40] McCullagh, P.; Nelder, J. A., Generalized Linear Models (1997), Chapman & Hall: Chapman & Hall London, New York · Zbl 0588.62104
[41] Lipovetsky, S., Equidistant regression modeling, Model Assisted Stat. Appl., 2, 71-80 (2007) · Zbl 1121.62065
[42] Golub, G. H.; van Loan, C. F., Matrix Computations (1996), Johns Hopkins University Press: Johns Hopkins University Press Baltimore, MD · Zbl 0865.65009
[43] Chambers, J. M.; Hastie, T. J., Statistical Models in S (1992), Wadsworth & Brooks: Wadsworth & Brooks Pacific Grove, CA · Zbl 0776.62007
[44] S-PLUS’2000, MathSoft Inc., Seattle, WA, 1999.; S-PLUS’2000, MathSoft Inc., Seattle, WA, 1999.
[45] Gantmacher, F. R., Applications of the Theory of Matrices (1959), Interscience: Interscience New York · Zbl 0085.01001
[46] Salton, G., Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer (1988), Addison-Wesley: Addison-Wesley Reading, MA
[47] Harville, D. A., Matrix Algebra from a Statistician’ Perspective (1997), Springer: Springer New York
[48] Lipovetsky, S.; Conklin, M., Regression by data segments via discriminant analysis, J. Mod. Appl. Stat. Methods, 4, 63-74 (2005)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.