×

The infinite Student’s \(t\)-factor mixture analyzer for robust clustering and classification. (English) Zbl 1248.68420

Summary: Recently, the Student’s \(t\)-factor mixture analyzer (\(t\)FMA) has been proposed. Compared with the mixture of Student’s \(t\)-factor analyzers (MtFA), the \(t\)FMA has better performance when processing high-dimensional data. Moreover, the factors estimated by the \(t\)FMA can be visualized in a low-dimensional latent space, which is not shared by the M\(t\)FMA. However, as the \(t\)FMA belongs to finite mixtures and the related parameter estimation method is based on the maximum likelihood criterion, it could not automatically determine the appropriate model complexity according to the observed data, leading to overfitting. In this paper, we propose an infinite Student’s \(t\)-factor mixture analyzer (i\(t\)FMA) to handle this issue. The i\(t\)FMA is based on the nonparametric Bayesian statistics which assumes infinite number of mixing components in advance, and automatically determines the proper number of components after observing the high-dimensional data. Moreover, we derive an efficient variational inference algorithm for the i\(t\)FMA. The proposed i\(t\)FMA and the related variational inference algorithm are used to cluster and classify high-dimensional data. Experimental results of some applications show that the i\(t\)FMA has good generalization capacity, offering a more robust and powerful performance than other competing approaches.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
62H30 Classification and discrimination; cluster analysis (statistical aspects)

Software:

BayesDA; PGMM; UCI-ml
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Spearman, C., The proof and measurement of association between two things, American Journal of Psychology, 15, 72-101 (1904)
[2] M.S. Bartlett, Factor analysis in psychology as a statistician sees it, in: Uppsala Symposium on Psychological Factor Analysis, Number 3 in Nordisk Psykologi’s Monograph Series, 1953, pp. 23-43.; M.S. Bartlett, Factor analysis in psychology as a statistician sees it, in: Uppsala Symposium on Psychological Factor Analysis, Number 3 in Nordisk Psykologi’s Monograph Series, 1953, pp. 23-43.
[3] Bartholomew, D.; Knott, M.; Moustaki, I., Latent Variable Models and Factor Analysis: A Unified Approach (2011), John Wiley series in Probability and Statistics · Zbl 1266.62040
[4] Z. Ghahramani, G.E. Hinton, The EM algorithm for mixtures of factor analyzers, Technical Report, CRG-TR-96-1, Department of Computer Science, University of Toronto, 1997.; Z. Ghahramani, G.E. Hinton, The EM algorithm for mixtures of factor analyzers, Technical Report, CRG-TR-96-1, Department of Computer Science, University of Toronto, 1997.
[5] Carin, L.; Baraniuk, R. G.; Cevher, V.; Dunson, D.; Jordan, M. I.; Sapiro, G.; Wakin, M. B., Learning low-dimensional signal models, IEEE Signal Processing Magazine, 28, 2, 381-396 (2011)
[6] Montanari, A.; Viroli, C., Maximum likelihood estimation of mixture of factor analyzers, Computational Statistics and Data Analysis, 55, 9, 2712-2723 (2011) · Zbl 1464.62134
[7] McLachlan, G. J.; Peel, D., Finite Mixture Models (2000), Wiley: Wiley New York · Zbl 0963.62061
[8] McLachlan, G. J.; Peel, D., Robust cluster analysis via mixtures of multivariate \(t\)-distributions, Lecture Notes in Computer Science, 1451, 658-666 (1998)
[9] Peel, D.; McLachlan, G. J., Robust mixture modelling using the \(t\) distribution, Statistics and Computing, 10, 4, 339-348 (2000)
[10] McLachlan, G. J.; Bean, R. W.; Jones, B. J., Extension of the mixture of factor analyzers model to incorporate the multivariate \(t\)-distribution, Computational Statistics and Data Analysis, 51, 11, 5327-5338 (2007) · Zbl 1445.62053
[11] Chatzis, S.; Kosmopoulos, D.; Varvarigou, T., Signal modeling and classification using a robust latent space model based on \(t\) distributions, IEEE Transactions on Signal Processing, 56, 3, 949-963 (2008) · Zbl 1390.94123
[12] Andrews, J. L.; McNicholas, P. D., Mixtures of modified \(t\)-factor analyzers for model-based clustering, classification, and discriminant analysis, Journal of Statistical Planning and Inference, 141, 4, 1479-1486 (2011) · Zbl 1204.62098
[13] Andrews, J. L.; McNicholas, P. D., Extending mixtures of multivariate \(t\)-factor analyzers, Statistics and Computing, 21, 3, 361-373 (2011) · Zbl 1255.62171
[14] McNicholas, P. D.; Murphy, T. B., Parsimonious Gaussian mixture models, Statistics and Computing, 18, 3, 285-296 (2008)
[15] McNicholas, P. D.; Murphy, T. B., Model-based clustering of microarray expression data via latent Gaussian mixture models, Bioinformatics, 26, 21, 2705-2712 (2010)
[16] Baek, J.; McLachlan, G. J., Mixtures of common \(t\)-factor analyzers for clustering high-dimensional microarray data, Bioinformatics, 27, 9, 1269-1276 (2011)
[17] Montanari, A.; Viroli, C., Heteroscedastic factor mixture analysis, Statistical Modelling, 10, 4, 441-460 (2010) · Zbl 07256833
[18] Viroli, C., Dimensionally reduced model-based clustering through mixtures of factor mixture analyzers, Journal of Classification, 27, 3, 363-388 (2010) · Zbl 1337.62141
[19] Baek, J.; McLachlan, G. J.; Flack, L. K., Mixtures of factor analyzers with common factor loadings: applications to the clustering and visualization of high-dimensional data, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32, 7, 1298-1309 (2010)
[20] Melnykov, V.; Maitra, R., Finite mixture models and model-based clustering, Statistics Surveys, 4, 80-116 (2010) · Zbl 1190.62121
[21] Xiong, Y.; Yeung, D. Y., Time series clustering with ARMA mixtures, Pattern Recognition, 37, 8, 1675-1689 (2004) · Zbl 1117.62488
[22] Yamazaki, K.; Watanabe, S., Singularities in mixture models and upper bounds of stochastic complexity, Neural Networks, 16, 7, 1029-1038 (2003) · Zbl 1255.68130
[23] Hjort, N.; Holmes, C.; Muller, P., Bayesian Nonparametrics: Principles and Practice (2010), Cambridge University Press
[24] Gershman, S. J.; Blei, D. M., A tutorial on Bayesian nonparametric models, Journal of Mathematical Psychology, 56, 1, 1-12 (2012) · Zbl 1237.62062
[25] Antoniak, C., Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems, Annals of Statistics, 2, 1152-1174 (1974) · Zbl 0335.60034
[26] Bouguila, N.; Ziou, D., A Dirichlet process mixture of generalized Dirichlet distributions for proportional data modeling, IEEE Transactions on Neural Networks, 21, 1, 107-122 (2010)
[27] Wei, X.; Li, C., The infinite Student’s \(t\)-mixture for robust modeling, Signal Processing, 92, 1, 224-234 (2012)
[28] Jordan, M. I., A introduction to variational methods for graphical models, Machine Learning, 37, 2, 183-233 (1999) · Zbl 0945.68164
[29] Bruneau, P.; Gelgon, M.; Picarougne, F., Parsimonious reduction of Gaussian mixture models with a variational-Bayes approach, Pattern Recognition, 43, 3, 850-858 (2010) · Zbl 1187.68432
[30] Blei, D. M.; Jordan, M. I., Variational inference for Dirichlet process mixtures, Bayesian Analysis, 1, 1, 121-144 (2006) · Zbl 1331.62259
[31] Ishwaran, H.; James, L., Gibbs sampling methods for stick-breaking priors, Journal of the American Statistical Association, 96, 453, 161-173 (2001) · Zbl 1014.62006
[32] Gelman, A.; Carlin, J. B.; Stern, H. S.; Rubin, D. B., Bayesian Data Analysis (2004), Chapman& Hall/CRC: Chapman& Hall/CRC London, UK · Zbl 1039.62018
[33] M. Opper, D. Saad (Eds.), Advanced Mean Field Methods: Theory and Practice, MIT Press, Cambridge, MA, 2001.; M. Opper, D. Saad (Eds.), Advanced Mean Field Methods: Theory and Practice, MIT Press, Cambridge, MA, 2001. · Zbl 0994.68172
[34] Wei, X.; Li, C., The Student’s \(t\)-hidden Markov model with truncated stick-breaking priors, IEEE Signal Processing Letters, 18, 6, 355-358 (2011)
[35] McNicholas, P. D.; Murphy, T. B.; McDaid, A. F.; Frost, D., Serial and parallel implementations of model-based clustering via parsimonious Gaussian mixture models, Computational Statistics and Data Analysis, 54, 3, 711-723 (2010) · Zbl 1464.62131
[36] B.G. Lindsay, Mixture models: theory, geometry and applications, Vol. 5 of NSF-CBMS Regional Conference Series in Probability and Statistics, Institute of Mathematical Statistics, Hayward, CA, 1995.; B.G. Lindsay, Mixture models: theory, geometry and applications, Vol. 5 of NSF-CBMS Regional Conference Series in Probability and Statistics, Institute of Mathematical Statistics, Hayward, CA, 1995. · Zbl 1163.62326
[37] Bohning, D.; Dietz, E.; Schaub, R.; Schlattmann, P.; Lindsay, B. G., The distribution of the likelihood ratio for mixtures of densities from the one-parameter exponential family, Annals of the Institute of Statistical Mathematics, 46, 2, 373-388 (1994) · Zbl 0802.62017
[38] Hastie, T.; Tibshirani, R., Discriminant analysis by Gaussian mixtures, Journal of the Royal Statistical Society: Series B, 58, 1, 155-176 (1996) · Zbl 0850.62476
[39] Hubert, L.; Arabie, P., Comparing partitions, Journal of Classification, 2, 1, 193-218 (1985)
[40] Yeoh, E. J., Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling, Cancer Cell, 1, 2, 133-143 (2002)
[41] McLachlan, G. J.; Bean, R. W.; Peel, D., A mixture model-based approach to the clustering of microarray expression data, Bioinformatics, 18, 3, 413-422 (2002)
[42] A. Asuncion, D.J. Newman, UCI Machine Learning Repository \([ \langle\) http://www.ics.uci.edu/∼mlearn/MLRepository.html \(\rangle \); A. Asuncion, D.J. Newman, UCI Machine Learning Repository \([ \langle\) http://www.ics.uci.edu/∼mlearn/MLRepository.html \(\rangle \)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.