Assessment of the number of components in Gaussian mixture models in the presence of multiple local maximizers. (English) Zbl 1280.62028

Summary: Gaussian mixtures are very flexible in representing the underlying structure of the data. However, likelihood inference for Gaussian mixtures with unrestricted covariance matrices is theoretically and practically challenging because the likelihood function is unbounded and often has multiple local maximizers. As shown in the numerical studies of this paper, the presence of multiple local maximizers including spurious local maximizers affects the performances of model selection criteria used to choose the number of components. We propose a new type of likelihood-based estimators, a gradient-based \(k\)-deleted maximum likelihood estimator, for Gaussian mixture models. The proposed estimator is designed to avoid spurious local maximizers and to choose a statistically desirable local maximizer in the presence of multiple local maximizers. We first prove the consistency of the proposed estimator and then examine, by a real-data example and simulation studies, the performance of the proposed method in the likelihood-based model selection criteria commonly used to assess the number of components in Gaussian mixture models.


62F10 Point estimation
62H12 Estimation in multivariate analysis
62F12 Asymptotic properties of parametric estimators
Full Text: DOI


[1] H. Akaike, Information theory and an extension of the maximum likelihood principle, in: B.N. Petrov, F. Csaki (Eds.), 2nd International Symposium on Information Theory, 1973, pp. 267-281. · Zbl 0283.62006
[2] Biernacki, C.; Celeux, G.; Govaert, G., Assessing a mixture model for clustering with the integrated completed likelihood, IEEE Trans. Pattern Anal. Mach. Intell., 22, 719-725, (2000)
[3] Biernacki, C.; Chrétien, S., Degeneracy in the maximum likelihood estimation of univariate Gaussian mixtures with EM, Statist. Probab. Lett., 61, 373-382, (2003) · Zbl 1038.62023
[4] Böhning, D.; Dietz, E.; Schaub, R.; Schlattmann, P.; Lindsay, B. G., The distribution of the likelihood ratio for mixtures of densities from the one-parameter exponential family, Ann. Inst. Statist. Math., 46, 373-388, (1994) · Zbl 0802.62017
[5] Bozdogan, H., Model selection and akaikes information criterion (aic): the general theory and its analytical extensions, Psychometrika, 52, 345-370, (1987) · Zbl 0627.62005
[6] Celeux, G.; Soromenho, G., An entropy criterion for assessing the number of clusters in a mixture model, J. Classification, 13, 195-212, (1996) · Zbl 0861.62051
[7] Chen, J.; Tan, X., Inference for multivariate normal mixtures, J. Multivariate Anal., 100, 1367-1383, (2009) · Zbl 1162.62052
[8] Chen, J.; Tan, X.; Zhang, R., Consistency of penalized mle for normal mixtures in mean and variance, Statist. Sinica, 18, 443-465, (2008)
[9] Ciuperca, G. A.; Ridolfi, A.; Idier, J., Penalized maximum likelihood estimator for normal mixtures, Scand. J. Statist., 30, 45-59, (2003) · Zbl 1034.62018
[10] Day, N. E., Estimating the components of a mixture of normal distributions, Biometrika, 56, 463-474, (1969) · Zbl 0183.48106
[11] Dempster, A.; Laird, N.; Rubin, D., Maximum likelihood from incomplete data via the EM algorithm, J. R. Stat. Soc. Ser. B, 39, 1-38, (1977) · Zbl 0364.62022
[12] Everitt, B. S.; Hand, D. J., Finite mixture distributions, (1981), Chapman & Hall London · Zbl 0466.62018
[13] Finch, S. J.; Mendell, N. R.; Thode, H. C., Probabilistic measures of adequacy of a numerical search for a global maximum, J. Amer. Statist. Assoc., 84, 1020-1023, (1989)
[14] Fraley, C.; Raftery, A. E., How many clusters? which clustering method? answer via model-based cluster analysis, Comput. J., 41, 578-588, (1998) · Zbl 0920.68038
[15] Fraley, C.; Raftery, A. E., Model-based clustering, discriminant analysis, and density estimation, J. Amer. Statist. Assoc., 97, 611-631, (2002) · Zbl 1073.62545
[16] Fraley, C.; Raftery, A. E., Bayesian regularization for normal mixture estimation and model-based clustering, J. Classification, 24, 155-181, (2007) · Zbl 1159.62302
[17] Frühwirth-Schnatter, S., Finite mixture and Markov switching models, (1996), Springer New York · Zbl 1108.62002
[18] Hathaway, R. J., A constrained formulation of maximum-likelihood estimation for normal mixture distributions, Ann. Statist., 13, 795-800, (1985) · Zbl 0576.62039
[19] Hennig, C., Methods for merging Gaussian mixture components, Adv. Data Anal. Classif., 4, 3-34, (2010) · Zbl 1306.62141
[20] Keribin, C., Consistent estimation of the order of mixture models, Sankhyā Ser. A, 62, 49-66, (2000) · Zbl 1081.62516
[21] Linde, Y.; Buzo, A.; Gray, R., An algorithm for vector quantizer design, IEEE Trans. Commun., 28, 84-94, (1980)
[22] Lindsay, B. G., (Mixture Models: Theory, Geometry, and Applications, NSF-CBMS Regional Conference Series in Probability and Statistics, vol. 5, (1995), IMS US)
[23] McLachlan, G. J.; Peel, D., Finite mixture models, (2000), John Wiley and Sons Ltd. New York · Zbl 0963.62061
[24] Schwarz, G., Estimating the dimension of a model, Ann. Statist., 6, 461-464, (1978) · Zbl 0379.62005
[25] Seidel, W.; S˘evc˘íková, H., Types of likelihood maxima in mixture models and their implication on the performance of tests, Ann. Inst. Statist. Math., 56, 631-654, (2004) · Zbl 1078.62014
[26] Seo, B.; Kim, D., Root selection in normal mixture models, Comput. Statist. Data Anal., 56, 2454-2470, (2012) · Zbl 1252.62013
[27] Seo, B.; Lindsay, B. G., A computational strategy for doubly smoothed MLE exemplified in the normal mixture model, Comput. Statist. Data Anal., 54, 1930-1941, (2010) · Zbl 1284.62234
[28] Small, C. G.; Wang, J.; Yang, Z., Eliminating multiple root problems in estimation, Statist. Sci., 15, 313-332, (2000)
[29] Soromenho, G., Comparing approaches for testing the number of components in a finite mixture model, Comput. Statist., 9, 65-78, (1994) · Zbl 0940.62043
[30] Tanaka, K.; Takemura, A., Strong consistency of the maximum likelihood estimator for finite mixtures of location-scale distributions when the scale parameters are exponentially small, Bernoulli, 12, 1003-1017, (2006) · Zbl 1117.62025
[31] Wald, A., Note on the consistency of the maximum likelihood estimate, Ann. Math. Statist., 20, 595-601, (1949) · Zbl 0034.22902
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.