Mixture densities, maximum likelihood and the EM algorithm. (English) Zbl 0536.62021

Redner, Richard A.; Walker, Homer F.

SIAM Rev. 26, 195-239 (1984).

This paper surveys the problem of estimating parameters of a finite mixture with mixing proportions \((\alpha_ 1,...,\alpha_ m)\) and mixing pdfs \(p(x| \Phi_ i)\) where each \(\Phi_ i\) is an \(r_ i\)- dimensional parameter. Different types of sampling schemes are considered based on whether the observations can be labelled or not, partially or completely. The main emphasis is on obtaining strongly consistent and CAN estimators.
The method of MLE using iterative solutions based on the method of scoring as well as other variants of Newton’s method is considered. The main problems with MLE are (i) log L may be unbounded above, (ii) log L may have several local maxima and even the largest one may not be unique. The iterative procedures essentially depend upon the Fisher information matrix or the Hessian of log L which has usually large condition number and consequently requires very large samples to obtain reasonably accurate estimates. This is illustrated by the problem of mixture of two normals. The EM algorithm (EMA) first proposed by A. P. Dempster, N. M. Laird and D. B. Rubin [J. R. Stat. Soc., Ser. B 39, 1-38 (1977; Zbl 0364.62022)] is reviewed extensively as it is applied to the mixture densities and it is shown that under suitable conditions one can guarantee the convergence of EMA leading to strongly consistent estimates which are CAN.
The application of EMA is considered in the case where \(p(x| \Phi_ i)\) belong to the exponential class of densities in general and is illustrated for the case of Poisson, binomial and multivariate normal distributions. The authors point out some of the deficiencies of the EMA and suggest developing an algorithm which is a hybrid mixture of EMA and Newton’s method or its variants. This is an excellent review paper which would appreciated by a very wide section of readers.

Reviewer: B.K.Kale

MSC:

62F12	Asymptotic properties of parametric estimators
65C99	Probabilistic methods, stochastic differential equations
65D15	Algorithms for approximation of functions
65H10	Numerical computation of solutions to systems of equations

Keywords:

maximum-likelihood estimation; incomplete data problems; condition number; local maximization; mixtures; sampling schemes; method of scoring; variants of Newton’s method; iterative procedures; Fisher information matrix; Hessian; EM algorithm; strongly consistent estimates; exponential class; Poisson; binomial; multivariate normal distributions; review paper

Citations:

Zbl 0364.62022