×

Maximum entropy algorithm with inexact upper entropy bound based on \(Fup\) basis functions with compact support. (English) Zbl 1416.62052

Summary: The maximum entropy (MaxEnt) principle is a versatile tool for statistical inference of the probability density function (pdf) from its moments as a least-biased estimation among all other possible pdf’s. It maximizes Shannon entropy, satisfying the moment constraints. Thus, the MaxEnt algorithm transforms the original constrained optimization problem to the unconstrained dual optimization problem using Lagrangian multipliers. The Classic Moment Problem (CMP) uses algebraic power moments, causing typical conventional numerical methods to fail for higher-order moments \((m>5-10)\) due to different sensitivities of Lagrangian multipliers and unbalanced nonlinearities. Classic MaxEnt algorithms overcome these difficulties by using orthogonal polynomials, which enable roughly the same sensitivity for all Lagrangian multipliers. In this paper, we employ an idea based on different principles, using \(Fup_n\) basis functions with compact support, which can exactly describe algebraic polynomials, but only if the \(Fup\) order-\(n\) is greater than or equal to the polynomial’s order. Our algorithm solves the CMP with respect to the moments of only low order \(Fup_{2}\) basis functions, finding a \(Fup_{2}\) optimal pdf with better balanced Lagrangian multipliers. The algorithm is numerically very efficient due to localized properties of \(Fup_{2}\) basis functions implying a weaker dependence between Lagrangian multipliers and faster convergence. Only consequences are an iterative scheme of the algorithm where power moments are a sum of \(Fup_{2}\) and residual moments and an inexact entropy upper bound. However, due to small residual moments, the algorithm converges very quickly as demonstrated on two continuous pdf examples – the beta distribution and a bi-modal pdf, and two discontinuous pdf examples – the step and double Dirac pdf. Finally, these pdf examples present that \(Fup\) MaxEnt algorithm yields smaller entropy value than classic MaxEnt algorithm, but differences are very small for all practical engineering purposes.

MSC:

62B10 Statistical aspects of information-theoretic topics
62E17 Approximations to statistical distributions (nonasymptotic)
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Abramov, R., A practical computational framework for the multidimensional moment-constrained maximum entropy principle, J. Comp. Phys., 211, 1, 198-209 (2006) · Zbl 1080.65007
[2] Abramov, R., An improved algorithm for the multidimensional moment-constrained maximum entropy problem, J. Comp. Phys., 226, 621-644 (2007) · Zbl 1125.65010
[3] Abramov, R., The multidimensional moment-constrained maximum entropy problem: a BFGS algorithm with constraint scaling, J. Comp. Phys., 228, 96-108 (2009) · Zbl 1157.65381
[4] Andricevic, R., Exposure concentration statistics in the subsurface transport, Adv. Water Res., 31, 8, 714-725 (2008)
[5] Berger, A. L.; Della Pietra, S. A.; Della Pietra, V. J., A maximum entropy approach to natural language processing, Comp. Linguistics, 22, 1, 39-71 (1996)
[6] Bandyopadhyay, K.; Bhattacharya, A.; Biswas, P.; Drabold, D., Maximum entropy and the problem of moments. A stable algorithm, Phys. Rev. E, 71, 5 (2005)
[7] Christakos, G., Modern Spatiotemporal Statistics (2000), Oxford University Press: Oxford University Press New York
[8] Gotovac, B.; Kozulić, V., On a selection of basis functions in numerical analyses of engineering problems, Int. J. Eng. Model., 12, 1-4, 25-41 (1999)
[9] Gotovac, B.; Kozulić, V., Numerical solving the initial value problems by \(R_{bf}\) basis functions, Struct. Eng. Mech., 14, 3, 263-285 (2002)
[10] Gotovac, H.; Andricevic, R.; Gotovac, B., Multi-resolution adaptive modeling of groundwater flow and transport problems, Adv. Water Res., 30, 1105-1126 (2007)
[11] Gotovac, H.; Cvetkovic, V.; Andricevic, R., Adaptive Fup multi-resolution approach to flow and advective transport in highly heterogeneous porous media: methodology accuracy and convergence, Adv. Water Res., 32, 885-905 (2009)
[12] Kolodiazhny, V. M.; Rvachev, V. A., Atomic functions: generalization to the multivariable case and promising applications, Cybern. Syst. Anal., 43, 6, 893-911 (2007) · Zbl 1151.35301
[13] Kravchenko, V. F.; Basarab, M. A., Approximations by atomic functions and numerical methods for Fredholm integral equations of the second kind, Differen. Equat., 37, 10, 1480-1490 (2001) · Zbl 1018.65149
[14] Kravchenko, V. F.; Basarab, M. A.; Perez-Meana, H., Spectral properties of atomic functions used in digital signal processing, J. Commun. Technol. Electron., 46, 494-511 (2001)
[15] Jaynes, E. T., Information theory and statistical mechanics, Phy. Rev., 106, 620-630 (1957) · Zbl 0084.43701
[16] Mead, L.; Papanicolaou, N., Maximum entropy in the problem of moments, J. Math. Phys., 25, 2404-2417 (1984)
[17] Ormoneit, D.; White, H., An efficient algorithm to compute maximum entropy densities, Econ. Rev., 18, 2, 127-140 (1999) · Zbl 0932.62006
[18] Rvachev, V. L.; Rvachev, V. A., Pro odnu finitnu funkciju, DAN URSR. Ser. A, 6, 705-707 (1971), (in Russian)
[19] Rvachev, V. L., Teorija R-funkcij i nekotorija jeje priloženija (in Russian), Kiev: Naukova dumka, 551 (1982)
[20] Shannon, C. E., Bell Syst. Tech. J., 27, 379-623 (1948), reprinted in C.E. Shannon and W. Weawer, The Mathematical Theory of Communication (University of Illinois Press, Urbana, 1949) · Zbl 1154.94303
[21] Steeb, W.-H.; Solms, F.; Stoop, R., Chaotic systems and maximum entropy formalism, J. Phys. A: Math. Gen., 27, 399-402 (1994) · Zbl 0875.58016
[22] Tung, Y.-K.; Yen, B.-K.; Melching, C. S., Hydrosystems Engineering Reliability Assessment and Risk Analysis (2006), McGraw-Hill: McGraw-Hill New York, p. 495
[23] Turek, I., A maximum-entropy approach to the density of states with the recursion method, J. Phys. C, 21, 3251-3260 (1988)
[24] Wu, X., Calculation of maximum entropy densities with application to income distribution, J. Econ., 115, 347-354 (2003) · Zbl 1016.62094
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.