The least weighted deviation. (English) Zbl 0705.62015

Guiasu, Silviu

Inf. Sci. 53, No. 3, 271-284 (1991).

Summary: The probability distribution p is found that minimizes the weighted deviation from a given probability distribution q when the range and the mean value of a random variable are known. A particular weighted deviation is taken into account only to the extent to which the optimization problem has a unique and calculable solution. The commonest known measures of deviation (Pearson’s chi-square, the Kullback-Leibler divergence, etc.) are reobtained as functional variants of the deviations used in a Euclidean normed space. An adequate choice of the weights may improve the accuracy in predicting the unknown probability distribution.
In the 19 th century, R. Wolf tossed an unfair die 20,000 times and obtained a probability distribution yielding a mean value of the number of pips equal to \(\mu =3.5983\). Knowing \(\mu\), can we predict Wolf’s distribution after tossing his die only 1,000 or fewer times? The choice of the weights in the weighted deviation used in prediction proves to be essential even in the case when these weights depend only on the given probability distribution q.

MSC:

62B10	Statistical aspects of information-theoretic topics
62E99	Statistical distribution theory

Keywords:

least weighted deviation; optimization; measures of deviation; Pearson’s chi-square; Kullback-Leibler divergence; Euclidean normed space

References:

[1]	Abrahams, J., On the selection of measures of distance between probability distributions, Inform. Sci., 26, 109-113 (1982) · Zbl 0507.60034
[2]	Berkson, J., Minimum chi-square, not maximum likelihood (with discussion), Ann. Statist., 4, 457-487 (1980) · Zbl 0456.62023
[3]	Fisher, R. A., The conditions under which \(χ^2\) measures the discrepancy between observations and hypothesis, J. Roy. Statist. Soc., 87, 442-450 (1924)
[4]	Guiasu, S., Information Theory with Applications (1977), McGraw-Hill: McGraw-Hill New York · Zbl 0379.94027
[5]	Jaynes, E. T., Where do we stand on maximum entropy?, (Levine, R. D.; Tribus, M., The Maximum Entropy Formalism (1979), MIT Press: MIT Press Cambridge, Mass.), 15-118
[6]	(Justice, J. H., Maximum Entropy and Bayesian Methods in Applied Statistics (1986), Cambridge U.P.,: Cambridge U.P., Cambridge, England) · Zbl 0597.00025
[7]	Kuliback, S., Information Theory and Statistics (1969), Dover: Dover New York
[8]	Neave, H. R., Statistics Tables (1979), George Allen & Unwin: George Allen & Unwin London
[9]	Pearson, K., On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling, Philos. Mag. 5th Ser., 50, 157-175 (1900) · JFM 31.0238.04
[10]	Rowlinson, J. S., Probability, information and entropy, Nature, 225, 1196 (1970)
[11]	Shannon, C. E., A mathematical theory of communication, Bell System Tech. J., 27, 379-423 (1948) · Zbl 1154.94303