Conditional density estimation in a regression setting. (English) Zbl 1129.62025

Efromovich, Sam

Ann. Stat. 35, No. 6, 2504-2535 (2007).

Summary: Regression problems are traditionally analyzed via univariate characteristics like the regression functions, scale functions and marginal density of regression errors. These characteristics are useful and informative whenever the association between the predictor and the response is relatively simple. More detailed information about the association can be provided by the conditional density of the response given the predictor. For the first time in the literature, this article develops a theory of minimax estimation of the conditional density for regression settings with fixed and random designs of predictors, bounded and unbounded responses and a vast set of anisotropic classes of conditional densities.
The study of fixed design regression is of special interest and novelty because the known literature is devoted to the case of random predictors. For the aforementioned models, the paper suggests a universal adaptive estimator which (i) matches performance of an oracle that knows both an underlying model and an estimated conditional density; (ii) is sharp minimax over a vast class of anisotropic conditional densities; (iii) is at least rate minimax when the response is independent of the predictor and thus a bivariate conditional density becomes a univariate density; (iv) is adaptive to an underlying design (fixed or random) of predictors.

MSC:

62G07	Density estimation
62C20	Minimax procedures in statistical decision theory
62E20	Asymptotic distribution theory in statistics

Keywords:

adaptation; parametric; analytic and Sobolev densities; anisotropic class; finite and infinite support; fixed and random designs; lower bound; MISE; oracle inequality; waste water treatment

References:

[1]	Arnold, B. C., Castillo, E. and Sarabia, J. M. (1999). Conditional Specification of Statistical Models. Springer, New York. · Zbl 0932.62001 · doi:10.1007/b97592
[2]	Bashtannyk, D. M. and Hyndman, R. J. (2001). Bandwidth selection for kernel conditional density estimation. Comput. Statist. Data Anal. 36 279-298. · Zbl 1038.62034 · doi:10.1016/S0167-9473(00)00046-3
[3]	Efromovich, S. (1985). Nonparametric estimation of a density with unknown smoothness. Theory Probab. Appl. 30 557-568. · Zbl 0593.62034 · doi:10.1137/1130067
[4]	Efromovich, S. (1989). On sequential nonparametric estimation of a density. Theory Probab. Appl. 34 228-239. · Zbl 0716.62077 · doi:10.1137/1134019
[5]	Efromovich, S. (1999). Nonparametric Curve Estimation : Methods , Theory and Applications. Springer, New York. · Zbl 0935.62039
[6]	Efromovich, S. (2000). On sharp adaptive estimation of multivariate curves. Math. Methods Statist. 9 117-139. · Zbl 1006.62033
[7]	Efromovich, S. (2001). Density estimation under random censorship and order restrictions: From asymptotic to small samples. J. Amer. Statist. Assoc. 96 667-684. · Zbl 1017.62029 · doi:10.1198/016214501753168334
[8]	Efromovich, S. (2005). Estimation of the density of regression errors. Ann. Statist. 33 2194-2227. · Zbl 1086.62053 · doi:10.1214/009053605000000435
[9]	Efromovich, S. (2005). Conditional density estimation in a regression setting: Small sample sizes and proofs. Technical report, Univ. New Mexico.
[10]	Efromovich, S. (2006). Dimension reduction, optimality and oracle approach in conditional density estimation. Technical report, Univ. Texas at Dallas.
[11]	Efromovich, S. (2007). Sequential design and estimation in heteroscedastic nonparametric regression; with discussion. Sequential Anal. 26 3-25. · Zbl 1112.62080 · doi:10.1080/07474940601109670
[12]	Efromovich, S. and Pinsker, M. S. (1982). Estimation of a square integrable probability density of a random variable. Problems Inform. Transmission 18 175-189. · Zbl 0533.62038
[13]	Efromovich, S. and Pinsker M. (1996). Sharp-optimal and adaptive estimation for heteroscedastic nonparametric regression. Statist. Sinica 6 925-942. · Zbl 0857.62037
[14]	Fan, J. (1992). Design-adaptive nonparametric regression. J. Amer. Statist. Assoc. 87 998-1004. · Zbl 0850.62354 · doi:10.2307/2290637
[15]	Fan, J. and Yao, Q. (2003). Nonlinear Time Series : Nonparametric and Parametric Methods . Springer, New York. · Zbl 1014.62103
[16]	Fan, J., Yao, Q. and Tong, H. (1996). Estimation of conditional densities and sensitivity measures in nonlinear dynamical systems. Biometrika 83 189-206. · Zbl 0865.62026 · doi:10.1093/biomet/83.1.189
[17]	Fan, J. and Yim, T. H. (2004). A cross-validation method for estimating conditional densities. Biometrika 91 819-834. · Zbl 1078.62032 · doi:10.1093/biomet/91.4.819
[18]	Golubev, G. K. (1991). Local asymptotic normality in problems of nonparametric estimation of functions, and lower bounds for quadratic risks. Theory Probab. Appl. 36 152-157. · Zbl 0738.62043 · doi:10.1137/1136014
[19]	Golubev, G. K. (1992). Nonparametric estimation of smooth densities of a distribution in \(L_2\). Problems Inform. Transmission 28 44-54. · Zbl 0785.62039
[20]	Golubev, G. K. and Levit, B. Y. (1996). Asymptotically efficient estimation for analytic distributions. Math. Methods Statist. 5 357-368. · Zbl 0872.62042
[21]	Hall, P., Racine, J. and Li, Q. (2004). Cross-validation and the estimation of conditional probability densities. J. Amer. Statist. Assoc. 99 1015-1026. · Zbl 1055.62035 · doi:10.1198/16214504000000548
[22]	Hall, P., Wolff, R. C. L. and Yao, Q. (1999). Methods for estimating a conditional distribution function. J. Amer. Statist. Assoc. 94 154-163. · Zbl 1072.62558 · doi:10.2307/2669691
[23]	Hall, P. and Yao, Q. (2005). Approximating conditional distribution functions using dimension reduction. Ann. Statist. 33 1404-1421. · Zbl 1072.62008 · doi:10.1214/009053604000001282
[24]	Hasminskii, R. and Ibragimov, I. (1990). On density estimation in the view of Kolmogorov’s ideas in approximation theory. Ann. Statist. 18 999-1010. · Zbl 0705.62039 · doi:10.1214/aos/1176347736
[25]	Hoffmann, M. and Lepski, O. (2002). Random rates in anisotropic regression (with discussion). Ann. Statist. 30 325-396. · Zbl 1012.62042 · doi:10.1214/aos/1021379858
[26]	Hyndman, R. J., Bashtannyk, D. M. and Grunwald, G. K. (1996). Estimating and visualizing conditional densities. J. Comput. Graph. Statist. 5 315-336.
[27]	Hyndman, R. J. and Yao, Q. (2002). Nonparametric estimation and symmetry tests for conditional density functions. J. Nonparametr. Statist. 14 259-278. · Zbl 1013.62040 · doi:10.1080/10485250212374
[28]	Ibragimov, I. A. and Hasminskii, R. Z. (1983). Estimation of distribution density belonging to a class of entire functions. Theory Probab. Appl. 27 551-562. · Zbl 0516.62043 · doi:10.1137/1127062
[29]	Kahane, J.-P. (1985). Some Random Series of Functions , 2nd ed. Cambridge Univ. Press. · Zbl 0571.60002
[30]	Kawata, T. (1972). Fourier Analysis in Probability Theory. Academic Press, New York. · Zbl 0271.60022
[31]	Neter, J., Kutner, M., Nachtsheim, C. and Wasserman, W. (1996). Applied Linear Statistical Models , 4th ed. McGraw-Hill, Boston. · Zbl 0347.62043
[32]	Nikolskii, S. M. (1975). Approximation of Functions of Several Variables and Imbedding Theorems . Springer, New York.
[33]	Pinsker, M. S. (1980). Optimal filtering of square integrable signals in Gaussian white noise. Problems Inform. Transmission 16 52-68. · Zbl 0452.94003
[34]	Prakasa Rao, B. L. S. (1983). Nonparametric Functional Estimation . Academic Press, New York. · Zbl 0542.62025
[35]	Rosenblatt, M. (1969). Conditional probability density and regression estimators. In Multivariate Analysis II (P. R. Krishnaiah, ed.) 25-31. Academic Press, New York.
[36]	Schipper, M. (1996). Optimal rates and constants in \(L_2\)-minimax estimation of probability density functions. Math. Methods Statist. 5 253-274. · Zbl 0872.62043