[1] 
Babu, G. J. and Singh, K. (1983). Inference on means using the bootstrap. Ann. Statist. 11 9991003. · Zbl 0539.62043
· doi:10.1214/aos/1176346267

[2] 
BarndorffNielsen, O. E. (1983). On a formula for the distribution of the maximum likelihood estimator. Biometrika 70 343365. JSTOR: · Zbl 0532.62006
· doi:10.1093/biomet/70.2.343
· http://links.jstor.org/sici?sici=00063444%28198308%2970%3A2%3C343%3AOAFFTD%3E2.0.CO%3B2N&origin=euclid 
[3] 
BarndorffNielsen, O. E. (1986). Inference on full or partial parameters based on the standardized signed log likelihood ratio. Biometrika 73 307322. JSTOR: · Zbl 0605.62020
· http://links.jstor.org/sici?sici=00063444%28198608%2973%3A2%3C307%3AIOFOPP%3E2.0.CO%3B2Z&origin=euclid 
[4] 
BarndorffNielsen, O. E. (1994). Adjusted versions of profile likelihood and likelihood roots, and extended likelihood. J. Roy. Statist. Soc. Ser. B 56 125140. JSTOR: · Zbl 0798.62005
· http://links.jstor.org/sici?sici=00359246%281994%2956%3A1%3C125%3AAVOPLA%3E2.0.CO%3B2Z&origin=euclid 
[5] 
BarndorffNielsen, O. E. and Chamberlin, S. R. (1994). Stable and invariant adjusted directed likelihoods. Biometrika 81 485499. JSTOR: · Zbl 0812.62030
· doi:10.1093/biomet/81.3.485
· http://links.jstor.org/sici?sici=00063444%28199408%2981%3A3%3C485%3ASAIADL%3E2.0.CO%3B2%23&origin=euclid 
[6] 
Beran, R. (1987). Prepivoting to reduce level error of confidence sets. Biometrika 74 457468. JSTOR: · Zbl 0663.62045
· doi:10.1093/biomet/74.3.457
· http://links.jstor.org/sici?sici=00063444%28198709%2974%3A3%3C457%3APTRLEO%3E2.0.CO%3B20&origin=euclid 
[7] 
Bickel, P. J. (1987). Comment on ”Better bootstrap confidence intervals” by B. Efron. J. Amer. Statist. Assoc. 82 191. JSTOR: · Zbl 0622.62039
· doi:10.2307/2289144
· http://links.jstor.org/sici?sici=01621459%28198703%2982%3A397%3C171%3ABBCI%3E2.0.CO%3B2H&origin=euclid 
[8] 
Bickel, P. J. (1988). Discussion of ”Theoretical comparison of bootstrap confidence intervals” by P. Hall. Ann. Statist. 16 959961. · Zbl 0663.62046
· doi:10.1214/aos/1176350933

[9] 
Cox, D. R. and Reid, N. (1987). Parameter orthogonality and approximate conditional inference. J. Roy. Statist. Soc. Ser. B 49 139. JSTOR: · Zbl 0616.62006
· http://links.jstor.org/sici?sici=00359246%281987%2949%3A1%3C1%3APOAACI%3E2.0.CO%3B2E&origin=euclid 
[10] 
Cox, D. R. and Reid, N. (1993). A note on the calculation of adjusted profile likelihood. J. Roy. Statist. Soc. Ser. B 55 467472. JSTOR: · Zbl 0797.62015
· http://links.jstor.org/sici?sici=00359246%281993%2955%3A2%3C467%3AANOTCO%3E2.0.CO%3B20&origin=euclid 
[11] 
DiCiccio, T. J. (1984). On parameter transformations and interval estimation. Biometrika 71 477485. JSTOR: · Zbl 0566.62022
· doi:10.1093/biomet/71.3.477
· http://links.jstor.org/sici?sici=00063444%28198412%2971%3A3%3C477%3AOPTAIE%3E2.0.CO%3B2M&origin=euclid 
[12] 
DiCiccio, T. J. and Efron, B. (1992). More accurate confidence intervals in exponential families. Biometrika 79 231245. JSTOR: · Zbl 0752.62027
· doi:10.1093/biomet/79.2.231
· http://links.jstor.org/sici?sici=00063444%28199206%2979%3A2%3C231%3AMACIIE%3E2.0.CO%3B2W&origin=euclid 
[13] 
DiCiccio, T. J. and Martin, M. (1993). Simple modifications for signed roots of likelihood ratio statistics. J. Roy. Statist. Soc. Ser. B 55 305316. JSTOR: · Zbl 0794.62014
· http://links.jstor.org/sici?sici=00359246%281993%2955%3A1%3C305%3ASMFSRO%3E2.0.CO%3B24&origin=euclid 
[14] 
DiCiccio, T. J. and Romano, J. P. (1995). On bootstrap procedures for secondorder accurate confidence limits in parametric models. Statist. Sinica 5 141160. · Zbl 0829.62035 
[15] 
Efron, B. (1979). Bootstrap methods: another look at the jackknife. Ann. Statist. 7 126. · Zbl 0406.62024
· doi:10.1214/aos/1176344552

[16] 
Efron, B. (1981). Nonparametric estimates of standard error: the jackknife, the bootstrap, and other methods. Biometrika 68 589599. JSTOR: · Zbl 0487.62031
· doi:10.1093/biomet/68.3.589
· http://links.jstor.org/sici?sici=00063444%28198112%2968%3A3%3C589%3ANEOSET%3E2.0.CO%3B22&origin=euclid 
[17] 
Efron, B. (1987). Better bootstrap confidence intervals (with discussion). J. Amer. Statist. Assoc. 82 171200. JSTOR: · Zbl 0622.62039
· doi:10.2307/2289144
· http://links.jstor.org/sici?sici=01621459%28198703%2982%3A397%3C171%3ABBCI%3E2.0.CO%3B2H&origin=euclid 
[18] 
Efron, B. (1993). Bay es and likelihood calculations from confidence intervals. Biometrika 80 326. JSTOR: · Zbl 0773.62021
· doi:10.1093/biomet/80.1.3
· http://links.jstor.org/sici?sici=00063444%28199303%2980%3A1%3C3%3ABALCFC%3E2.0.CO%3B29&origin=euclid 
[19] 
Efron, B. (1994). Missing data, imputation, and the bootstrap (with comment and rejoinder). J. Amer. Statist. Assoc. 89 463478. JSTOR: · Zbl 0806.62033
· doi:10.2307/2290846
· http://links.jstor.org/sici?sici=01621459%28199406%2989%3A426%3C463%3AMDIATB%3E2.0.CO%3B24&origin=euclid 
[20] 
Efron, B. and Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman and Hall, New York. · Zbl 0835.62038 
[21] 
Hall, P. (1986). On the bootstrap and confidence intervals. Ann. Statist. 14 14311452. · Zbl 0611.62047
· doi:10.1214/aos/1176350168

[22] 
Hall, P. (1988). Theoretical comparison of bootstrap confidence intervals (with discussion). Ann. Statist. 16 927985. · Zbl 0663.62046
· doi:10.1214/aos/1176350933

[23] 
Hall, P. and Martin, M. A. (1988). On bootstrap resampling and iteration. Biometrika 75 661671. JSTOR: · Zbl 0659.62053
· doi:10.1093/biomet/75.4.661
· http://links.jstor.org/sici?sici=00063444%28198812%2975%3A4%3C661%3AOBRAI%3E2.0.CO%3B2K&origin=euclid 
[24] 
Hougaard, P. (1982). Parametrizations of nonlinear models. J. Roy. Statist. Soc. Ser. B 44 244252. JSTOR: · Zbl 0494.62032
· http://links.jstor.org/sici?sici=00359246%281982%2944%3A2%3C244%3APONM%3E2.0.CO%3B2X&origin=euclid 
[25] 
Lawley, D. N. (1956). A general method for approximating to the distribution of the likelihood ratio criteria. Biometrika 43 295303. JSTOR: · Zbl 0073.13602
· doi:10.1093/biomet/43.34.295
· http://links.jstor.org/sici?sici=00063444%28195612%2943%3A3%2F4%3C295%3AAGMFAT%3E2.0.CO%3B2J&origin=euclid 
[26] 
Loh, W.Y. (1987). Calibrating confidence coefficients. J. Amer. Statist. Assoc. 82 155162. JSTOR: · Zbl 0608.62057
· doi:10.2307/2289142
· http://links.jstor.org/sici?sici=01621459%28198703%2982%3A397%3C155%3ACCC%3E2.0.CO%3B2C&origin=euclid 
[27] 
McCullagh, P. (1984). Local sufficiency. Biometrika 71 233244. JSTOR: · Zbl 0573.62026
· doi:10.1093/biomet/71.2.233
· http://links.jstor.org/sici?sici=00063444%28198408%2971%3A2%3C233%3ALS%3E2.0.CO%3B2Q&origin=euclid 
[28] 
McCullagh, P. (1987). Tensor Methods in Statistics. Chapman and Hall, London. · Zbl 0732.62003 
[29] 
McCullagh, P. and Tibshirani, R. (1990). A simple method for the adjustment of profile likelihoods. J. Roy. Statist. Soc. Ser. B 52 325344. JSTOR: · Zbl 0716.62039
· http://links.jstor.org/sici?sici=00359246%281990%2952%3A2%3C325%3AASMFTA%3E2.0.CO%3B27&origin=euclid 
[30] 
Peers, H. W. (1965). On confidence points and Bayesian probability points in the case of several parameters. J. Roy. Statist. Soc. Ser. B 27 916. JSTOR: · Zbl 0144.41403
· http://links.jstor.org/sici?sici=00359246%281965%2927%3A1%3C9%3AOCPABP%3E2.0.CO%3B2O&origin=euclid 
[31] 
Pierce, D. and Peters, D. (1992). Practical use of higher order asy mptotics for multiparameter exponential families (with discussion) J. Roy. Stat. Soc. Ser. B 54 701725. JSTOR:
· http://links.jstor.org/sici?sici=00359246%281992%2954%3A3%3C701%3APUOHOA%3E2.0.CO%3B2H&origin=euclid 
[32] 
Sprott, D. A. (1980). Maximum likelihood in small samples: estimation in the presence of nuisance parameters. Biometrika 67 515523. JSTOR: · Zbl 0447.62040
· doi:10.1093/biomet/67.3.515
· http://links.jstor.org/sici?sici=00063444%28198012%2967%3A3%3C515%3AMLISSE%3E2.0.CO%3B2B&origin=euclid 
[33] 
Tibshirani, R. (1988). Variance stabilization and the bootstrap. Biometrika 75 433444. JSTOR: · Zbl 0651.62039
· doi:10.1093/biomet/75.3.433
· http://links.jstor.org/sici?sici=00063444%28198809%2975%3A3%3C433%3AVSATB%3E2.0.CO%3B20&origin=euclid 
[34] 
Tibshirani, R. (1989). Noninformative priors for one parameter of many. Biometrika 76 604608. JSTOR: · Zbl 0678.62010
· doi:10.1093/biomet/76.3.604
· http://links.jstor.org/sici?sici=00063444%28198909%2976%3A3%3C604%3ANPFOPO%3E2.0.CO%3B2E&origin=euclid 
[35] 
such as those described by Hall (1988). But why not replace them altogether with more informative tools? The bootstrap affords a unique opportunity for obtaining a large amount of information very simply. The process of setting confidence intervals merely picks two points off a bootstrap histogram, ignoring much relevant information about shape and other important features. ”Confidence pictures” (e.g., Hall, 1992, Appendix III), essentially smoothed and transformed bootstrap histograms, are one alternative to confidence intervals. Graphics such as these provide a simple but powerful way to convey information lost in numerical summaries. The opportunities offered by dy namic graphics are also attractive, particularly when confidence information needs to be passed to a lay audience. (Consider, e.g., the need to provide information about the errors associated with predictions from opinion polls.) Bootstrap methods and new graphical way s of presenting information offer, together, exciting prospects for conveying information about uncertainty. 
[36] 
jackknifeafterbootstrap plot for t (Efron, 1992). The ingenious idea that underlies this is that we can get the effect of bootstrapping the reduced data set y1 yj1 yj+1 yn by considering only those bootstrap samples in which yj did not appear. The horizontal dotted lines are quantiles of t t for all 999 bootstrap replicates, while the solid lines join the corresponding quantiles for the subsets of bootstrap replicates in which each of the 20 observations did not appear. The xaxis shows empirical influence values lj which measure the effect on t of putting more mass on each of the observations separately. If F represents the empirical distribution function of the data, which puts mass n1 on each of the observations y1 yn and t F is the corresponding statistic, we can write of Davison and Hinkley (1996). We have developed a library of bootstrap functions in SPLUS which facilitates this ty pe of analysis. The library may be obtained by anony mous ftp to markov.stats.ox.ac.uk and retrieving the file pub/canty/bootlib.sh.Z. 
[37] 
ABC, use bootstrap calibration directly on the crude percentilebased procedures these methods refine, and which seem currently favored in published applications of the bootstrap, as any literature search confirms. In doing so, we retain the desirable properties of these basic procedures (stability of length and endpoints, invariance under parametrization etc.) yet improve their coverage accuracy. The price is one of great computational expense, although, as is demonstrated by Lee and Young (1995), there are approximations which can bring such bootstrap iteration within the reach of even a modest computational budget. An advantage of this solution lies in its simplicity: there is no need to explain the mechanics of the method, in the way that is done for the BCa and ABC methods in Sections 24 of DiCiccio and Efron’s paper. Which solution is best? To answer this requires a careful analysis of what we believe the bootstrap methodology to be. Our view is that willingness to use extensive computation to extract information from a data sample, by simulation or resampling, is quite fundamental. In other words, in comparing different methods, computational expense should not be a factor. All things being equal, we naturally look for computational efficiency, but things are hardly ever equal. How do the two solutions, that provided by DiCiccio and Efron and that involving the iterated percentile bootstrap, compare? There are two concerns here, theoretical performance and empirical performance, and the two might conflict. We demonstrate by considering the simple problem of constructing a twosided nonparametric bootstrap confidence interval for a scalar population mean. 
[38] 
and Beran (1987). The calibration method of Loh (1987) corresponds to the method of Beran (1987) when applied to a bootstrap confidence interval. For the confidence interval problem the method of Hall (1986) amounts to making an additive adjustment, estimated by the bootstrap, to the endpoints of the confidence interval, while the method of Beran (1987) amounts to making an additive adjustment, again estimated by bootstrapping, to the nominal coverage level of the bootstrap interval. The method of calibration described by DiCiccio and Efron in Section 7 of their paper is a subtle variation on the latter procedure, and one which should be used with care. DiCiccio and Efron use a method in which the bootstrap is used to calibrate separately the nominal levels of the lower and upper limits of the interval, rather than the overall nominal level. Theoretical and empirical evidence which we shall present elsewhere leads to the conclusion that, all things being taken into consideration, preference should be shown to methods which adjust nominal coverage, rather than the interval endpoints. We shall therefore focus on the question of how to calibrate the nominal coverage of a bootstrap confidence interval. The major difference between the two approaches to adjusting nominal coverage is that the method as illustrated by DiCiccio and Efron is only effective in reducing coverage error of the twosided interval to order n2 when the onesided coveragecorrected interval achieves a coverage error of order n3/2, as is the case with the ABC interval, but not the percentile interval. The effect of bootstrap calibration on the coverage error of onesided intervals is discussed by Hall and Martin (1988) and by Martin (1990), who show that bootstrap coverage correction produces improvements in coverage accuracy of order n1/2, therefore reducing coverage error from order n1/2 to order n1 for percentile intervals, but from order n1 to order n3/2 for the ABC interval. If the onesided corrected interval has coverage error of order n3/2, then separate correction of the upper and lower limits gives a twosided interval with coverage error of order n2, due to the fact that the order n3/2 term involves an even poly nomial. With the percentile interval, the coverage error, of order n1, of the coveragecorrected onesided interval ty pically involves an odd poly nomial, and terms of that order will not cancel when determining the coverage error of the twosided interval, which remains of order n1. On the face of it, therefore, we should be wary of the calibration method described by DiCiccio and Efron, although the problems with it do not arise with the ABC interval. 
[39] 
cedure: see also Martin (1990). Application of these methods to the intervals under consideration here allows closer examination of coverage error. Table 1 gives information on the theoretical leading terms in expansions of the coverage error of the percentile interval (denoted IP), iterated percentile interval (denoted IPITa and IPITb), ABC interval (denoted IABC) and iterated ABC interval (denoted by IABCITa and IABCITb). Figures refer to twosided intervals of nominal coverage 90% and are shown for the two methods of nominal coverage calibration, for each of four underlying distributions. The intervals IPITa and IABCITa calibrate the overall nominal coverage, while the other two iterated intervals use calibration in the way discussed by DiCiccio and Efron. What is immediately obvious from the table is that the order of coverage error only tells part of the story. Compare the coefficients of n1 for the interval IPITb with the coefficients of n2 for the other iterated intervals. However, if we focus on those intervals that ensure a coverage error of order n2, it appears that the two ty pes of iterated ABC interval are not significantly different, but that the iterated percentile interval has a leading error term consistently and significantly smaller than that of the ABC method. This same conclusion is true for any nominal coverage in the range 0.90.99. 
[40] 
Daniels, H. E. and Young, G. A. (1991). Saddlepoint approximation for the Studentized mean, with an application to the bootstrap. Biometrika 78 169179. JSTOR:
· doi:10.1093/biomet/78.1.169
· http://links.jstor.org/sici?sici=00063444%28199103%2978%3A1%3C169%3ASAFTSM%3E2.0.CO%3B2S&origin=euclid 
[41] 
Davison, A. C. and Hinkley, D. V. (1988). Saddlepoint approximations in resampling methods. Biometrika 75 417431. JSTOR: · Zbl 0651.62018
· doi:10.1093/biomet/75.3.417
· http://links.jstor.org/sici?sici=00063444%28198809%2975%3A3%3C417%3ASAIRM%3E2.0.CO%3B2%23&origin=euclid 
[42] 
Davison, A. C. and Hinkley, D. V. (1996). Bootstrap Methods and Their Application. Cambridge Univ. Press. · Zbl 0886.62001 
[43] 
Davison, A. C., Hinkley, D. V. and Worton, B. J. (1992). Bootstrap likelihoods. Biometrika 79 113130. JSTOR: · Zbl 0753.62026
· doi:10.1093/biomet/79.1.113
· http://links.jstor.org/sici?sici=00063444%28199203%2979%3A1%3C113%3ABL%3E2.0.CO%3B23&origin=euclid 
[44] 
DiCiccio, T. J., Martin, M. A. and Young, G. A. (1992). Fast and accurate approximate double bootstrap confidence intervals. Biometrika 79 285295. JSTOR: · Zbl 0751.62013
· doi:10.1093/biomet/79.2.285
· http://links.jstor.org/sici?sici=00063444%28199206%2979%3A2%3C285%3AFAAADB%3E2.0.CO%3B2N&origin=euclid 
[45] 
DiCiccio, T. J., Martin, M. A. and Young, G. A. (1993). Analy tical approximations for iterated bootstrap confidence intervals. Statistics and Computing 2 161171. 
[46] 
Efron, B. (1992). Jackknifeafterbootstrap standard errors and influence functions (with discussion). J. Roy Statist. Soc. Ser. B 54 83127. JSTOR: · Zbl 0782.62051
· http://links.jstor.org/sici?sici=00359246%281992%2954%3A1%3C83%3AJSEAIF%3E2.0.CO%3B2B&origin=euclid 
[47] 
Efron, B. and LePage, R. (1992). Introduction to bootstrap. In Exploring the Limits of Bootstrap (R. LePage and L. Billard, eds.) 310. Wiley, New York. · Zbl 0835.62038 
[48] 
Gleser, L. J. and Hwang, J. T. (1987). The nonexistence of 100 1 percent confidence sets of finite expected diameter in errorsinvariables and related models. Ann. Statist. 15 13511362. · Zbl 0638.62035
· doi:10.1214/aos/1176350597

[49] 
Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer, New York. · Zbl 0744.62026 
[50] 
Hall, P. and Jing, BY. (1995). Uniform coverage bounds for confidence intervals and BerryEsseen theorems for Edgeworth expansion. Ann. of Statist. 23 363375. · Zbl 0824.62043
· doi:10.1214/aos/1176324525

[51] 
Jensen, J. L. (1986). Similar tests and the standardized log likelihood ratio statistic. Biometrika 73 567572. JSTOR: · Zbl 0608.62021
· doi:10.1093/biomet/73.3.567
· http://links.jstor.org/sici?sici=00063444%28198612%2973%3A3%3C567%3ASTATSL%3E2.0.CO%3B25&origin=euclid 
[52] 
Lee, S. M. S. and Young, G. A. (1995). Asy mptotic iterated bootstrap confidence intervals. Ann. Statist. 23 13011330. Lee, S. M. S. and Young, G. A. (1996a). Sequential iterated bootstrap confidence intervals. J. Roy. Statist. Soc. Ser. B 58 235252. Lee, S. M. S. and Young, G. A. (1996b). Asy mptotics and resampling methods. Computing Science and Statistics. To appear. Lu, K. L. and Berger, J. O. (1989a). Estimation of normal means: frequentist estimation of loss. Ann. Statist. 17 890906. Lu, K. L. and Berger, J. O. (1989b). Estimated confidence for multivariate normal mean confidence set. J. Statist. Plann. Inference 23 120. · Zbl 0838.62034
· doi:10.1214/aos/1176324710

[53] 
Martin, M. A. (1990). On bootstrap iteration for coverage correction in confidence intervals. J. Amer. Statist. Assoc. 85 11051118. JSTOR: · Zbl 0736.62040
· doi:10.2307/2289608
· http://links.jstor.org/sici?sici=01621459%28199012%2985%3A412%3C1105%3AOBIFCC%3E2.0.CO%3B2V&origin=euclid 
[54] 
Owen, A. B. (1988). Empirical likelihood ratio confidence intervals for a single functional. Biometrika 75 237249. JSTOR: · Zbl 0641.62032
· doi:10.1093/biomet/75.2.237
· http://links.jstor.org/sici?sici=00063444%28198806%2975%3A2%3C237%3AELRCIF%3E2.0.CO%3B2P&origin=euclid 
[55] 
Owen, A. B. (1990). Empirical likelihood ratio confidence regions. Ann. Statist. 18 90120. · Zbl 0712.62040
· doi:10.1214/aos/1176347494
