Bootstrap Methods

BOOTSTRAP Methods

Summary. The bootstrap that we describe here is one of a class of resampling methods. Its rise to prominence came only after the publication of Efron (1979) who connected it to the jackknife. Efron and many others discovered that it had wide applicability and many possible extensions. We describe the history and sketch the research developments and applications.

Use of the bootstrap idea goes back at least to Simon (1969) who used it as a tool to teach statistics. But the properties of the bootstrap and its connection to the jackknife and other resampling methods, was not realized until Efron (1979). Similar resampling methods such as the jackknife and subsampling go back to the late 1940s and 1960s respectively (Quenouille [1956] and Tukey [1958] for the jackknife and Hartigan [1969] and McCarthy [1969] for subsampling).

In 1979 the impact that the bootstrap would have was not really appreciated and the motivation for Efron's paper was to better understand the jackknife and its properties. But over the past 30 years it has had a major impact on both theoretical and applied statistics with the applications sometimes leading the theory and vice versa. The impact of Efron's work has been so great that he was awarded with the President's Medal of Science by former President George W. Bush and Johnson and Kotz (1992) included the 1979 Annals of Statistics paper in their three volume work on breakthroughs in statistics.

After the publication of Efron's paper, Simon's interest in bootstrapping was revitalized and he and Peter Bruce formed the company Resampling Stats which publicized the methodology and provided elementary software for teaching and basic data analysis (see Simon and Bruce [1991]). The bootstrap is not simply another statistical technique but is rather a general approach to statistical inference with very broad applicability and very mild modeling assumptions.

There are now a number of excellent books that specialize in bootstrap or resampling in general. Included in this list are Efron (1982), Efron and Tibshirani (1993), Davison and Hinkley (1997), Chernick (1999) and (2007), Chernick and LaBudde (2010), Hall (1992), Manly (1997), Lunneborg (2000), Politis, Romano and Wolf (1999), Shao and Tu (1995) and Good (1998) and (2004). Many other texts devote chapters to the bootstrap including a few introductory statistics texts. There are even a couple of books that cover subcategories of bootstrapping (Westfall and Young [1993] and Lahiri [2003]).

Formally, denote by X = (X1, . . . , Xn) a random sample from X with unknown distribution F and consider a random variable T = T(X1, . . . , Xn; F) which may be as simple as T= X-m with m the expected value or integral of x dF(x) or a more complicated one such a nonparametric kernel density estimator of the density f given by T = sum( Kx - Xi /h), where h denotes the bandwidth parameter and K is a kernel function. One of the main goals in statistical inference is to determine the sampling distribution of T, namely PF (T(X, F)less than or equal x). If Fn denotes the empirical distribution of X, from the sample X, then the bootstrap version of T is given by T* = T(X1*, X2*,..., Xn*; Fn) where X* = ( X1*, X2*,..., Xn*) is a random sample from Fn. This is known as the naive bootstrap. We may also replace F by a smoothed version of Fn (called the smooth bootstrap or generalized bootstrap) or by a parametric estimate of F (i.e. a function of parameter estimates, parametric bootstrap). See Efron (1979, 1982) as introductory references, and Silverman and Young (1987) and Dudewicz (1992) for detailed coverage on the smooth bootstrap.

As it has been noted, the main goal is to estimate the sampling distribution of T. The bootstrap estimator of PF(T(X, F)less than or equal x) is given by PFn (T(X*, Fn)less than or equal x) = P*(T(X, Fn)less than or equal x) where P* is the associated probability in the bootstrap world (the distribution associated with sampling with replacement from Fn in the naive bootstrap case). Since in most cases this probability cannot be computed, Monte Carlo methods are used in order to obtain an approximation, based on B bootstrap replicates X*j , with j = 1, 2,..., B: is just number of j; such that T[X*j, Fn]less than or equal x)/B,where number refers to the cardinality of the set. Once we have approximated the statistic's distribution, other inference problems such as bias and variance estimation, confidence interval construction or hypothesis testing, etc. can be tackled. In confidence intervals, the variable of interest is usually given by T = Lh-L, where L is an unknown parameter of the distribution and Lh is the corresponding estimator. The simplest case is based on the direct approximation of the distribution of T, known as the percentile method. Several refinements such as bias-corrected percentile, percentile-t (also called bootstrap-t) or other corrections, based on Edgeworth expansions, have been proposed. For more complete coverage of bootstrap intervals, see Hall (1988, 1992), Efron and Tibshirani (1993), Chernick (2007) or Davison and Hinkley (1997).

Generally speaking, the bootstrap methodology aims to reproduce from the sample the mechanism generating the data which may be a probability distribution, a regression model, a time series, etc. Nowadays, bootstrap methods have been applied to solve different inference problems, including bandwidth selection in curve estimation (Cao, 1993), distribution calibration in empirical processes or empirical regression processes (Stute et al., 1998) or inference for incomplete data detailed below, among others.

For the sake of simplicity, we may distinguish two perspectives regarding what gets bootstrapped. First, data involved in the statistic may be directly observed, where the basic bootstrap resampling procedures introduced above can be applied. Secondly, data may exhibit a complex generating mechanism. As a special case, consider a parametric regression model yi = mL(xi)+ei, i = 1, where mL() is the regression function and ei denotes the associated ith error. For fixed design and parametric regression function, we may proceed by resampling the residuals eri = yi -mLh(xi) where Lh is a parameter estimator. Naive bootstrap samples e*i drawn from the empirical distribution of the centered residuals eri - e) are used to get the bootstrap regression model y*i = mLh(xi)+ e*i. This approach is called model-based bootstrapping.

Efron also introduced the bootstrap in this context. Each of the bootstrap samples can provide an estimate of the regression parameter(s) following the same estimation procedure that was used with the original fitted model (e.g., ordinary least squares). From all the bootstrap replicates we get a Monte Carlo approximation to the bootstrap distribution of the regression parameter(s) and this is then used to make inferences about the parameter(s) based on this approximation to the sampling distribution(s) for the parameter estimate(s).

In model-based inference, Paparoditis and Politis (2005) underscored the importance of the choice of residuals. For example, to maximize power in bootstrap-based hypothesis testing, residuals are obtained using a sequence of parameter estimators that converge to the true parameter value under both the null and alternative hypotheses.

Different modifications of this simple idea allow for adapting to random design, heteroscedastic models or situations where the regression function is not totally specified or is unknown, such as in nonparametric regression. From the first references, especially in parametric regression, by Bickel and Freedman (1981) and Freedman (1981), several advances have been introduced in this context. For the nonparametric case, see Chapter 14 in Schimek (2000). Also Shi (1991) introduced the local bootstrap which handles heteroscedasticity in parametric linear and nonparametric models.

Although the bootstrap originally started with independent sample observations, just as in the regression models described above there are extensions to dependent data: time series, spatial statistics, point processes, spatio-temporal models, and more. For example, similar to the ideas of bootstrap in regression models, given an explicit dependence structure such an autoregressive model, we may write: yi = m(yi-1 ,yi-2, ..., yi-p) and proceed by resampling from the residuals.

When an explicit parametric equation is not available, an alternative is the block bootstrap, which consists of resampling blocks of subsamples, trying to capture the dependence in the data. Bootstrap replicates obtained by these methods may not be stationary. In order to solve this problem, Politis and Romano (1994a) propose the stationary bootstrap. These bootstrap procedures can be adapted for predicting the future values of the process. An overview of bootstrap methods for estimation and prediction in time series can be found in Cao (1999). Paparoditis and Politis (2001) introduced the tapered block bootstrap. The idea of the block bootstrap has also been extended to the spatial setting (see Lahiri, 2003).

There is theoretical justification for bootstrapping time series. As an example, Politis and Romano (1994b) established convergence of certain sums of stationary time series that can facilitate bootstrap resampling. The block bootstrap was among the early proposals for time series data. While the method is very straightforward, there are associated problems like independence between blocks while maintaining the dependence structure within the block. The size of the block is a crucial quantity that should be determined to assure success in block bootstrapping.

The AR-sieve method was also introduced as a residual-based method similar to the model-based approach. Local bootstrap was also introduced but in the context of a local regression framework (nonparametric) and to account for the nonparametric model, resampling allows the empirical distribution to vary locally in the time series.

Buhlman (2002) compared different methods for time series bootstrapping. The block bootstrap is recognized as the most general and simplest generalization of the original independent resamples, but is sometimes criticized for the possible artifacts it may exhibit when blocks are linked together. Blocking can potentially introduce some dependence structure in addition to those naturally existing in the data.

The AR-sieve is less sensitive to selection of a model than it is to the block to the block length. The local bootstrap for nonparametric estimation is observed to yield slower rates of convergence. Generally, the AR-sieve is relatively advantageous among the bootstrap approaches for time series data. Early work on the sieve bootstrap are the articles Kreiss (1992) and Paparoditis and Streiberg (1992).

There are also frequency domain approaches to bootstrapping time series. See Davison and Hinkley (1997), Chernick (2007) or Lahiri (2003). Papers on this topic include Franke and Hardle (1992) and Dahlhaus and Janas (1996) among others.

Recently, the bootstrap has been introduced in more complex and complicated models. In modeling non-stationary volatility, Xu (2008) used an autoregression around a polynomial trend with stable autoregressive roots to illustrate how non-stationary volatility affects the consistency, convergence rates and asymptotic distributions of the estimators. (Westerlund and Edgerton, 2007) proposed a bootstrap test for the null hypothesis of cointegration in panel data. Dumanjug, C., Barrios, E., and Lansangan, J. (2009) developed a block bootstrap method in a spatial-temporal model.

The use of the bootstrap as a tool for calibrating the distribution of a statistic has been extended to most topics in statistical inference. Not trying to be exhaustive, it is worth considering the emergence of bootstrap in the following fields of study: i) Incomplete data. When dealing with censored data, the empirical estimator of the distribution is replaced by the Kaplan-Meier estimator (Efron, 1981). ii) Missing information. If some observations are missing or imputed, Bootstrap estimators must be suitably adapted (Efron, 1994). iii) Hypothesis testing in regression models. Here the goal is to check whether a parametric regression model mL fits the data, the distribution of a test statistic T = D(mh+,mLh), where mh+ is a nonparametric estimator of the regression function m, D is a distance measure and mLh is the parametric estimate of mL. In this context, a broad literature can be cited, such as Hardle and Mammen (1993) or Cao and Gonzalez-Manteiga (1993). For a recent review on the topic, see Gonzalez-Manteiga and Crujeiras (2008). iv) Model selection. Early on Gong (1982) and (1986) showed how to the bootstrap could be used to determine the variability in the choice of the covariates that are part of a final regression model selected by a stepwise procedure (stepwise logistic regression in her case). This was not followed up until recently. Mahiane et al. (2010) is one of many recent applications of versions of the idea. v) Small area inference. The bootstrap has also shown a great development in finite populations (Shao and Tu, 1995), especially in recent years with the appearance of small area models. See Hall and Maiti (2006) and Lahiri (2003). vi) The bootstrap has also recently been used in learning theory and high dimensional data. As an example, see the application of bootstrapping in the regression model with functional data, Ferraty,F., Van Keilegom, I. and Vieu, P. (2009) or the bootstrap for variable choice in regression or classification models, Hall, P. Lee, E.R. and Park, B.U.(2009).

The continuing development of bootstrap methods has been motivated by the increasing progress in computational speed and efficiency. Other resampling methods include Markov Chain Monte Carlo, commonly called MCMC (see Smith and Roberts [1993], for instance) and subsampling, Hartigan (1969) and Politis, Romano and Wolf (1999).

The bootstrap's popularity rests in its relaxation of distribution assumptions that can be restrictive and its wide variety of applications as described above. We see that the development of the bootstrap evolved as follows. Efron introduced it in (1979) with some theoretical and heuristic development. Theoretical activity followed quickly with Athreya, Bickel, Freedman, Singh, Beran and Hall providing notable contributions in the early 1980s. Efron realized its practical value early on and efforts to make the scientific community aware of its potential were the Diaconis and Efron (1983) article in Scientific American and the article by Efron and Tibshirani (1986).

So by the early 1990s enough theory and successful applications had developed to lead to an explosion of papers, mostly applied and some extending the theory. The literature was so large that Chernick (1999) contains more than 1,600 references. An excellent and nearly up-to-date survey article on the bootstrap is Lahiri (2006).

For the bootstrap to "work", the bootstrap estimates must be consistent. But even when the first results on consistency of the bootstrap estimate of a mean were derived, Bickel, Freedman and others realized that there were cases where the bootstrap is inconsistent. Two notable examples are (1) the sample mean when the population distribution has an infinite variance but the ordinary sample mean appropriately normalized still converges to a stable law and (2) the maximum of a sample when the population distribution is in the domain of attraction of an extreme value distribution. These examples are covered in Chapter 9 of Chernick (2007).

These results on bootstrap inconsistency were already published in the 1980s and they led to a concern about what the real limitations of the bootstrap are. The volume edited by LePage and Billard (1992) and the monograph by Mammen (1992) address these concerns. Consistency is one requirement but what about the small sample properties? This was addressed beautifully using simulation as illustrated in Shao and Tu (1995) and Efron (1983). The work of Efron and others on small sample accuracy of bootstrap estimates of error rates in classification is summarized in Chapter 2 of Chernick (2007).

Efron's bootstrap principle states that the nonparametric bootstrap mimics sampling from a population by letting the empiric distribution Fn play the role of the unknown population distribution F and letting the bootstrap distribution Fn* play the role of Fn. This is to say in words what was described using formal mathematics earlier in this article.

Efron thought it was natural to take the size of a bootstrap sample to be n but others saw no reason why a value m

Prior to the formal introduction of the m-out-of-n bootstrap, Athreya(1987) showed that for heavy-tailed distributions a trimmed mean could converge to the population mean with the advantage that the sampling distribution of the trimmed mean has second moments. Using this idea he proved consistency of the m-out-of-n bootstrap for the mean in the infinite variance case. Other early investigations included Bretagnolle (1983) and Swanepoel (1986). Fukuchi (1994) did the same as Athreya for extremes. Both results require m and n to approach infinity at a rate that would have m/n approach 0. Politis and Romano (1993) show consistency for m-out-of-n under very mild conditions as long as m2/n approaches 0 as n tends to infinity. The m-out-of-n bootstrap has been further studied by Bickel, Gotze and van Zwet (1997) and Politis, Romano and Wolf (1999). Zelterman (1993) found a different way to modify the bootstrap to make it consistent for the extreme values. This is all summarized in Chapter 9 of Chernick (2007).

The theoretical developments from 1995 to the present have been in the area of (1) modifying the bootstrap to fix inconsistency in order to widen its applicability and (2) extending the theory to dependent situations (as previously mentioned). Lahiri (2003) is the ideal reference for a detailed account of these developments with dependent data.

Bootstrap confidence intervals have been a concern and Efron recognized early on that getting the asymptotic coverage nearly correct in small samples required more sophistication than his simple percentile method bootstrap. So the bias corrected bootstrap was developed to do that. However, in Schenker (1985) the example of variance estimation for a particular chi square population distribution showed that even the BC method had coverage problems in small samples. Efron (1987) introduced the BCa method which remedied the problem discovered by Schenker.

However, in recent years variance estimation for other examples with skewed or heavy-tailed distributions has shown all bootstrap confidence interval methods to be problematic in small samples. A large Monte Carlo investigation, Chernick and La Budde (2010), compares the coverage of various bootstrap confidence intervals for the variance estimate from a variety of population distributions when sample sizes are small.

They also provide an idea of rates of convergence by showing how the coverage improves as the sample size gets large. An interesting surprise is that in some situations for small sample sizes the lower order bootstrap work better than the higher order ones. This is because they are simpler and do not involve estimating biases and acceleration constants which depend on third order moments of the distribution. For the lognormal population they show that at the small sample sizes (20 to 100) the coverage error is shockingly high for all methods.

References

Andrews, D. (2000). Inconsistency of the bootstrap when a parameter is on the boundary of the parameter space. Econometrica 68, 399-405.

Athreya, K. B. (1987). Bootstrap estimation of the mean in the infinite variance case. Ann. Statist. 15, 724-731.

Beran, R. (1997). Diagnosing bootstrap success. Annals of the Institute of Statistical Mathematics, 49, 1-24.

Bickel, P. J. and Freedman, D.A. (1981). Some asymptotic theory for the bootstrap. Ann. Statist. 6, 1196-1217.

Bickel, P. J., Gotze, F. and van Zwet, W. R. (1997). Resampling fewer than n observations: gains, loses, and remedies for losses. Statist. Sin. 7, 1-32.

Bretagnolle, J. (1983). Lois limites du bootstap de certaines fonctionelles. Ann. Inst. Henri Poincare Sec. B 19, 281, 296.

Buhlman, P. (1997). Sieve bootstrap for time series, Bernoulli, 3, 123-148.

Buhlman, P. (2002). Bootstrap for Time Series, Statistical Science, 17, 52-72.

Cao, R. (1993). Bootstrapping the mean integrated squared error. Journal of Multivariate Analysis, 45, 137-160.

Cao, R. (1999). An overview of bootstrap methods for estimating and predicting in time series. Test 8, 95-116.

Cao, R. and Gonzalez-Manteiga, W. (1993). Bootstrap methods in regression smoothing. Journal of Nonparametric Statistics 2, 379-388.

Chernick, M. R. (1999). Bootstrap Methods: A Practitioners Guide. Wiley, New York.

Chernick, M. R. (2007). Bootstrap Methods: A Guide for Practitioners and Researchers, 2nd Edition Wiley, Hoboken.

Chernick, M. R. and LaBudde, R. (2010). Revisiting qualms about bootstrap confidence intervals. Am. J. Math. Manag. Sci. Am. J. Math. Manag. Sci. 29, 437-456.

Chernick, M. R. and LaBudde, R. (2011). An Introduction to the Bootstrap with Applications in R Wiley, Hoboken.

Dahlhaus, R. and Janas, D. (1996). A frequency domain bootstrap for ratio statistics in time series analysis. Ann. Statist. 24, 1914-1933.

Davison, A. C. and Hinkley, D. V. (1997). Bootstrap Methods and Their Application. Cambridge University Press, Cambridge.

Diaconis, P. and Efron, B. (1983). Computer-intensive methods in statistics. Sci. Am. 248, 116-130.

Dudewicz. E. J. (1992). The generalized bootstrap. In Bootstrapping and Related Techniques, Proceedings Trier FRG (K.-H. Jockel, G. Rothe and W. Sendler, editors). Lecture Notes in Economics and Mathematical Systems 376, 31-37, Springer-Verlag, Berlin.

Dumanjug, C., Barrios, E., and Lansangan, J. (2009) Bootstrap procedures in a spatial-temporal model, Journal of Statistical Computation and Simulation to appear.

Efron, B. (1979). Bootstrap methods: another look at the jackknife. Ann. Statist. 7, 1-26.

Efron, B. (1981). Censored data and the bootstrap. J. Am. Statist. Assoc. 74, 312-319.

Efron, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans. SIAM, Philadelphia.

Efron, B. (1983). Estimating the error rate of a prediction rule: improvements on cross-validation. J. Am. Statist. Assoc. 78, 316-331.

Efron, B. (1987). Better bootstrap confidence intervals (with discussion). J. Am. Statist. Assoc. 82, 316-331.

Efron, B. (1994). Missing data, imputation and the bootstrap. J. Am. Statist. Assoc. 89, 463-479.

Efron, B. (2000). The bootstrap and modern statistics. J. Am. Statist. Assoc. 95, 1293-1296.

Efron, B. and Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals and other measures of statistical accuracy. Statist. Sci. 1, 54-77.

Efron, B. and Tibshirani, R.J. (1993). An Introduction to the Bootstrap. Chapman and Hall, New York.

Ferraty, F., Van Keilegom, I. and Vieu, P. (2009). On the validity of the bootstrap in nonparametric functional regression. Scandinavian Journal of Statistics (to appear).

Franke and Hardle (1992). On bootstrapping kernel spectral estimates. Ann. Statist. 10, 121- 145.

Freedman, D.A. (1981) Bootstrapping regression models, Ann. Statist. 6, 1218- 1228.

Fukuchi, J. I. (1994). Bootstrapping extremes of random variables. Ph.D. Dissertation, Iowa State University, Ames.

Gong, G. (1982). Some ideas to using the bootstrap in assessing model variability in regression. Proc. Comput. Sci. Statist. 24, 169-173.

Gong, G. (1986). Cross-validation, the jackknife and the bootstrap: Excess error in forward logistic regression. J. Am. Statist. Assoc. 81, 108-113.

Gonzalez-Manteiga, W. and Cao, R. (1993) Testing the hypothesis of a general linear model using nonparametric regression estimation. Test 2, 223-249.

Gonzalez-Manteiga, W. and Crujeiras, R.M. (2008). A review on goodness-of-fit tests for regression models. Pyrenees International Workshop on Statistics, Probability and OperationsResearch: SPO 2007, 21-59.

Good, P. (1998). Resampling Methods: A Practical Guide to Data Analysis. Birkhauser, Boston.

Good, P. (2004). Permutation, Parametric, and Bootstrap Tests of Hypotheses. 3rd Edition. Springer-Verlag, New York.

Hall, P. (1988). Theoretical comparison of bootstrap confidence intervals. Ann. Statist. 16, 927-953.

Hall, P. (1992). The Bootstrap and Edgeworth Expansion. Springer-Verlag, New York.

Hall, P. and Maiti, T. (2006). On parametric bootstrap methods for small area prediction. Journal of the Royal Statistical Society. Series B 68, 221-238.

Hall, P. Lee, E.R. and Park, B.U. (2009) Bootstrap-based penalty choice for the Lasso achieving oracle performance. Statistica Sinica, 19, 449-471.

Hardle, W. and Mammen, E. (1993) Comparing nonparametric versus parametric regression fits. Ann. Statist. 21, 1926-1947.

Hartigan, J. A. (1969). Using subsample values as typical values. J. Am. Statist. Assoc. 64, 1303-1317.

Kotz, S. and Johnson, N. L. (1992). Breakthroughs in Statistics Volume II: Methodology and Distribution 565-593. Springer-Verlag, New York.

Kreiss, J.P. (1992): Bootstrap procedures for AR (infinity) processes, in "Bootstrapping and related techniques", Springer Verlag 376, 107-113.

Lahiri, P. (2003). On the impact of bootstrap in survey sampling and small-area estimation. Statistical Science, 18, 199-210.

Lahiri, S. N. (2003). Resampling Methods for Dependent Data. Springer-Verlag, New York.

Lahiri, S. N. (2006). Bootstrap Methods: A Review. In Frontiers in Statistics (J. Fan and H.L. Koul, editors) 231-265, Imperial College Press, London.

LePage, R. and Billard, L. (editors) (1992). Exploring the Limits of Bootstrap. Wiley, New York.

Lunneborg, C. E. (2000). Data Analysis by Resampling: Concepts and Applications. Brooks/Cole, Pacific Grove.

Mahiane, S. G., Nguema, E.-P. Ndong, Pretorius, C., and Auvert, B. (2010). Mathematical models for coinfection by two sexually transmitted agents: the human immunodeficiency virus and herpes simplex virus type 2 case. Appl. Statist 59, 547-572.

Mammen, E. (1992). When Does the Bootstrap Work? Asymptotic Results and Simulations. Springer-Verlag, Heidelberg.

Manly, B. F. J. (1997). Randomization, Bootstrap and Monte Carlo Methods in Biology. 2nd Editon, Chapman and Hall, London.

McCarthy, P. J. (1969). Pseudo-replication: Half-samples. Int. Statist. Rev. 37, 239-263.

Paparoditis, E. and Streiberg,B. (1992). Order identification statistics in stationary autoregressive moving-average models: vector autocorrelations and the bootstrap. J. of Time Ser. Anal. 13, 415-434.

Paproditis, E. and Politis, D. N. (2001). Tapered block bootstrap. Biometrika 88, 1105-1119.

Paparoditis, E. and Politis, D. (2005). Bootstrap hypothesis testing in regression models, Statistics and Probability Letters, 74: 356-365.

Politis, D.N. and Romano J. P. (1993). Estimating the distribution of a studentized statistic by subsampling. Bull. Intern. Statist. Inst. 49th Session 2, 315-316.

Politis, D.N. and Romano J. P. (1994a). The stationary bootstrap. J. Am. Statist. Assoc. 89, 1303-1313.

Politis, D.N. and Romano, J. (1994b). Limit theorems for weakly dependent Hilbert Space valued random variables with application to the stationary bootstrap, Statist. Sin. 4: 461-476.

Politis, D. N., Romano, J. P., and Wolf, M. (1999). Subsampling. Springer-Verlag, Berlin.

Quenouille, M. H. (1956). Notes on bias in estimation. Biometrika 43: 353-360.

Schenker, N. (1985). Qualms about bootstrap confidence intervals. J. Am. Statist. Assoc. 80, 360-361.

Schimek, M.G. (2000) Smoothing and Regression. Wiley, Hoboken.

Shao, J. and Tu, D. (1995). The Jackknife and Bootstrap. Springer-Verlag, New York.

Shi, X. (1991). Some asymptotic results for jackknifing the sample quantile. Ann. Statist. 19, 496-503.

Silverman, B.W. and G.A. Young (1987) The bootstrap: smooth or not to smooth? Biometrika, 74, 469-479.

Simon, J. L. (1969). Basic Research Methods in Social Science. Random House, New York.

Simon, J. L. and Bruce, P. (1991). Resampling: A tool for everyday statistical work. Chance 4, 22-32.

Smith, A. F. M. and Roberts, G. O. (1993) Bayesian computation via the Gibbs sampler and related Markov Chain Monte Carlo methods. Journal of the Royal Statistical Society. Series B. 55, 3-23.

Stute, W., Gonzlez-Manteiga, W. and Presedo-Quindimil, M.A. (1998) Bootstrap approximations in model checks for regression. J. Am. Statist. Assoc. 93, 141-149.

Swanepoel, J. W. H. (1986). A note on proving the (modified) bootstrap works. Commun. Statist. Theory Methods 15, 1399-1415.

Tukey, J. W. (1958). Bias and confidence in not quite large samples (abstract). Ann. Math. Statist. 29, 614.

Westerland, J. and Edgerton, D. (2007). A panel bootstrap cointegration test, Economic Letters, 97, 185-190.

Westfall, P. and Young, S. S. (1993) Resampling-Based Multiple Testing: Examples and Methods for p-value Adjustment. Wiley, New York.

Xu, K. (2008). Bootstrapping autoregression under nonstationary volatility, Econometrics Journal, 11: 1-26.

Zelterman. D.(1993). A semiparametric bootstrap technique for simulating extreme order statistics. J. Am. Statist. Assoc. 88, 477-485.

Michael R. Chernick, W. Gonzalez-Manteiga, R.M. Crujeiras and Erniel B. Barrios

Based on an article from Lovric, Miodrag (2011), International Encyclopedia of Statistical Science. Heidelberg: Springer Science+Business Media, LLC