Before knowing what Bootstrap is, we need to know what RESAMP LING means.
RESAMPLING (800)
What
procedures that describe how to economically use available data to estimate a population parameter.
Why
Why using this ?
To resample form the original data — either directly or via a fitted model — to create replicate datasets
The result can be both a more accurate estimate of
the parameter (such as taking the mean of the estimates) and
a quantification of the uncertainty of the estimate (e.g adding a confidence interval).
very easy to use, requiring little mathematical knowledge
Why might not using this ?
computationally very expensive, requiring tens, hundreds, or even thousands of resamples in order to develop a robust estimate of the population parameter.
How
What
A kind of RESAMPLING to approximate the sampling distribution (i.e. a technique to estimate the confidence interval. (ITP p560)
Does not improve your point estimates. (ITP p559)
Allows us to estimate the variance without re-collecting more data (ITP p559)
Why
When do we use Bootstrap?
Better than using CI to estimate the point estimate. It can be easy to find the Mean (point estimate) using CI, but difficult to find other true parameter i.e. θˆ like you have an estimator to estimate MEDIAN or others.
For simple problems such as the sample b average, analyzing Var(θˆ ) is not difficult. However, if θˆ is a more complicated statistic, e.g., the median, analyzing Var(θˆ ) may not be as straightforward. So, if wanting to estimate the confidence interval, you need to use Bootstrap. (ITP p559).
In a nutshell, we use bootstrapping when the estimator does not have a simple expression for the variance
Why is it difficult to provide a confidence interval for estimators such as the median?
How
There are 2 examples, which applies to the same logic. First of all, we will introduce the brute force approach which does not work in real-world. Then, that is the reason we use Bootstrap.
Example 1 (ITP p559)
1.1. Brute force approach
Imagine we have the population distribution (Which does not happen in real-world often, but let's suppose).
There are some notation :
By having the population distribution, we can generate as many samples 's as we want.
Step 1. We draw replicate datasets , . . . , from
Since we are interested in constructing the confidence interval for , we need to analyze b the mean and variance of . The true mean and the estimated mean of are
Step 2 Compute M() and b V() based on the samples.
Problem :
we only have one dataset We do not have access to , . . . , , and we do not have access to . Therefore, we are not able to approximate the variance using the above brute force simulation.
1.2 Solution : Bootstrap
Assume: We do NOT have access to , but we have one dataset
Step 1: Generate datasets , . . . , from , by sampling with replacement from .
Step 2 : Compute and based on the samples
So, at the end, we have the output — for the final answer.
(ITS) James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning: with Applications in R (Springer Texts in Statistics) (2nd ed. 2021 ed.). Springer. (Here)
( ITP ) - The best source to get the gist. Chan, S. (2021). Introduction to Probability for Data Science. Michigan Publishing Services. (Here)
Confidence Intervals?
