Ensemble method

• It is the creation of a better classifier from a collection of weaker classifiers. (團結就是力量)

• They can make the model more robust (unlikely to be influenced by small changes in the training data).
• Boosting tries to achieve better accuracy.
• Bagging tries to reduce variance and prevent overfitting.

• You can think of Boosting is an advanced version of Bagging.
There are a few of ensemble methods learnt, namely Bagging, Boosting and
🌴
Random Forest
Assumption :
• The individual classifiers are moderately (> 50%) accurate.
• Individual classifiers are created independently.
• Pooling the results of the each classifier reduces the variance of the overall classification.
• Decision trees work well as the individual classifiers
• Disadvantage: model produced is not as easy to interpret as a single tree.

Bagging (Bootstrap Aggregation)

• Bagging is a method to decrease variance.

• Aims to create several subsets of data from training sample chosen randomly with replacement. Each collection of subset data is used to train their decision trees. [indiaMag]

• It works well for high-variance machine learning algorithm, typically decision trees.

The way it works :

For example, in the samples drawn from set 0-L (we called Replicate here), some of the observation can be happened more than once. (Sample with Replacement)

Step 2: Construct a single classifier for each replicate Step 3: Combine the classifiers by taking a Average of all the predictions from different trees.

Boosting

• Learners are learned sequentially with early learners fitting simple models to the data and then analysing data for errors. Consecutive trees (random sample) are fit and at every step, the goal is to improve the accuracy from the prior tree.

• If an observation was classified incorrectly, it tries to increase the weight of this observation and vice versa.

How to perform Boosting

• Assign equal weights to each point in training set; fit basic tree.

• Repeat n iterations:
• Update weights of misclassified items and normalise;
• update tree,
• building on current tree.

• Output the final classifier as weighted sum of votes from each tree.

1. Initialise the dataset and assign equal weight to each of the data point.

2. Provide this as input to the model and identify the wrongly classified data points.

3. Increase the weight of the wrongly classified data points.

4. if (got required results)   → Goto step 5
else   → Goto step 2

5. End

Here tells the difference between those two : This link is very useful

1. How we sample for # of replicate

In the case of Bagging, any element has the same probability (1 / # of dataset ) to appear in a new data set. However, for Boosting the observations are weighted and therefore some of them will take part in the new sets more often.

2. How we model

The training stage of the two models is different; parallel for Bagging (i.e., each model is built independently) and sequential for Boosting.

It is sequential in a sense that each classifier is trained on data, taking into account the previous classifiers’ success.

After each training step., the weights are redistributed. Misclassified data (the results of FP and TN) increases its weights to emphasise the most difficult cases. In this way, subsequent learners will prioritise on them during their training.

3. How we produce the result

The way they produce the result is different: Average for Bagging and weighted average for Bagging

In Bagging the result is obtained by averaging the responses of the N learners (or majority vote).

However, Boosting assigns a second set of weights, this time for the N classifiers, in order to take a weighted average of their estimates.

The way Boosting training learns is via evaluation. A learner with good a classification result on the training data will be assigned a higher weight than a poor one.

That's why the Boosting algorithm allocates weights to each resulting model.

FAQ

Math

Reference

Extra Resource