Ensemble Learning
Ensembles combine multiple learners to improve robustness or accuracy by exploiting diversity among their errors. The topic is split into focused lessons so definitions, implementation decisions, diagnostics and common failure modes can be learned separately.
Learn the mechanism one decision at a time
Work through the lessons in order if the topic is new. If you already know the basics, open the specific leaf lesson that matches the operation, diagnostic or failure mode you need.
1Definition→
2Mechanism→
3Example→
4Diagnostic→
5Decision
BaggingFits learners independently on resampled data/features and averages or votes. Random Forest is the canonical example. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
02BoostingFits learners sequentially so later stages focus on residual/error structure left by earlier stages. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
03StackingTrains heterogeneous base models and a meta-model on out-of-fold base predictions to learn how to combine them. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
04Voting and averagingSimple hard voting, soft voting and weighted averaging can be effective when individual models are complementary. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
05Leakage-safe stackingMeta-model training must use predictions generated out-of-fold; in-sample base predictions leak target fit into the stacker. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.