Machine Learning · Chapter 34 of 40
Ensemble Methods
ENSEMBLES combine many models to beat any single one. Popular families: BAGGING (Random Forest), BOOSTING (XGBoost, LightGBM, CatBoost), STACKING (meta-model over base models).
Boosting is the reigning champion of tabular ML competitions.
Example 1 (python)
from sklearn.ensemble import GradientBoostingClassifier
m = GradientBoostingClassifier(n_estimators=200, learning_rate=0.05).fit(X_train, y_train)Sequential boosting.
Example 2 (python)
# XGBoost is faster and often more accurate
# import xgboost as xgb
# m = xgb.XGBClassifier()The industry favourite.
Key points
- Combine many models to reduce error.
- Bagging: parallel trees (RF).
- Boosting: sequential correction (XGB).
- Stacking: model of models.
💡 Note: XGBoost, LightGBM and CatBoost win most Kaggle tabular competitions — always try one as a strong baseline.