Definition
Ensembles aggregate predictions from several models. Bagging (e.g., random forests) reduces variance; boosting (e.g., XGBoost, LightGBM) reduces bias by focusing on hard examples.
They dominate many tabular ML competitions and production scoring systems.
In simple terms
Ask a panel of experts instead of one person, diverse opinions often average out individual mistakes.
Where you see it
- Credit risk and fraud models using gradient boosting.
- Kaggle-winning stacks of diverse models.
How it works
1.Train diverse models
Different data samples, features, or algorithms.
2.Aggregate
Vote, average, or learn a meta-model.
3.Deploy
Serve the ensemble or distill into a smaller model.
Why it matters
- For structured data, ensembles often beat single deep nets with less tuning drama.
Often confused
Ensembles are always too slow for production.
Tree ensembles are often fast; LLM ensembles are expensive, cost depends on the base model.