Machine Learning · Chapter 21 of 40
Handling Missing Data
Missing values (NaN) crash most models. Options: DROP rows, IMPUTE with mean/median/mode, or use a model that handles NaN natively (like HistGradientBoosting).
Always impute after splitting train/test to avoid leakage.
Example 1 (python)
from sklearn.impute import SimpleImputer
imp = SimpleImputer(strategy='median')
X_train_i = imp.fit_transform(X_train)
X_test_i = imp.transform(X_test)Median fills NaNs.
Example 2 (python)
# Or drop rows with too many missing
df = df.dropna(thresh=len(df.columns)-2)Drop rows missing many columns.
Key points
- Missing data crashes most models.
- Impute or drop.
- Fit imputer on train only.
- Consider adding a 'was_missing' flag.
💡 Note: Adding a boolean 'was_missing' column often boosts accuracy — the missingness itself is informative.