Chapter 19 — Model Evaluation, Tuning, and Ensemble Methods

A model is only as good as our evaluation is honest. This chapter covers cross-validation, hyperparameter tuning with grid search, learning curves, and ensemble methods that combine many models for stronger performance.

Learning Objectives

Evaluate models with k-fold cross-validation.
Tune hyperparameters with GridSearchCV.
Diagnose bias vs variance with learning curves.
Build ensembles: random forests and gradient boosting.
Compare models fairly on the same benchmark.
Avoid overfitting and data leakage.

Prerequisites / Imports

In [1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import (train_test_split, cross_val_score,
                                     GridSearchCV, learning_curve)
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

1 A Benchmark Dataset

A moderately challenging binary classification problem.

In [1]:

X, y = make_classification(n_samples=1000, n_features=12, n_informative=8, n_redundant=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
print('train:', X_train.shape, 'test:', X_test.shape)

train: (700, 12) test: (300, 12)

2 k-Fold Cross-Validation

CV estimates performance more reliably than a single split by averaging over $k$ folds.

In [1]:

tree = DecisionTreeClassifier(random_state=42)
scores = cross_val_score(tree, X_train, y_train, cv=5)
print('5-fold CV accuracies:', np.round(scores, 3))
print('mean:', round(scores.mean(), 3), 'std:', round(scores.std(), 3))

5-fold CV accuracies: [0.829 0.829 0.75  0.814 0.771]
mean: 0.799 std: 0.032

3 Grid Search for Hyperparameter Tuning

Systematically search combinations of hyperparameters and pick the best by CV.

In [1]:

param_grid = {'max_depth': [3, 5, 7, None], 'min_samples_leaf': [1, 2, 5]}
grid = GridSearchCV(DecisionTreeClassifier(random_state=42), param_grid, cv=5)
grid.fit(X_train, y_train)
print('best params:', grid.best_params_)
print('best CV score:', round(grid.best_score_, 3))
print('test accuracy:', round(accuracy_score(y_test, grid.predict(X_test)), 3))

best params: {'max_depth': None, 'min_samples_leaf': 1}
best CV score: 0.799
test accuracy: 0.817

4 Learning Curves: Bias vs Variance

Plot training and validation scores vs training-set size to diagnose under/overfitting.

In [1]:

train_sizes, train_scores, val_scores = learning_curve(
    DecisionTreeClassifier(max_depth=5, random_state=42), X_train, y_train,
    train_sizes=np.linspace(0.1, 1.0, 8), cv=5)
plt.figure(figsize=(7,4))
plt.plot(train_sizes, train_scores.mean(axis=1), 'o-', label='train')
plt.plot(train_sizes, val_scores.mean(axis=1), 'o-', label='validation')
plt.fill_between(train_sizes, val_scores.mean(axis=1)-val_scores.std(axis=1),
                 val_scores.mean(axis=1)+val_scores.std(axis=1), alpha=0.2)
plt.title('Learning curve (decision tree, depth 5)'); plt.xlabel('training samples'); plt.ylabel('accuracy')
plt.legend(); plt.show()

5 Ensembles: Random Forest

A random forest averages many decorrelated decision trees, reducing variance.

In [1]:

rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X_train, y_train)
print('random forest test accuracy:', round(accuracy_score(y_test, rf.predict(X_test)), 3))
print('mean CV accuracy:', round(cross_val_score(rf, X_train, y_train, cv=5).mean(), 3))

random forest test accuracy: 0.853
mean CV accuracy: 0.853

6 Feature Importance

Tree ensembles provide feature importances, useful for insight and feature selection.

In [1]:

imp = pd.Series(rf.feature_importances_, index=[f'f{i}' for i in range(X.shape[1])]).sort_values()
plt.figure(figsize=(7,5))
imp.plot(kind='barh')
plt.title('Random forest feature importances'); plt.show()

7 Gradient Boosting

Boosting builds trees sequentially, each correcting the previous model's errors.

In [1]:

gb = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42)
gb.fit(X_train, y_train)
print('gradient boosting test accuracy:', round(accuracy_score(y_test, gb.predict(X_test)), 3))

gradient boosting test accuracy: 0.86

Case Study: Model Bake-Off

Compare several models with 5-fold CV, pick the best, and report its test accuracy.

In [1]:

models = {
    'LogisticRegression': LogisticRegression(max_iter=5000),
    'DecisionTree': DecisionTreeClassifier(max_depth=5, random_state=42),
    'RandomForest': RandomForestClassifier(n_estimators=200, random_state=42),
    'GradientBoosting': GradientBoostingClassifier(random_state=42),
}
results = {}
for name, m in models.items():
    cv = cross_val_score(m, X_train, y_train, cv=5).mean()
    m.fit(X_train, y_train)
    test = accuracy_score(y_test, m.predict(X_test))
    results[name] = (round(cv, 3), round(test, 3))
res = pd.DataFrame(results, index=['CV accuracy', 'Test accuracy']).T
res

	CV accuracy	Test accuracy
LogisticRegression	0.693	0.690
DecisionTree	0.734	0.707
RandomForest	0.853	0.853
GradientBoosting	0.841	0.863

Exercises

Run 10-fold cross-validation on a decision tree and report mean accuracy.
Grid-search max_depth and min_samples_leaf for a decision tree.
Plot a learning curve and state whether the model overfits.
Train a random forest and compare its CV accuracy to a single tree.
List the top 5 features by random forest importance.
Train a GradientBoostingClassifier and report test accuracy.
Explain why ensembles often beat single trees.
Describe overfitting in terms of the learning-curve gap.
Why should hyperparameter tuning use CV rather than the test set?
Run the model bake-off on a different sklearn dataset.

Python Data Science: From Foundations to Applications — Chapter 19