Chapter 5 — Basic Practice

Until now we have looked at learning algorithms one at a time, mostly in their clean, mathematical form. Real projects are messier: the data arrives as raw logs, some values are missing, and you have to make dozens of practical choices before you can write model.fit(X, y). This chapter is about those choices — the everyday craft of applied machine learning.

In this chapter you will learn:

• How to turn raw data into features a model can use (one-hot encoding, binning, scaling)
• How to pick a reasonable learning algorithm for your problem
• Why you need three separate datasets: train, validation, and test
• What underfitting and overfitting look like, and how to find the "sweet spot"
• How regularization (L1 and L2) keeps models from overfitting
• How to measure a model's quality with confusion matrices, precision/recall, and cross-validation
• How to tune hyperparameters with grid search and random search

5.1 Feature Engineering

When someone hands you five years of user-interaction logs and asks "will this customer stay?", you cannot just pour the logs into a library. First you have to build a dataset: a table where each row is one example and each column is a feature — a number that describes that example.

Feature engineering is the art of transforming raw data into that table of informative features. A feature is informative (it has high predictive power) when it genuinely helps predict the label. For predicting whether a user keeps using an app, "average session length" is informative; "user's shoe size" probably is not.

A model has low bias when it predicts the training labels well. Good features are what let a model achieve low bias without cheating. The rest of this section walks through the most common feature-engineering moves.

One-Hot Encoding

Most learning algorithms want numbers, not words. If you have a categorical feature like color with values red / yellow / green, you need to convert it.

The tempting shortcut is to map red→1, yellow→2, green→3. Don't. That invents a fake ordering ("green is three times red") and most algorithms will take that order seriously and overfit to it.

Instead, split the one categorical column into one 0/1 column per category — a 1 says "this example is red", 0 says it is not:

red    -> [1, 0, 0]
yellow -> [0, 1, 0]
green  -> [0, 0, 1]

This is one-hot encoding. You add columns, but you avoid lying to the model about an order that does not exist.

A close cousin is a feature cross: combine two categorical features into one so the model can learn a rule that depends on their combination (for example, "red on a weekend" might behave differently from "red on a weekday"). You make it by concatenating the two values into a new category, then one-hot encoding that new category.

Everyday analogy: Think of a light-switch panel where each color gets its own on/off switch, instead of one dial with the numbers 1–3 painted on it.

%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# A tiny dataset: a few customers with a categorical 'color' and two numeric features.
df = pd.DataFrame({
    "color": ["red", "yellow", "green", "red", "green", "yellow"],
    "age":   [22, 31, 45, 19, 55, 38],
    "spend": [120, 80, 300, 60, 500, 200],
})
print("Before (raw data, with a categorical column):")
print(df)
print()

# One-hot encoding: turn 'color' into one binary column per category.
# BAD idea: mapping red=1, yellow=2, green=3 would invent a fake order.
# GOOD idea: give each color its own 0/1 column.
df_oh = pd.get_dummies(df, columns=["color"], prefix="is")
print("After one-hot encoding:")
print(df_oh)
print()

# A feature cross: combine 'color' with a 'is_weekend' flag so the model can
# learn rules that depend on the COMBINATION, not just each piece alone.
df2 = pd.DataFrame({
    "color":      ["red", "yellow", "green", "red", "green"],
    "is_weekend": [1, 0, 1, 0, 1],
})
df2["color_x_weekend"] = df2["color"] + "_" + df2["is_weekend"].astype(str)
crossed = pd.get_dummies(df2, columns=["color_x_weekend"], prefix="cross")
print("Feature cross: a new 0/1 column for each (color, weekend) combination:")
print(crossed)

Before (raw data, with a categorical column):
    color  age  spend
0     red   22    120
1  yellow   31     80
2   green   45    300
3     red   19     60
4   green   55    500
5  yellow   38    200

After one-hot encoding:
   age  spend  is_green  is_red  is_yellow
0   22    120     False    True      False
1   31     80     False   False       True
2   45    300      True   False      False
3   19     60     False    True      False
4   55    500      True   False      False
5   38    200     False   False       True

Feature cross: a new 0/1 column for each (color, weekend) combination:
    color  is_weekend  cross_green_1  cross_red_0  cross_red_1  cross_yellow_0
0     red           1          False        False         True           False
1  yellow           0          False        False        False            True
2   green           1           True        False        False           False
3     red           0          False         True        False           False
4   green           1           True        False        False           False

Binning

Sometimes you want to go the other way: you have a continuous number but you would rather treat it as a few categories. This is binning (or bucketing): chop the range into bins and record which bin each value falls into.

For example, instead of storing exact age as one number, you could create bins "0–5", "6–10", "11–15", and so on. Each becomes its own 0/1 indicator feature.

Why bother? It gives the model a hint that inside a bin the exact value does not matter, which can let it learn the same pattern from fewer examples.

Everyday analogy: A wine menu that groups bottles into "under $20", "$20–50", and "over $50" instead of listing every exact price — coarse buckets are often all you need.

# Binning (bucketing): turn a continuous 'age' into a few categorical bins.
ages = pd.DataFrame({"age": [3, 7, 13, 22, 31, 45, 19, 55, 38, 60]})

# Define bin edges and human-readable labels.
bins   = [0, 5, 10, 15, 25, 40, 100]
labels = ["0-5", "6-10", "11-15", "16-25", "26-40", "41+"]

ages["age_bin"] = pd.cut(ages["age"], bins=bins, labels=labels, right=True)
print("Age with its bin label:")
print(ages)
print()

# One-hot encode the bins so a learning algorithm gets numeric 0/1 features.
age_oh = pd.get_dummies(ages, columns=["age_bin"], prefix="age")
print("Binned age turned into 0/1 indicator features:")
print(age_oh)

Age with its bin label:
   age age_bin
0    3     0-5
1    7    6-10
2   13   11-15
3   22   16-25
4   31   26-40
5   45     41+
6   19   16-25
7   55     41+
8   38   26-40
9   60     41+

Binned age turned into 0/1 indicator features:
   age  age_0-5  age_6-10  age_11-15  age_16-25  age_26-40  age_41+
0    3     True     False      False      False      False    False
1    7    False      True      False      False      False    False
2   13    False     False       True      False      False    False
3   22    False     False      False       True      False    False
4   31    False     False      False      False       True    False
5   45    False     False      False      False      False     True
6   19    False     False      False       True      False    False
7   55    False     False      False      False      False     True
8   38    False     False      False      False       True    False
9   60    False     False      False      False      False     True

Normalization and Standardization

Numeric features often live on wildly different scales: age might range 0–100 while income ranges 0–1,000,000. Many algorithms train faster and more stably when all features sit in a similar small range. Two standard recipes:

• Normalization (min-max scaling): squeeze every feature into [0, 1].
  • x_norm = (x - min) / (max - min)
  • Gives a fixed bounded range, but a single outlier can crush the other values into a tiny slice.
• Standardization (z-score): shift to mean 0 and standard deviation 1.
  • x_std = (x - mean) / std
  • Handles outliers more gracefully and is the default choice for most algorithms, especially unsupervised ones.

Which to use? There is no universal answer. A common rule of thumb: prefer standardization for data near a bell curve or with outliers; use normalization otherwise. When in doubt and you have time, try both.

Everyday analogy: Normalization is like grading every test onto a 0–100 scale; standardization is like reporting each score as "how many standard deviations above the class average." Both put different things on a common footing.

from sklearn.preprocessing import MinMaxScaler

# Reuse the numeric columns from the one-hot encoded table (cell above).
numeric = df_oh[["age", "spend"]].copy()

print("Before scaling (note the very different ranges):")
print(numeric.describe().loc[["min", "max", "mean", "std"]].round(2))
print()

# 1) Normalization (min-max): squeeze every feature into [0, 1].
mm = MinMaxScaler()
normed = pd.DataFrame(mm.fit_transform(numeric), columns=numeric.columns)
print("After MinMaxScaler (everything in [0, 1]):")
print(normed.describe().loc[["min", "max", "mean"]].round(2))
print()

# 2) Standardization (z-score): mean 0, std 1.
scaler = StandardScaler()
scaled = pd.DataFrame(scaler.fit_transform(numeric), columns=numeric.columns)
print("After StandardScaler (mean ~ 0, std ~ 1):")
print(scaled.describe().loc[["min", "max", "mean", "std"]].round(2))

Before scaling (note the very different ranges):
        age   spend
min   19.00   60.00
max   55.00  500.00
mean  35.00  210.00
std   13.78  167.21

After MinMaxScaler (everything in [0, 1]):
       age  spend
min   0.00   0.00
max   1.00   1.00
mean  0.44   0.34

After StandardScaler (mean ~ 0, std ~ 1):
       age  spend
min  -1.27  -0.98
max   1.59   1.90
mean  0.00  -0.00
std   1.10   1.10

Dealing with Missing Features

Real tables have holes: someone forgot to log a value, or a sensor failed. You generally have three options:

1. Drop the rows (or columns) with missing values — fine if you have plenty of data.
2. Use an algorithm that handles missing values natively (some tree implementations do).
3. Impute — fill the holes with a sensible stand-in value.

Common imputation tricks:

• Replace a missing value with the column mean (or median).
• Replace it with a value outside the normal range, so the model can learn "missing means something unusual."
• Replace it with the midpoint of the range, so a missing value barely nudges the prediction.
• Get fancy: train a small model to predict the missing feature from the other features.

Whatever you choose, apply the same imputation at prediction time that you used during training.

from sklearn.impute import SimpleImputer

# Real data often has holes. Here a couple of values are missing (NaN).
messy = pd.DataFrame({
    "age":   [22, 31, np.nan, 19, 55, 38],
    "spend": [120, np.nan, 300, 60, 500, 200],
})
print("Data with missing values:")
print(messy)
print()

# Strategy 'mean': replace each hole with that column's average.
imp = SimpleImputer(strategy="mean")
filled = pd.DataFrame(imp.fit_transform(messy), columns=messy.columns)
print("After mean imputation (holes filled with column averages):")
print(filled.round(1))

Data with missing values:
    age  spend
0  22.0  120.0
1  31.0    NaN
2   NaN  300.0
3  19.0   60.0
4  55.0  500.0
5  38.0  200.0

After mean imputation (holes filled with column averages):
    age  spend
0  22.0  120.0
1  31.0  236.0
2  33.0  300.0
3  19.0   60.0
4  55.0  500.0
5  38.0  200.0

5.2 Learning Algorithm Selection

There is no single "best" algorithm — the right choice depends on your problem. Before reaching for a model, ask yourself a few questions:

• Does the model need to be explainable? If a non-technical audience must understand predictions, prefer simple, transparent models (linear/logistic regression, kNN, a single decision tree). If pure accuracy matters more, "black box" models (neural nets, ensembles) are often stronger.
• Does the data fit in memory? If yes, almost any algorithm is on the table. If not, look for incremental / out-of-core learners.
• How many examples and features? Neural networks and gradient boosting scale to millions of rows and features; SVMs are more modest.
• Categorical or numerical features? Some algorithms need everything numeric, so you one-hot encode first.
• Is the relationship linear? Linear/logistic regression and a linear-SVM suit linear data; nonlinear data may need kernels, ensembles, or neural nets.
• How fast must training and prediction be? Linear models predict almost instantly; kNN is slower at prediction time.

A quick cheat-sheet by problem type:

Situation	Typical first choice
Small data, need explainability	Linear / logistic regression, decision tree
Medium tabular data, max accuracy	Random forest, gradient boosting
Very large data, roughly linear	Linear / logistic regression (scaled features)
Nonlinear, a few thousand examples	SVM with RBF kernel, small neural net
Images, text, audio	(Ch. 6) neural networks / deep learning

When you cannot decide, the practical answer is: try a few on a validation set and let the numbers decide.

5.3 Three Sets: Train, Validation, Test

This is one of the most important habits in all of machine learning. Once your dataset is ready, shuffle it and split it into three pieces:

1. Training set — the biggest piece. The learning algorithm uses it to fit the model.
2. Validation set — used to choose the algorithm and tune hyperparameters. The algorithm never trains on it.
3. Test set — used once, at the very end, to estimate how the final model will do in the real world.

Why three, not one? A model that simply memorized the training data would score perfectly on the training set but be useless on new data — so we need a hold-out set to measure generalization.

Why two hold-out sets, not one? Because every time you use a hold-out set to make a decision (pick an algorithm, tweak a hyperparameter), you leak a little information into it. The validation set is allowed to be "used up" by tuning; the test set must stay untouched until the final, single assessment. A common split is 70% / 15% / 15%, though huge datasets can get away with 95% / 2.5% / 2.5%.

Everyday analogy: Training set = the practice problems you study from. Validation set = the mock exams you use to adjust your strategy. Test set = the real final exam, taken once.

from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

# A small labeled dataset: 200 examples, 2 classes, 6 features.
X, y = make_classification(n_samples=200, n_features=6, n_informative=3,
                           n_redundant=1, random_state=42)

# Step 1: peel off the TEST set first (15%) -- touch only once, at the very end.
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y)

# Step 2: split what is left into TRAIN (~70% of original) and VAL (~15%).
# 0.15 / 0.85 ~= 0.1765, so val gets ~15% of the full dataset.
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.1765, random_state=42, stratify=y_temp)

print("Three sets created from 200 examples:")
print(f"  train: {X_train.shape[0]} examples  (used to FIT the model)")
print(f"  val:   {X_val.shape[0]} examples    (used to CHOOSE model / hyperparameters)")
print(f"  test:  {X_test.shape[0]} examples   (used ONCE to assess the final model)")

Three sets created from 200 examples:
  train: 139 examples  (used to FIT the model)
  val:   31 examples    (used to CHOOSE model / hyperparameters)
  test:  30 examples   (used ONCE to assess the final model)

5.4 Underfitting and Overfitting

Two things can go wrong with a model, and they are opposite extremes:

• Underfitting (high bias): the model is too simple for the data, so it does poorly even on the training set. A straight line trying to follow a curvy sine wave is a classic underfit. Fix: use a more powerful model, or engineer better features.
• Overfitting (high variance): the model is too complex — it memorizes the training data, noise and all, and does great on it but poorly on the validation/test sets. A wiggly degree-15 polynomial through a noisy sine is a classic overfit. Fix: simplify the model, get more data, reduce features, or regularize.

The name "variance" comes from statistics: if you had drawn a different training set, an overfit model would change dramatically. That sensitivity to the exact training sample is exactly why it fails on new data.

Our goal is the sweet spot in between — complex enough to capture the real pattern, simple enough to ignore the noise. The next cell makes this visible by fitting polynomials of increasing degree to a noisy sine wave and watching the training and validation errors.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

# A noisy sine wave: the TRUE relationship is y = sin(x), hidden under noise.
rng = np.random.RandomState(42)
x = np.sort(rng.rand(40) * 2 * np.pi)            # 40 points spread over [0, 2*pi)
y = np.sin(x) + rng.normal(0, 0.2, size=x.size)  # signal + noise

# Split: 25 for training, 15 for validation.
x_tr, x_va, y_tr, y_va = train_test_split(x, y, test_size=15, random_state=42)
x_tr2 = x_tr[:, None]   # sklearn wants a 2-D X
x_va2 = x_va[:, None]

# Try polynomials of degree 0 (constant) up to 15 (very wiggly).
degrees = range(0, 16)
train_err, val_err = [], []
for d in degrees:
    m = make_pipeline(PolynomialFeatures(d), LinearRegression())
    m.fit(x_tr2, y_tr)
    train_err.append(mean_squared_error(y_tr, m.predict(x_tr2)))
    val_err.append(mean_squared_error(y_va, m.predict(x_va2)))

# --- Plot 1: error vs degree (the bias-variance picture) ---
plt.figure(figsize=(7, 4))
plt.plot(list(degrees), train_err, "o-", label="train error", color="tab:blue")
plt.plot(list(degrees), val_err, "s-", label="validation error", color="tab:orange")
plt.xlabel("polynomial degree  (model complexity -->)")
plt.ylabel("mean squared error")
plt.title("Underfitting vs. Overfitting: find the sweet spot")
plt.legend()
plt.show()

best_d = list(degrees)[int(np.argmin(val_err))]
print(f"Sweet spot (lowest validation error): degree {best_d}")
print(f"  train MSE = {train_err[best_d]:.4f}   val MSE = {val_err[best_d]:.4f}")

# --- Plot 2: three concrete fits -- underfit, good, overfit ---
grid = np.linspace(0, 2 * np.pi, 200)[:, None]
fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))
for ax, d, title in zip(axes, [1, best_d, 15],
                        ["Underfit (deg 1)", f"Good fit (deg {best_d})", "Overfit (deg 15)"]):
    m = make_pipeline(PolynomialFeatures(d), LinearRegression())
    m.fit(x_tr2, y_tr)
    ax.scatter(x_tr, y_tr, s=20, color="tab:blue", label="train")
    ax.scatter(x_va, y_va, s=20, color="tab:orange", label="val", marker="s")
    ax.plot(grid, m.predict(grid), color="black", lw=1.5, label="model")
    ax.plot(grid, np.sin(grid), color="green", ls="--", lw=1, label="true sin(x)")
    ax.set_title(title); ax.set_xlabel("x"); ax.set_ylim(-2, 2)
axes[0].set_ylabel("y")
axes[0].legend(fontsize=7)
plt.tight_layout()
plt.show()

Sweet spot (lowest validation error): degree 4
  train MSE = 0.0344   val MSE = 0.0332

Reading the plots

• In the error-vs-degree plot, training error (blue) keeps dropping as the polynomial gets more flexible — more complexity always fits the training data better.
• Validation error (orange) drops at first, then turns back up. The bottom of that orange curve is the sweet spot: the model is just complex enough.
• To the left of the sweet spot the model underfits (too stiff). To the right it overfits (too wiggly, chasing noise).
• The three fit plots show this concretely: degree 1 is a near-flat line that misses the wave; the best degree traces the sine nicely; degree 15 contorts itself through every noisy point and would fail badly on new data.

5.5 Regularization

Even a simple linear model can overfit when you have many features but few examples: it assigns non-zero weights to almost every feature, hunting for spurious patterns. Regularization adds a penalty to the learning objective so the model is nudged toward smaller, simpler weights — a bit more bias, a lot less variance. This trade-off is the famous bias–variance tradeoff.

For linear regression, instead of only minimizing the mean squared error, we minimize:

error + C * penalty

where C controls how strongly we penalize complexity. Two flavors of penalty:

• L2 (Ridge): penalize the sum of squared weights, sum of w_j^2. Weights shrink toward zero but rarely hit exactly zero. Usually gives the best hold-out performance.
• L1 (Lasso): penalize the sum of absolute weights, sum of |w_j|. Many weights are driven to exactly zero — so L1 also performs feature selection, automatically dropping useless features. Great for explainability.

Combine both and you get elastic net. Regularization is not just for linear models — it shows up in neural networks too (plus tricks like dropout, covered in Chapter 8).

Everyday analogy: L2 is like gently turning down the volume on every instrument in a band so none dominates; L1 is like muting the instruments that are not really contributing to the song.

from sklearn.linear_model import Ridge, Lasso
from sklearn.datasets import make_regression

# A regression problem with 30 features, but only 5 actually matter.
# The rest are noise -- perfect for seeing how L1 zeros out useless features.
X, y = make_regression(n_samples=200, n_features=30, n_informative=5,
                       noise=10, random_state=42)

# Same data, two regularized models. 'alpha' is the strength of the penalty.
ridge = Ridge(alpha=10.0).fit(X, y)
lasso = Lasso(alpha=1.0, max_iter=10000).fit(X, y)

ridge_coef = ridge.coef_
lasso_coef = lasso.coef_

print(f"Ridge (L2): {int(np.sum(ridge_coef != 0))} non-zero coefficients "
      f"out of {len(ridge_coef)}")
print(f"Lasso (L1): {int(np.sum(lasso_coef != 0))} non-zero coefficients "
      f"out of {len(lasso_coef)}  --> it dropped {int(np.sum(lasso_coef == 0))} noise features")

# Plot coefficient magnitudes side by side.
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
axes[0].bar(range(len(ridge_coef)), np.abs(ridge_coef), color="tab:blue")
axes[0].set_title("Ridge (L2): shrunk but rarely zero")
axes[0].set_xlabel("feature index"); axes[0].set_ylabel("|coefficient|")

axes[1].bar(range(len(lasso_coef)), np.abs(lasso_coef), color="tab:orange")
axes[1].set_title("Lasso (L1): many exactly zero")
axes[1].set_xlabel("feature index"); axes[1].set_ylabel("|coefficient|")
plt.tight_layout()
plt.show()

Ridge (L2): 30 non-zero coefficients out of 30
Lasso (L1): 11 non-zero coefficients out of 30  --> it dropped 19 noise features

Reading the coefficient plot

Both models saw the same 30-feature dataset where only 5 features truly mattered. Look at the difference:

• Ridge (L2) shrinks every coefficient but keeps almost all of them non-zero — it hushes the noise features rather than removing them.
• Lasso (L1) zeroes out most of the 25 noise features and keeps only a handful. The bars that remain point to the features that actually carry signal.

That automatic feature selection is why L1 is handy when you want a model you can explain: "these few inputs are what really drive the prediction."

5.6 Model Performance Assessment

You have a model. How good is it, really? You measure it on the test set — data the model has never seen.

For regression, it is straightforward: compare predictions to true values with mean squared error (MSE). A sanity check: your model should beat the "mean model" (the trivial model that always predicts the training average). If test MSE is much higher than training MSE, you are overfitting.

For classification, the toolbox is richer:

• Confusion matrix — a table of predicted vs. actual labels, showing where the model gets confused.
• Accuracy — fraction of examples classified correctly. Simple, but misleading when classes are imbalanced.
• Precision — of everything the model called positive, how much really was? TP / (TP + FP).
• Recall — of everything that really is positive, how much did the model catch? TP / (TP + FN).
• F1-score — the harmonic mean of precision and recall, a single number balancing both.
• ROC / AUC — a curve and its area that summarize the trade-off between true-positive and false-positive rates across thresholds.

You usually have to trade precision for recall (or vice versa). For a spam filter, you want high precision (don't bury a real email) and accept lower recall (some spam gets through). For a cancer screen, you want high recall (don't miss a sick patient) and accept lower precision (some false alarms).

Cross-validation is the trick for when you don't have enough data for a separate validation set: split the training data into k folds (typically 5), train on k−1 folds and validate on the held-out fold, rotate so each fold is validated once, and average the k scores. That gives a much more reliable estimate than a single split.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.metrics import (confusion_matrix, classification_report,
                             accuracy_score)

# A binary classification dataset, slightly imbalanced (70/30).
X, y = make_classification(n_samples=300, n_features=8, n_informative=5,
                           n_redundant=1, weights=[0.7, 0.3], random_state=42)

# Train/test split: train a model, then assess on the test set.
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          random_state=42, stratify=y)

clf = RandomForestClassifier(n_estimators=50, max_depth=4,
                             random_state=42, n_jobs=1)
clf.fit(X_tr, y_tr)
y_pred = clf.predict(X_te)

# 5-fold cross-validation on the training set: a robust, less jittery score.
cv_scores = cross_val_score(clf, X_tr, y_tr, cv=5, n_jobs=1)
print("5-fold CV accuracy: %.3f +/- %.3f" % (cv_scores.mean(), cv_scores.std()))
print("Test-set accuracy:  %.3f" % accuracy_score(y_te, y_pred))
print()

# Confusion matrix: rows = actual, columns = predicted.
cm = confusion_matrix(y_te, y_pred)
print("Confusion matrix (rows = actual, cols = predicted):")
print(cm)
print()

# Full report: precision, recall, f1-score per class.
print("Classification report:")
print(classification_report(y_te, y_pred, target_names=["class 0", "class 1"]))

5-fold CV accuracy: 0.849 +/- 0.026
Test-set accuracy:  0.880

Confusion matrix (rows = actual, cols = predicted):
[[50  2]
 [ 7 16]]

Classification report:
              precision    recall  f1-score   support

     class 0       0.88      0.96      0.92        52
     class 1       0.89      0.70      0.78        23

    accuracy                           0.88        75
   macro avg       0.88      0.83      0.85        75
weighted avg       0.88      0.88      0.88        75

Reading the assessment output

• The 5-fold CV accuracy is averaged over five different train/validation splits, so it is far less jittery than a single split. Its standard deviation tells you how sensitive the score is to the exact split.
• The confusion matrix shows the four counts for a binary problem: true positives, false positives, false negatives, true negatives. The off-diagonal numbers are the mistakes.
• The classification report gives precision, recall, and F1 per class. For the minority class, precision and recall usually tell you more than raw accuracy.
• Notice the test accuracy can differ from the CV accuracy — that gap is exactly why we keep a held-out test set.

ROC Curves and AUC

Some classifiers (logistic regression, random forests, neural nets) output a confidence score or probability rather than a hard yes/no. By sliding the decision threshold from 0 to 1, you get a family of (false-positive-rate, true-positive-rate) points. Plot them and you get the ROC curve.

The area under the curve (AUC) boils that curve down to one number:

• AUC = 1.0 — a perfect classifier.
• AUC = 0.5 — no better than a coin flip (the diagonal line).
• AUC < 0.5 — something is wrong (your labels may be flipped).

AUC is handy because it summarizes performance across all thresholds at once, so you can compare models without first committing to a threshold.

from sklearn.metrics import roc_curve, roc_auc_score

# Use the classifier from the previous cell. Many models can output a
# probability per example -- we use that "confidence score" to draw a ROC curve.
y_score = clf.predict_proba(X_te)[:, 1]   # probability of the positive class

fpr, tpr, thresholds = roc_curve(y_te, y_score)
auc = roc_auc_score(y_te, y_score)

plt.figure(figsize=(5, 4))
plt.plot(fpr, tpr, color="tab:blue", lw=2, label=f"ROC curve (AUC = {auc:.3f})")
plt.plot([0, 1], [0, 1], color="gray", ls="--", label="random (AUC = 0.5)")
plt.xlabel("false positive rate")
plt.ylabel("true positive rate (recall)")
plt.title("ROC curve")
plt.legend(loc="lower right")
plt.show()

print(f"AUC = {auc:.3f}  (>0.5 beats random; 1.0 is perfect)")

AUC = 0.901  (>0.5 beats random; 1.0 is perfect)

5.7 Hyperparameter Tuning

A hyperparameter is a setting you choose before training — not something the algorithm learns. Examples: C for an SVM, max_depth for a tree, the learning rate for gradient descent. Picking good values is called tuning, and two simple strategies dominate:

• Grid search: list a discrete set of values for each hyperparameter, then try every combination. Thorough, but the number of combinations explodes quickly (7 values × 2 kernels = 14 models; add a third hyperparameter and it multiplies again).
• Random search: give a range/distribution for each hyperparameter and sample a fixed number of random combinations. Surprisingly often, random search finds a model that is nearly as good in far fewer trials, because it explores each hyperparameter more broadly.

When you don't have enough data for a validation set, grid search + cross-validation is the standard combo: GridSearchCV tries each combination and scores it with k-fold CV, then picks the best. (Bayesian and other smarter methods exist too — they use past trials to choose the next ones to evaluate.)

A practical tip: for a hyperparameter like C that can span many orders of magnitude, search on a logarithmic scale (0.001, 0.01, 0.1, 1, 10, 100, 1000) rather than evenly spaced values.

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Grid search = try EVERY combination in a small table of hyperparameters.
# Here we tune an SVM: the penalty C (log-spaced) and the kernel.
# GridSearchCV scores each combination with 5-fold cross-validation.
param_grid = {
    "C":      [0.1, 1, 10, 100],
    "kernel": ["linear", "rbf"],
}
# 4 x 2 = 8 combinations, each scored with 5-fold CV -> 40 fits.
# n_jobs=1 keeps it single-threaded and quick on this small dataset.
svc = SVC(random_state=42)
grid = GridSearchCV(svc, param_grid, cv=5, n_jobs=1)
grid.fit(X_tr, y_tr)

print("Grid search tried", len(grid.cv_results_['params']), "combinations")
print("Best parameters:    ", grid.best_params_)
print("Best CV accuracy:    %.3f" % grid.best_score_)
print("Test-set accuracy:   %.3f (best model on held-out test)"
      % accuracy_score(y_te, grid.predict(X_te)))
print()

# Random search = sample a fixed number of combos from given value lists.
# Same search space, but we only try 6 random combinations.
rand = RandomizedSearchCV(svc,
                          {"C": [0.1, 1, 10, 100, 1000], "kernel": ["linear", "rbf"]},
                          n_iter=6, cv=5, random_state=42, n_jobs=1)
rand.fit(X_tr, y_tr)
print("Random search (6 samples) best: ", rand.best_params_,
      " CV=%.3f" % rand.best_score_)

Grid search tried 8 combinations
Best parameters:     {'C': 1, 'kernel': 'rbf'}
Best CV accuracy:    0.902
Test-set accuracy:   0.893 (best model on held-out test)

Random search (6 samples) best:  {'kernel': 'rbf', 'C': 10}  CV=0.884

Reading the tuning output

• GridSearchCV tried all 8 combinations (4 values of C × 2 kernels), each scored with 5-fold CV — 40 model fits in all — and reported the best.
• The best CV accuracy is the cross-validated score of the winning combination; the test accuracy is how that winning model does on the untouched test set.
• Random search sampled fewer combinations yet landed on a comparably good setting — that is the appeal: less compute, similar result.
• In a real project you would then zoom in around the best values with a finer grid, and finally lock in the model with one test-set evaluation.

Key Takeaways

• Feature engineering turns raw data into a table of informative features; one-hot encode categories, bin continuous values when it helps, scale numbers, and impute missing values so the model sees clean numeric input.
• Pick an algorithm by asking about explainability, data size, feature types, linearity, and speed — when unsure, let a validation set decide.
• Always split into train / validation / test. Train to fit, validation to choose, test once to assess.
• Underfitting = too simple (high bias); overfitting = too complex (high variance). Find the sweet spot by watching validation error as model complexity grows.
• Regularization (L1/L2) adds a penalty for complexity. L1 (lasso) zeros out useless features; L2 (ridge) gently shrinks all weights.
• Measure classification with a confusion matrix, precision/recall/F1, and ROC/AUC — not accuracy alone, especially for imbalanced data. Use cross-validation when data is scarce.
• Tune hyperparameters with grid or random search, searching sensitive parameters on a log scale, and combine with cross-validation (GridSearchCV).

What's Next

We have stayed in the world of "shallow" models and practical recipes. In Chapter 6 — Neural Networks and Deep Learning, we open the black box that powers modern image, text, and speech systems: artificial neurons, layers, backpropagation, and how to train them.

Exercises

These exercises cover the practical craft from Chapter 5: feature engineering, the three-way data split, the bias–variance trade-off, regularization, performance metrics, and hyperparameter tuning. Try each before reading the hint.

1. (Conceptual) Why is mapping red→1, yellow→2, green→3 a bad idea for a categorical feature, and what does one-hot encoding do instead? Hint: the number mapping invents a fake ordering ("green is three times red"); one-hot makes one 0/1 column per category with no implied order.
2. (Conceptual) When would you prefer standardization over min-max normalization, and vice versa? Hint: standardization handles outliers and bell-curve data better; normalization gives a fixed [0,1] range but a single outlier can crush the other values into a tiny slice.
3. (Conceptual) Why split into three datasets instead of two? What is the test set for, and when are you allowed to touch it? Hint: the validation set gets "used up" by tuning decisions; the test set stays untouched until one single final assessment.
4. (Conceptual) Define underfitting and overfitting in terms of bias and variance, and describe how to find the sweet spot. Hint: underfitting = high bias (too simple, poor even on training); overfitting = high variance (memorizes noise); the sweet spot is where validation error is minimized as complexity grows.
5. (Conceptual) Contrast L1 (lasso) and L2 (ridge) regularization: which one performs feature selection and why? Hint: L1 drives many weights to exactly zero (automatic feature selection); L2 shrinks all weights toward zero but rarely to exactly zero.
6. (Conceptual) Give a situation where raw accuracy is misleading and precision/recall matter more, and explain the precision–recall trade-off using a spam filter vs a cancer screen. Hint: imbalanced classes; spam wants high precision (don't bury real email), cancer wants high recall (don't miss a sick patient).
7. (Conceptual) Why tune a hyperparameter like C on a logarithmic scale, and why combine grid search with cross-validation? Hint: C matters across orders of magnitude; CV gives a robust score without needing a separate validation set.

Hands-On Coding Problems

8. (Coding) One-hot encode a small categorical column ["red","yellow","green","red","green"] and print the resulting 0/1 table. Hint: pd.get_dummies or OneHotEncoder(sparse_output=False).
9. (Coding) Compare MinMaxScaler and StandardScaler on a 1D feature that contains an outlier (e.g. [1,2,3,4,5,100]). Print the min, max, mean, and std of the scaled values for each. Hint: .fit_transform then reshape; notice how the outlier crushes the min-max range.
10. (Coding) Build a single ColumnTransformer/Pipeline that one-hot encodes a categorical column, imputes a numeric column's missing value, and scales it — then fit it on a tiny messy DataFrame. Hint: SimpleImputer + OneHotEncoder + StandardScaler inside ColumnTransformer.
11. (Coding) Overfitting demo: fit polynomials of degree 1, 3, 5, and 15 to a noisy sine wave, record training and validation MSE for each, and plot MSE vs degree; identify the sweet spot. Hint: PolynomialFeatures(degree) + LinearRegression; use train_test_split.
12. (Coding) L1 vs L2: on a dataset where only 5 of 30 features truly matter, fit Lasso and Ridge, then plot their coefficients and count how many weights Lasso zeros out. Hint: make_regression(n_informative=5, n_features=30); count with (np.abs(coef) < 1e-4).sum().
13. (Coding) On an imbalanced make_classification dataset, compute the confusion matrix, precision, recall, and F1 with sklearn, and print classification_report. Hint: confusion_matrix, precision_score, recall_score, f1_score, classification_report.
14. (Coding) Use GridSearchCV to tune an SVC over C=[0.1,1,10,100] and kernel=["linear","rbf"] with 5-fold CV; print the best parameters and the test-set accuracy of the best model. Hint: GridSearchCV(SVC(), param_grid, cv=5).fit(Xtr, ytr) then .best_params_ and .score(Xte, yte).

# Exercise 8: one-hot encoding a categorical column
import pandas as pd

colors = ["red", "yellow", "green", "red", "green"]

# TODO: one-hot encode `colors` and print the result
# Hint: pd.get_dummies(colors)  or  sklearn.preprocessing.OneHotEncoder(sparse_output=False)
print("encoded:\n", None)  # placeholder

# Exercise 9: MinMaxScaler vs StandardScaler with an outlier
import numpy as np
from sklearn.preprocessing import MinMaxScaler, StandardScaler

x = np.array([1, 2, 3, 4, 5, 100]).reshape(-1, 1)

# TODO: fit_transform with both scalers, then print min/max/mean/std of each
mm = None  # placeholder
ss = None  # placeholder

print("MinMax scaled  :", mm.ravel())
print("Standard scaled:", ss.ravel())

# Exercise 10: one-hot + impute + scale in one ColumnTransformer
import pandas as pd
import numpy as np
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

df = pd.DataFrame({
    "color":  ["red", "yellow", "green", "red", "yellow", None],
    "age":    [25, np.nan, 40, 35, 50, 30],
    "label":  [0, 1, 1, 0, 1, 0],
})

# TODO: build a ColumnTransformer that one-hot encodes "color",
# imputes (mean) then scales "age", and feed it into a pipeline with LogisticRegression.
pre = None  # placeholder

# TODO: fit on df[["color","age"]] and df["label"], then print .score

# Exercise 11: polynomial degree vs overfitting
import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(0)
x = np.sort(rng.uniform(0, 6, 60))
y = np.sin(x) + rng.normal(0, 0.2, size=60)
X = x.reshape(-1, 1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

degrees = [1, 3, 5, 15]
train_mse, val_mse = [], []
for d in degrees:
    # TODO: build PolynomialFeatures(degree=d) + LinearRegression, fit on Xtr,
    #       record mean_squared_error on Xtr and Xte
    train_mse.append(0.0)  # placeholder
    val_mse.append(0.0)    # placeholder

# TODO: plot train_mse and val_mse vs degrees; mark the degree with the lowest val_mse
plt.show()

# Exercise 12: L1 (Lasso) vs L2 (Ridge) on mostly-noise features
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import Lasso, Ridge
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=100, n_features=30, n_informative=5,
                       noise=5.0, random_state=0)

# TODO: fit Lasso(alpha=0.1) and Ridge(alpha=1.0), then get .coef_ from each
lasso = None  # placeholder
ridge = None  # placeholder

# TODO: plot both coefficient vectors; print how many lasso weights are ~0
zero_count = 0  # placeholder: (np.abs(lasso.coef_) < 1e-4).sum()
print("Lasso zeroed-out weights:", zero_count)
plt.show()

# Exercise 13: confusion matrix, precision, recall, F1
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (confusion_matrix, precision_score,
                             recall_score, f1_score, classification_report)

X, y = make_classification(n_samples=500, weights=[0.9, 0.1], random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

# TODO: fit LogisticRegression(max_iter=500) and predict on Xte
clf = None  # placeholder
pred = None  # placeholder

# TODO: print confusion_matrix, precision_score, recall_score, f1_score, and classification_report

# Exercise 14: GridSearchCV on an SVM
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.svm import SVC

X, y = make_classification(n_samples=400, n_features=5, n_informative=4,
                           n_redundant=0, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

param_grid = {"C": [0.1, 1, 10, 100], "kernel": ["linear", "rbf"]}

# TODO: run GridSearchCV(SVC(), param_grid, cv=5) on Xtr/ytr, print best_params_ and .score on Xte
gs = None  # placeholder

print("best params:", None)
print("test accuracy:", None)