Machine Learning, Clearly - All Chapters in One Notebook

This single notebook contains the entire companion book: the title/preface followed by Chapters 1-11 in order. Just scroll down to read everything sequentially. Every code cell has already been run, so plots and printed results are visible without running anything.

Adapted and edited by [Your Name], based on Andriy Burkov, The Hundred-Page Machine Learning Book* (2019).*

---

Tip: use the Jupyter Table of Contents sidebar (in JupyterLab: the list icon on the left) to jump between chapter headings.

---

Front Matter - Title, Preface & How to Read

Machine Learning, Clearly

A Beginner's Notebook Companion

By Miftahur Rahman, Ph.D.

Published by IRays Teknology Ltd

Technical edition, September 2026

Based on Andriy Burkov, The Hundred-Page Machine Learning Book* (2019)*

---

"All models are wrong, but some are useful." — George Box

How to Read This Book

This is a set of interactive Jupyter notebooks. Start here, then read the chapters in order (1 → 11), since later chapters build on earlier ones. At the end of every chapter you'll find a What's Next note pointing to the next one.

• Recommended: read the chapters in order (1 → 11), because later chapters build on earlier ones.
• Every code cell has already been run, so you can read the outputs (plots and printed results) without running anything yourself.
• To re-run a notebook yourself: open it, then Kernel → Restart & Run All. All required libraries (numpy, pandas, scikit-learn, matplotlib, scipy, and a little tensorflow in Chapter 6) are already installed in your Anaconda environment.

Preface

This book is a set of interactive Jupyter notebooks that re-teaches the ideas in Andriy Burkov's The Hundred-Page Machine Learning Book in a slower, friendlier, hands-on way. Burkov's book is famous for packing the practical core of machine learning into about a hundred pages. This companion unpacks those pages into explanations you can read at a relaxed pace, with runnable Python in every chapter so you can see each idea in action rather than just read about it.

Who this is for. Beginners who want a clear, intuitive first tour of machine learning, and practitioners who want a quick refresher with code. No deep math background is required — we introduce notation gently and always pair formulas with examples.

A note on authorship. The original ideas, structure, and technical content come from Andriy Burkov's book. This companion was adapted and edited for clarity and interactivity; any errors or awkward explanations in the adaptation are mine.

— [Your Name]

How These Notebooks Are Organized

• Each chapter is its own .ipynb file and is self-contained — you can read chapters independently, though they build on each other in order.
• Every notebook starts with an "In this chapter you will learn" list and ends with Key Takeaways and a pointer to the next chapter.
• Python cells use small, synthetic datasets so they run in seconds on any laptop, with no internet or large downloads required.
• Math is kept light: we explain symbols in words before using them, and pair every formula with a concrete example.

---

Copyright

© 2026 IRays-Teknology-Ltd. All rights reserved.

First technical edition, September 2026.

List price: $9.99 USD (print paperback). ISBN: 978-0-0000000-0-0 (placeholder — assign your own before publishing).

No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law. For permission requests, contact the publisher at IRays-Teknology-Ltd.

A Note on This Work

This book is an interactive companion that re-explains, in simpler language with runnable Python examples, the ideas in Andriy Burkov's The Hundred-Page Machine Learning Book (2019). The original concepts, structure, and technical content are attributed to Burkov; this edition was adapted and edited by Miftahur Rahman, Ph.D. Readers are advised to consult the original work for the author's full treatment. The Python examples and pedagogical explanations are original to this companion.

Disclaimer

The information in this book is provided "as is" for educational purposes. While every effort has been made to ensure accuracy, neither the publisher nor the editor warrants the correctness of all examples and accepts no liability for any use thereof.

---

---

Chapter 1 — Introduction

Welcome! This is the first chapter of our beginner-friendly journey through machine learning, adapted from Andriy Burkov's The Hundred-Page Machine Learning Book. In this chapter we build the big-picture mental model that the rest of the book fills in.

In this chapter you will learn:

• What machine learning really is (and why the name is a bit misleading)
• The four main types of learning
• How a supervised learning system turns data into a model
• Why a model trained on old data can make good predictions on brand-new data

1.1 What Is Machine Learning?

Here is the honest truth that the book opens with: machines don't really "learn." A typical "learning machine" finds a mathematical formula that, when applied to a collection of inputs (the training data), produces the desired outputs. That same formula also gives correct outputs for most other inputs — as long as those new inputs come from the same kind of distribution as the training data.

Why isn't that learning? Because if you slightly distort the input, the output can become completely wrong. If you learned to play a video game by staring straight at the screen, you would still play well if someone tilted the screen a little. A machine learning model trained only on a straight screen would usually fail on a tilted one — unless it was also trained to handle rotation.

So why the name? Mostly marketing. Arthur Samuel coined "machine learning" at IBM in 1959. Like "artificial intelligence," the term stuck even though it is used by analogy to animal learning, not literally.

A practical working definition:

Machine learning is the process of solving a practical problem by (1) gathering a dataset, and (2) algorithmically building a statistical model based on that dataset, then using that model to solve the problem.

The Machine Learning Recipe

Almost every machine learning project follows the same three-step recipe:

1. Gather data — collect examples of the phenomenon you care about.
2. Build a model — run a learning algorithm on the data to produce a mathematical formula/model.
3. Use the model — feed it new inputs to get predictions or decisions.

Let's make this concrete with a tiny, visual example in code.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# Step 1 -- Gather data: two clusters = 'spam' (+1) and 'not spam' (-1)
X, y_raw = make_blobs(n_samples=200, centers=2, cluster_std=1.0, random_state=42)
y = np.where(y_raw == 0, -1, 1)

plt.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:orange", label="+1 (spam)")
plt.scatter(X[:, 0][y == -1], X[:, 1][y == -1], color="tab:blue", label="-1 (not spam)")
plt.title("Step 1 — Gather data: two classes of examples")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

1.2 Types of Learning

Learning comes in four main flavors, depending on what kind of data you have:

Type	Data	Goal
Supervised	labeled examples {(xᵢ, yᵢ)}	predict the label y for a new x
Unsupervised	unlabeled examples {xᵢ}	find structure (clusters, reduced dimensions, outliers)
Semi-supervised	a few labeled + many unlabeled	same as supervised, but leverage the unlabeled data
Reinforcement	states + actions + rewards	learn a policy that maximizes long-term reward

Let's unpack each one.

1.2.1 Supervised Learning

In supervised learning the dataset is a collection of labeled examples {(xᵢ, yᵢ)} for i = 1…N.

• Each xᵢ is a feature vector — a list of numbers that describe one example. If an example is a person, feature 1 could be height in cm, feature 2 weight in kg, feature 3 gender, and so on.
• Each feature at position j always stores the same kind of information across all examples (feature 2 is always "weight in kg").
• The label yᵢ is what we want to predict. It is usually either a class (spam / not_spam) or a real number (a price, a temperature).

The goal of a supervised learning algorithm is to use the dataset to produce a model that takes a feature vector x as input and outputs enough information to deduce its label.

Everyday analogy: You are given thousands of flashcards. On the front is a photo (the feature vector); on the back is "cat" or "dog" (the label). After studying them, you can label a new photo you have never seen. That is supervised learning.

1.2.2 Unsupervised Learning

In unsupervised learning the dataset is a collection of unlabeled examples {xᵢ}. There are no labels to learn from — the algorithm must find structure on its own.

Typical tasks:

• Clustering — group similar examples together; the model returns a cluster id for each input.
• Dimensionality reduction — compress a feature vector into one with fewer features while keeping the important information.
• Outlier detection — output a number saying how "unusual" an example is compared to a typical one.

Everyday analogy: You are handed a big pile of mixed photos with no captions and asked to sort them into albums of similar-looking pictures. Nobody told you the categories — you discover them. That is unsupervised learning.

from sklearn.cluster import KMeans

# Unlabeled data: 3 natural groups (we PRETEND we don't know the labels)
X_un, _ = make_blobs(n_samples=300, centers=3, cluster_std=1.1, random_state=7)

# Let the algorithm find 3 clusters on its own -- no labels given!
kmeans = KMeans(n_clusters=3, n_init=10, random_state=7)
labels = kmeans.fit_predict(X_un)

plt.scatter(X_un[:, 0], X_un[:, 1], c=labels, cmap="viridis", s=20)
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1],
            marker="X", s=200, color="red", edgecolors="white", label="found centers")
plt.title("Unsupervised: KMeans discovers 3 clusters with no labels")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(

1.2.3 Semi-Supervised Learning

Here the dataset has both labeled and unlabeled examples — usually far more unlabeled than labeled. The goal is the same as supervised learning; the hope is that the many unlabeled examples reveal more about the data's distribution and so help build a better model.

It can feel counter-intuitive that more unlabeled data helps. But unlabeled examples still carry information: a bigger sample reflects the true distribution better, and a good algorithm can exploit that.

1.2.4 Reinforcement Learning

In reinforcement learning an agent "lives" in an environment. It perceives the environment's state as a feature vector, chooses an action, and receives a reward (and moves to a new state). The goal is to learn a policy — a function from states to the best action — that maximizes the expected average reward.

This is for sequential decision making with long-term goals: game playing, robotics, logistics. This book (and therefore our notebooks) focuses on one-shot decisions where examples are independent, so we leave reinforcement learning aside.

1.3 How Supervised Learning Works

Let's walk through the whole supervised pipeline once, end to end, using spam detection as the example.

Step 1 — Gather data. Collect, say, 10,000 emails, each manually labeled "spam" or "not_spam".

Step 2 — Convert each email into a feature vector. A classic method is the bag of words: take a dictionary of 20,000 English words and make a 20,000-dimensional vector where position j is 1 if word j appears in the email, else 0. Now every email is a vector of 0s and 1s, and every label is text.

Step 3 — Turn labels into numbers. Many algorithms need numeric labels. The book illustrates with a Support Vector Machine (SVM), which wants the positive class ("spam") as +1 and the negative class ("not_spam") as −1.

Step 4 — Run the learning algorithm to get a model. SVM imagines every feature vector as a point in a high-dimensional space (20,000 dimensions here!) and draws a hyperplane that separates the +1 points from the −1 points. The boundary between classes is called the decision boundary.

The hyperplane has the equation w·x + b = 0, where w is a vector (same size as x) and b is a number. The prediction for a new x is:

ŷ = sign(w·x + b) → +1 means spam, −1 means not spam.

Step 5 — Find the best w and b. The machine optimizes: it wants every training example on the correct side of the boundary, and it wants the largest possible margin (the gap between the closest +1 and −1 points). A big margin means better generalization — doing well on new data, not just the training data.

To get a big margin, SVM minimizes the length (norm) of w, because the distance between the two margin lines equals 2 / ‖w‖ — so a smaller ‖w‖ means a bigger gap.

Let's visualize this with a linear SVM on our 2D dataset, and draw the decision boundary and the margin.

from sklearn.svm import SVC

# Same two-blob dataset as the first plot
X, y_raw = make_blobs(n_samples=200, centers=2, cluster_std=1.0, random_state=42)
y = np.where(y_raw == 0, -1, 1)

# Step 4: train a linear SVM. Large C -> essentially a 'hard' margin.
clf = SVC(kernel="linear", C=1000, random_state=42)
clf.fit(X, y)

# Plot the data
plt.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:orange", label="+1 (spam)")
plt.scatter(X[:, 0][y == -1], X[:, 1][y == -1], color="tab:blue", label="-1 (not_spam)")

# Draw the decision boundary w.x + b = 0 and the margins w.x + b = +/-1
ax = plt.gca()
xx = np.linspace(*ax.get_xlim(), 200)
w = clf.coef_[0]
b = clf.intercept_[0]
yy          = -(w[0] * xx + b) / w[1]       # decision boundary
yy_margin_p = -(w[0] * xx + b - 1) / w[1]   # margin: w.x + b = +1
yy_margin_m = -(w[0] * xx + b + 1) / w[1]   # margin: w.x + b = -1
plt.plot(xx, yy, "k-",  label="decision boundary  w·x + b = 0")
plt.plot(xx, yy_margin_p, "k--", alpha=0.6, label="margin  w·x + b = ±1")
plt.plot(xx, yy_margin_m, "k--", alpha=0.6)

# Highlight the support vectors (points lying on the margin)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=160, facecolors="none", edgecolors="red", linewidths=1.8,
            label="support vectors")
plt.title("Step 4 — SVM finds the widest-margin separating hyperplane")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend(loc="upper left", fontsize=8)
plt.show()

Reading the plot

• The solid black line is the decision boundary: points on one side are predicted +1, on the other side −1.
• The two dashed lines are the margins — the closest the training points get to the boundary on each side.
• The circled red points are the support vectors: the examples that sit exactly on the margin. They "support" the boundary — move one, and the boundary moves.
• SVM chose the boundary that makes the gap (margin) between the classes as wide as possible.

The book notes that any classification algorithm creates some decision boundary — straight, curved, or complex. The shape of that boundary is what makes one algorithm different from another, and it also determines accuracy. Two other practical differentiators are training speed and prediction speed — sometimes a slightly less accurate model that trains or predicts much faster is the better choice.

1.4 Why the Model Works on New Data

Why should a model built from old examples predict new ones correctly?

Look at the SVM plot again. If the two classes are separable by a boundary, then each class lives in its own region of space. If your training examples were collected randomly and independently from the same process, then a new example is statistically likely to land near other examples of its own class — not far away in empty space. So a boundary that separated the training examples well will, with high probability, also separate new examples well.

Errors happen when a new example lands on the "wrong" side of the boundary — but because that is less likely, correct predictions outnumber errors. A larger training set makes it even less likely that a new example is very different from anything seen before. And SVM's maximum-margin trick explicitly pushes the boundary as far as possible from both classes, which is exactly what helps on new data.

The formal study of when and under what conditions a learner will "probably output an approximately correct" model is called PAC learning (Probably Approximately Correct). You don't need the math now — just the intuition above.

Mini demo: bag-of-words features

Let's make the "bag of words" idea tangible with a tiny vocabulary of 6 words and a few short emails.

# A tiny 'dictionary' of 6 words (alphabetically sorted, like the book's
# 20,000-word idea, just much smaller)
vocab = ["free", "hello", "money", "now", "prize", "team"]

emails = {
    "hello team":            "not_spam",
    "free prize money now":  "spam",
    "team meeting now":      "not_spam",
    "free money now":        "spam",
}

def bag_of_words(text, vocab):
    # Convert text into a 0/1 feature vector over the vocabulary
    words = set(text.split())
    return [1 if word in words else 0 for word in vocab]

# Show the feature vectors and numeric labels (+1 spam, -1 not_spam)
print(f"{'email':<24}{'vector':<28}label")
print("-" * 60)
for text, label in emails.items():
    vec = bag_of_words(text, vocab)
    y_num = 1 if label == "spam" else -1
    print(f"{text:<24}{str(vec):<28}{y_num}")

print()
print("Notice: 'meeting' is not in the vocabulary, so it is simply ignored.")
print("That is exactly how a bag-of-words model treats unknown words.")

email                   vector                      label
------------------------------------------------------------
hello team              [0, 1, 0, 0, 0, 1]          -1
free prize money now    [1, 0, 1, 1, 1, 0]          1
team meeting now        [0, 0, 0, 1, 0, 1]          -1
free money now          [1, 0, 1, 1, 0, 0]          1

Notice: 'meeting' is not in the vocabulary, so it is simply ignored.
That is exactly how a bag-of-words model treats unknown words.

Key Takeaways

• "Machine learning" is really finding a formula that maps inputs to outputs and generalizes to new inputs. It is learning by analogy, not literally.
• The recipe is always: gather data → build a model → use the model.
• Four learning types by data: supervised (labeled), unsupervised (unlabeled), semi-supervised (mostly unlabeled), reinforcement (states / actions / rewards).
• A feature vector describes an example; a label is what we predict.
• Supervised learning builds a decision boundary; the algorithm determines the boundary's shape.
• SVM finds the widest-margin boundary, which helps it generalize to new data.
• Models generalize because new data statistically resembles training data — bigger training sets and bigger margins help.

What's Next

In Chapter 2 — Notation and Definitions, we'll learn the vocabulary and math notation (vectors, random variables, Bayes' rule, and more) that the rest of the book relies on. Don't worry — we'll keep it friendly and grounded in examples.

---

---

Chapter 2 — Notation and Definitions

Before we build any real models, we need a shared vocabulary. This chapter collects the notation and a handful of core ideas — vectors, random variables, Bayes' rule, and the difference between classification and regression — that every later chapter leans on. We'll keep the math light and ground each idea in a tiny runnable Python example.

In this chapter you will learn:

• How to read the basic notation: scalars, vectors, matrices, sets, and the real numbers $\mathbb{R}$
• What a dot product and a vector norm are, and the difference between the 1-norm and the 2-norm
• What a random variable is, and the difference between a probability mass function (PMF) and a probability density function (PDF)
• What an "unbiased estimator" means and why the sample mean is one
• How to use Bayes' rule to flip a conditional probability around
• The ideas behind parameter estimation: maximum likelihood (MLE) and MAP
• The difference between classification and regression
• The difference between model-based and instance-based learning
• The difference between shallow and deep learning

2.1 Notation — Scalars, Vectors, and Sets

A scalar is just a single number, like $15$ or $-3.25$. We write scalars as ordinary italic letters: $x$, $a$, $c$.

A vector is an ordered list of scalars. We write vectors in bold, like $\mathbf{x}$ or $\mathbf{w}$. You can picture a vector two ways: as an arrow pointing in a direction, or as a point sitting at a location in space. The individual numbers inside the vector are its attributes (or components), and we pick one out with an index: $x^{(j)}$ means "the $j$-th attribute of $\mathbf{x}$." (Don't confuse this with a power like $x^2$ — to square an attribute we'd write $(x^{(j)})^2$.)

Everyday analogy: a vector is like a row in a spreadsheet. Each column is an attribute, and the whole row is one example.

A matrix is a grid of numbers (a table of vectors). We usually give matrices capital letters like $W$. A matrix with 2 rows and 3 columns has shape $2 \times 3$.

A set is an unordered collection of unique things, written with a calligraphic capital like $\mathcal{S}$. A finite set uses braces: $\{1, 3, 18\}$. The set of all real numbers — everything from $-\infty$ to $+\infty$ — gets the special symbol $\mathbb{R}$. When something $x$ belongs to a set $\mathcal{S}$ we write $x \in \mathcal{S}$.

The two stars of supervised learning are the feature vector $\mathbf{x}$ (the inputs describing an example) and the label $y$ (the thing we want to predict). A whole dataset of $N$ examples is often written $\{(\mathbf{x}_i, y_i)\}_{i=1}^{N}$.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# Three 2D vectors (like the book's example)
a = np.array([2, 3])
b = np.array([-2, 5])
c = np.array([1, 0])

# A tiny matrix: 2 rows x 3 columns
W = np.array([[1, 2, 3],
              [4, 5, 6]])

print("vector a =", a, "   a[0] =", a[0], "(the 1st attribute)")
print("vector b =", b)
print("vector c =", c)
print("matrix W (2x3) =\n", W)
print("W.shape =", W.shape, "-> 2 rows, 3 columns")

# Plot the same vectors two ways: as arrows AND as points at their tips
fig, ax = plt.subplots()
origin = np.array([0, 0])
for vec, name, col in [(a, "a = [2, 3]", "tab:red"),
                       (b, "b = [-2, 5]", "tab:blue"),
                       (c, "c = [1, 0]", "tab:green")]:
    ax.annotate("", xy=vec, xytext=origin,
                arrowprops=dict(arrowstyle="->", color=col, lw=2))  # arrow view
    ax.scatter(*vec, color=col, zorder=5)                            # point view
    ax.text(vec[0] + 0.1, vec[1] + 0.1, name, color=col, fontsize=11)
ax.axhline(0, color="gray", lw=0.5)
ax.axvline(0, color="gray", lw=0.5)
ax.set_xlim(-3, 4)
ax.set_ylim(-1, 6)
ax.set_title("Vectors shown as arrows (and their tips as points)")
ax.set_xlabel("dimension 1")
ax.set_ylabel("dimension 2")
plt.show()

vector a = [2 3]    a[0] = 2 (the 1st attribute)
vector b = [-2  5]
vector c = [1 0]
matrix W (2x3) =
 [[1 2 3]
 [4 5 6]]
W.shape = (2, 3) -> 2 rows, 3 columns

Reading the plot

Each vector appears twice on purpose: once as an arrow from the origin, and once as a point at the arrow's tip. Both pictures are valid and we'll switch between them throughout the book — arrows are handy for thinking about direction, points are handy for thinking about data. The matrix W above is just a 2×3 grid of numbers; in code, W.shape tells us "2 rows, 3 columns."

2.1 (continued) — Dot Product and Norms

The dot product (a.k.a. inner product or scalar product) of two same-length vectors $\mathbf{w}$ and $\mathbf{x}$ is a single number:

$$\mathbf{w}\,\mathbf{x} \;=\; \sum_{i=1}^{m} w^{(i)} x^{(i)} \;=\; w^{(1)}x^{(1)} + w^{(2)}x^{(2)} + \dots + w^{(m)}x^{(m)}.$$

It's a way to "combine" two vectors into one number. A large positive dot product tends to mean the vectors point in a similar direction and are long; a dot product near zero often means they're roughly perpendicular.

A norm $\|\mathbf{x}\|$ measures a vector's "length." The two you'll meet most are:

• 2-norm (Euclidean length — the everyday straight-line distance):

$$\|\mathbf{x}\|_2 = \sqrt{\sum_i \bigl(x^{(i)}\bigr)^2}$$

• 1-norm (sum of absolute values — "taxicab" distance):

$$\|\mathbf{x}\|_1 = \sum_i \bigl|x^{(i)}\bigr|$$

Analogy: the 2-norm is how a bird flies from A to B (a straight line). The 1-norm is how a taxi drives through a city grid (only horizontal and vertical moves).

# Dot product: combine two same-length vectors into one scalar
w = np.array([1, 2, 3])
x = np.array([4, 5, 6])

dot = np.dot(w, x)            # 1*4 + 2*5 + 3*6
print("w =", w)
print("x =", x)
print("dot product w·x =", dot, "  (manual check:", 1*4 + 2*5 + 3*6, ")")

# Norms: measure a vector's "length"
v = np.array([3, 4])
l2 = np.linalg.norm(v)                 # default = 2-norm: sqrt(3^2 + 4^2) = 5
l1 = np.linalg.norm(v, ord=1)          # 1-norm: |3| + |4| = 7
print("\nvector v =", v)
print("2-norm (Euclidean) ||v||_2 =", l2)
print("1-norm (taxicab)    ||v||_1 =", l1)

# Visual: a straight-line walk vs a grid walk for v = [3, 4]
fig, ax = plt.subplots()
ax.plot([0, v[0]], [0, v[1]], color="tab:blue", lw=2, label="2-norm path (straight)")
ax.plot([0, v[0], v[0]], [0, 0, v[1]], color="tab:orange", lw=2, ls="--",
        label="1-norm path (taxicab)")
ax.scatter(*v, color="black", zorder=5)
ax.text(v[0] + 0.1, v[1] + 0.1, "v = [3, 4]", fontsize=11)
ax.set_xlim(-0.5, 4.5)
ax.set_ylim(-0.5, 5)
ax.set_title("Two ways to measure the length of v = [3, 4]")
ax.set_xlabel("dimension 1")
ax.set_ylabel("dimension 2")
ax.legend()
plt.show()

w = [1 2 3]
x = [4 5 6]
dot product w·x = 32   (manual check: 32 )

vector v = [3 4]
2-norm (Euclidean) ||v||_2 = 5.0
1-norm (taxicab)    ||v||_1 = 7.0

Reading the output

The dot product of $\mathbf{w}=[1,2,3]$ and $\mathbf{x}=[4,5,6]$ is $1\cdot4 + 2\cdot5 + 3\cdot6 = 32$. For $\mathbf{v}=[3,4]$, the 2-norm is $5$ (the straight-line length, matching the Pythagorean triple 3-4-5), while the 1-norm is $7$ (the longer taxi route: 3 across + 4 up). Same vector, two different notions of "length" — both show up in machine learning.

2.2 Random Variables

A random variable (written as a capital italic letter like $X$) is a variable whose values come from some random process. There are two flavors:

• Discrete: takes a countable set of values (a die roll: 1–6; or a color: red / yellow / blue).
• Continuous: takes values anywhere in an interval (height, weight, time).

A random variable's behavior is described by its probability distribution:

• For a discrete variable we use a probability mass function (PMF) — a list like $\Pr(X=\text{red})=0.3$, $\Pr(X=\text{yellow})=0.45$, $\Pr(X=\text{blue})=0.25$. Every probability is $\ge 0$ and they all sum to $1$.
• For a continuous variable we use a probability density function (PDF) — a curve where the area under the curve in any region gives the probability of landing there. The total area under the whole curve is $1$. (For any single exact value the probability is $0$, which is why we talk about areas, not heights.)

Two summary numbers describe a distribution:

• The mean (also called the expected value or expectation), written $\mu = \mathbb{E}[X]$, is the "center of mass." For a discrete variable: $\mathbb{E}[X] = \sum_i x_i \Pr(X=x_i)$.
• The variance $\text{Var}(X) = \mathbb{E}\bigl[(X-\mu)^2\bigr]$ (and its square root, the standard deviation $\sigma$) measures how spread out the values are.

Analogy: the mean is where the distribution would balance on your fingertip; the standard deviation is how wide it wobbles.

Most of the time we don't know the true distribution — we only see a sample (a dataset) of observed values. The next two sections build on exactly that situation.

# A continuous random variable: X ~ Normal(mean=5, std=2)
true_mean, true_std = 5.0, 2.0
rng = np.random.default_rng(42)

# Draw a sample of 5000 values
sample = rng.normal(loc=true_mean, scale=true_std, size=5000)

fig, axes = plt.subplots(1, 2, figsize=(11, 4))

# Left: a histogram of the samples approximates the PDF
axes[0].hist(sample, bins=40, density=True, color="tab:blue", alpha=0.7)
axes[0].axvline(true_mean, color="tab:red", ls="--", label=f"true mean = {true_mean}")
axes[0].set_title("Histogram of samples (approximates the PDF)")
axes[0].set_xlabel("value")
axes[0].set_ylabel("density")
axes[0].legend()

# Right: the running sample mean homes in on the true mean (Law of Large Numbers)
n_vals = np.arange(1, len(sample) + 1)
running_mean = np.cumsum(sample) / n_vals
axes[1].plot(n_vals, running_mean, color="tab:blue", lw=1, label="sample mean so far")
axes[1].axhline(true_mean, color="tab:red", ls="--", label=f"true mean = {true_mean}")
axes[1].set_title("Sample mean approaches the true mean as n grows")
axes[1].set_xlabel("number of samples so far (n)")
axes[1].set_ylabel("running mean")
axes[1].legend()
plt.tight_layout()
plt.show()

print(f"Sample mean over all 5000 draws: {sample.mean():.3f}   (true mean: {true_mean})")
print(f"Sample std : {sample.std():.3f}   (true std : {true_std})")

Sample mean over all 5000 draws: 4.960   (true mean: 5.0)
Sample std : 1.999   (true std : 2.0)

Reading the plots

On the left, the histogram is a rough, blocky picture of the bell-shaped PDF — the more samples we draw, the closer the bars hug the true curve. On the right, the running sample mean bounces around wildly at first (a single draw can drag it far from 5) but settles onto the true mean $5$ as $n$ grows. This settling is the Law of Large Numbers, and it's the reason datasets work: with enough examples, sample statistics reveal the true distribution.

2.3 Unbiased Estimators

Since we usually can't see the true distribution, we estimate its statistics from a sample $\mathcal{S}_X = \{x_1, \dots, x_N\}$. An estimator $\hat{\theta}$ is a formula that turns a sample into a guess for some statistic $\theta$ (such as the mean $\mu$).

We call $\hat{\theta}$ an unbiased estimator of $\theta$ if, averaged over all possible samples, it lands on the true value:

$$\mathbb{E}\bigl[\hat{\theta}(\mathcal{S}_X)\bigr] = \theta.$$

In plain words: the estimator isn't systematically too high or too low. If you could draw infinitely many samples and average their $\hat{\theta}$'s, you'd hit $\theta$ exactly.

Good news: the sample mean $\hat{\mu} = \frac{1}{N}\sum_{i=1}^N x_i$ is an unbiased estimator of the true mean. A subtler point: the sample variance is only unbiased when you divide by $N-1$, not $N$. Dividing by $N$ gives a value that is systematically too small — a biased estimator. Let's see both effects in code.

# Show the sample MEAN is unbiased, and that variance is biased (÷N) vs unbiased (÷N-1)
true_mean, true_std = 5.0, 2.0
true_var = true_std ** 2
rng = np.random.default_rng(7)

N = 10                 # each sample is small
repeats = 8000         # but we take many of them
sample_means = np.empty(repeats)
biased_vars = np.empty(repeats)
unbiased_vars = np.empty(repeats)

for r in range(repeats):
    s = rng.normal(loc=true_mean, scale=true_std, size=N)
    sample_means[r] = s.mean()
    biased_vars[r] = s.var(ddof=0)       # divide by N      -> biased
    unbiased_vars[r] = s.var(ddof=1)     # divide by (N-1)  -> unbiased

print(f"true mean = {true_mean},   true variance = {true_var}")
print(f"avg of sample means       = {sample_means.mean():.4f}   (unbiased: matches true mean)")
print(f"avg of biased  variance   = {biased_vars.mean():.4f}   (too low)")
print(f"avg of unbiased variance  = {unbiased_vars.mean():.4f}   (matches true variance)")

fig, ax = plt.subplots()
ax.hist(sample_means, bins=40, density=True, color="tab:blue", alpha=0.7)
ax.axvline(true_mean, color="tab:red", ls="--", label=f"true mean = {true_mean}")
ax.axvline(sample_means.mean(), color="tab:green", ls=":",
           label=f"avg sample mean = {sample_means.mean():.3f}")
ax.set_title("Distribution of the sample mean over many small samples")
ax.set_xlabel("sample mean value")
ax.set_ylabel("density")
ax.legend()
plt.show()

true mean = 5.0,   true variance = 4.0
avg of sample means       = 4.9972   (unbiased: matches true mean)
avg of biased  variance   = 3.5897   (too low)
avg of unbiased variance  = 3.9886   (matches true variance)

Reading the output

Even though each individual sample of size 10 has a noisy mean, the average of all 8000 sample means lands right on 5 — that's what "unbiased" means. For variance, the picture is different: dividing by $N$ systematically underestimates the spread (you'll see a number below 4), while dividing by $N-1$ corrects it back to the true variance of 4. This is why libraries like numpy let you choose ddof ("delta degrees of freedom").

2.4 Bayes' Rule

Conditional probability $\Pr(X=x \mid Y=y)$ reads as "the probability that $X=x$ given that $Y=y$ has already happened." Bayes' rule lets you flip that condition around:

$$\Pr(X=x \mid Y=y) = \frac{\Pr(Y=y \mid X=x)\,\Pr(X=x)}{\Pr(Y=y)}.$$

People remember it as: posterior = (likelihood × prior) / evidence. We often drop the denominator and just write the proportionality:

$$\Pr(y \mid x) \;\propto\; \Pr(x \mid y)\,\Pr(y).$$

This is enormously useful when it's easier to measure "how likely is this evidence if the cause were $y$" than to measure "how likely is the cause $y$ given the evidence" directly — Bayes' rule bridges that gap.

Analogy: you hear footsteps at night (evidence). Footsteps are very likely if your roommate is home (high likelihood) and your roommate is usually home (high prior), but unlikely for a burglar — so Bayes' rule says "probably the roommate."

# The classic base-rate example: a rare disease and a pretty good test
p_disease = 0.01                       # prior: 1% of people have the disease
p_no_disease = 1 - p_disease
p_pos_given_disease = 0.99             # sensitivity (true positive rate)
p_pos_given_no_disease = 1 - 0.95      # 1 - specificity = false positive rate (5%)

# Evidence: overall probability of getting a positive test
p_positive = (p_pos_given_disease * p_disease
              + p_pos_given_no_disease * p_no_disease)

# Bayes' rule: probability you actually have the disease given a positive test
p_disease_given_pos = (p_pos_given_disease * p_disease) / p_positive

print("Given a POSITIVE test:")
print(f"  P(disease)        = {p_disease:.2f}       (prior)")
print(f"  P(+ | disease)    = {p_pos_given_disease:.2f}       (likelihood / sensitivity)")
print(f"  P(+)              = {p_positive:.4f}    (evidence)")
print(f"  P(disease | +)    = {p_disease_given_pos:.4f}  <-- posterior")
print(f"\nSo a positive test means only a {p_disease_given_pos*100:.1f}% chance you're actually sick.")

fig, ax = plt.subplots(figsize=(5, 3.5))
ax.bar(["prior\nP(disease)", "posterior\nP(disease | +)"],
       [p_disease, p_disease_given_pos], color=["tab:gray", "tab:red"])
ax.set_ylim(0, 1)
ax.set_ylabel("probability")
ax.set_title("Bayes' rule: a 1% prior becomes a 16% posterior")
for i, v in enumerate([p_disease, p_disease_given_pos]):
    ax.text(i, v + 0.02, f"{v*100:.1f}%", ha="center")
plt.show()

Given a POSITIVE test:
  P(disease)        = 0.01       (prior)
  P(+ | disease)    = 0.99       (likelihood / sensitivity)
  P(+)              = 0.0594    (evidence)
  P(disease | +)    = 0.1667  <-- posterior

So a positive test means only a 16.7% chance you're actually sick.

Reading the result

The posterior is only about 16%, even with a test that's 99% sensitive and 95% specific! That feels shocking, but Bayes' rule explains it: because the disease is rare (1% prior), most positive tests come from the large healthy population producing false positives. Forgetting the prior and over-trusting the test is called the base-rate fallacy — and it's exactly the trap Bayes' rule helps us avoid.

2.5 Parameter Estimation — MLE and MAP

Often we assume the data comes from a known family of distributions (say a Gaussian) but we don't know its parameters $\theta$ (e.g. $\mu$ and $\sigma$). Parameter estimation is the job of guessing good values of $\theta$ from data.

• Maximum Likelihood Estimation (MLE): pick the $\theta$ that makes the observed data most probable:

$$\theta_{\text{MLE}} = \arg\max_\theta \prod_{i=1}^N \Pr(x_i \mid \theta).$$

We usually take logs and maximize the log-likelihood instead — it turns the product into a sum (easier math) and avoids multiplying many tiny probabilities into a number too small for the computer to store.

• Maximum A Posteriori (MAP): like MLE, but we also hold a prior belief $\Pr(\theta)$ about which parameters are plausible. We pick the $\theta$ that maximizes the posterior:

$$\theta_{\text{MAP}} = \arg\max_\theta \;\Pr(x \mid \theta)\,\Pr(\theta).$$

With a flat (uniform) prior, MAP collapses to MLE. With a strong prior, MAP pulls the estimate toward what we believed before seeing any data.

Analogy: MLE trusts only the data. MAP trusts the data and your prior hunch, blended together. With almost no data, the prior dominates; with tons of data, the data dominates and the two agree.

# Estimate the mean of a Gaussian via MLE vs MAP
true_mean, true_std = 7.0, 2.0
rng = np.random.default_rng(3)
N = 5                                      # a TINY sample, so the prior will matter
data = rng.normal(loc=true_mean, scale=true_std, size=N)

# MLE for the mean of a Gaussian = the sample mean
mle = data.mean()

# MAP with a Gaussian prior on the mean: prior ~ N(prior_mean, prior_sd^2)
prior_mean, prior_sd = 0.0, 3.0
# For a Gaussian likelihood + Gaussian prior, the MAP estimate is a
# precision-weighted average of the MLE and the prior mean:
lik_prec = N / (true_std ** 2)             # how strongly the data speaks
prior_prec = 1 / (prior_sd ** 2)           # how strongly the prior speaks
map_est = (lik_prec * mle + prior_prec * prior_mean) / (lik_prec + prior_prec)

print(f"true mean             = {true_mean}")
print(f"sample (N={N})         = {np.round(data, 2)}")
print(f"MLE mean (data only)  = {mle:.3f}")
print(f"MAP mean (+ prior)    = {map_est:.3f}   (pulled toward prior mean {prior_mean})")

# Plot the log-likelihood and log-posterior across candidate means
grid = np.linspace(-2, 10, 300)
loglik = -0.5 * np.sum((data[None, :] - grid[:, None]) ** 2 / true_std ** 2, axis=1)
logpost = loglik - 0.5 * ((grid - prior_mean) ** 2 / prior_sd ** 2)

fig, ax = plt.subplots()
ax.plot(grid, loglik - loglik.max(), color="tab:blue", label="log-likelihood (MLE)")
ax.plot(grid, logpost - logpost.max(), color="tab:orange", label="log-posterior (MAP)")
ax.axvline(mle, color="tab:blue", ls="--", label=f"MLE = {mle:.2f}")
ax.axvline(map_est, color="tab:orange", ls="--", label=f"MAP = {map_est:.2f}")
ax.axvline(prior_mean, color="tab:green", ls=":", label=f"prior mean = {prior_mean}")
ax.set_title("MLE vs MAP for the mean of a Gaussian (tiny sample)")
ax.set_xlabel("candidate mean μ")
ax.set_ylabel("log value (normalized)")
ax.legend()
plt.show()

true mean             = 7.0
sample (N=5)         = [11.08  1.89  7.84  5.86  6.09]
MLE mean (data only)  = 6.553
MAP mean (+ prior)    = 6.018   (pulled toward prior mean 0.0)

Reading the plot

The blue curve (log-likelihood) peaks at the MLE, which is just the sample mean of our five draws. The orange curve (log-posterior) is the likelihood tilted by the prior centered at $0$, so its peak — the MAP estimate — sits between the sample mean and the prior mean. With only five data points the prior still has real pull; collect hundreds of points and the two curves would peak in almost the same spot, because the data would overwhelm the prior.

2.6 Classification vs Regression

Supervised learning splits into two big families depending on what the label $y$ looks like:

	Classification	Regression
Label type	a category from a finite set	a real number
Example	spam / not-spam; dog / cat / bird	house price; temperature; wait time
Output	a class label (or a probability per class)	a continuous value
Variants	binary (2 classes) or multiclass (3+)	—

When there are exactly two classes it's binary classification; with three or more it's multiclass classification.

Analogy: classification answers "which bucket?" (sorting mail into bins). Regression answers "how much?" (appraising a house's price).

from sklearn.linear_model import LogisticRegression, LinearRegression
from sklearn.datasets import make_classification, make_regression

SEED = 42

# --- Classification: labels are categories (0 / 1) ---
Xc, yc = make_classification(n_samples=120, n_features=2, n_redundant=0,
                             n_informative=2, n_clusters_per_class=1,
                             random_state=SEED)
clf = LogisticRegression(max_iter=200)
clf.fit(Xc, yc)

# --- Regression: labels are real numbers ---
Xr, yr = make_regression(n_samples=120, n_features=1, noise=15.0, random_state=SEED)
reg = LinearRegression()
reg.fit(Xr, yr)

fig, axes = plt.subplots(1, 2, figsize=(11, 4))

# Left: classification with a straight decision boundary
axes[0].scatter(Xc[:, 0][yc == 0], Xc[:, 1][yc == 0], color="tab:blue", label="class 0")
axes[0].scatter(Xc[:, 0][yc == 1], Xc[:, 1][yc == 1], color="tab:orange", label="class 1")
xx = np.linspace(Xc[:, 0].min(), Xc[:, 0].max(), 50)
w0, w1 = clf.coef_[0]
b = clf.intercept_[0]
axes[0].plot(xx, -(w0 * xx + b) / w1, "k-", label="decision boundary")
axes[0].set_title("Classification: LogisticRegression (label = category)")
axes[0].set_xlabel("feature 1")
axes[0].set_ylabel("feature 2")
axes[0].legend()

# Right: regression with a fitted line
axes[1].scatter(Xr.ravel(), yr, color="tab:blue", alpha=0.7, label="data")
axes[1].plot(Xr.ravel(), reg.predict(Xr), color="tab:red", lw=2, label="fitted line")
axes[1].set_title("Regression: LinearRegression (label = number)")
axes[1].set_xlabel("feature")
axes[1].set_ylabel("target (real number)")
axes[1].legend()
plt.tight_layout()
plt.show()

Reading the plots

On the left, logistic regression draws a decision boundary — a line that separates "class 0" from "class 1"; new points are labeled by which side of the line they fall on. On the right, linear regression fits a line through the cloud of points so it can output a real number for any input. Same general idea (learn from labeled examples), but the kind of answer differs: a category versus a quantity.

2.7 Model-Based vs Instance-Based Learning

• Model-based learning uses the training data to build a compact model with learned parameters (the weights $\mathbf{w}$ and bias $b$ in SVM or logistic regression, for example). Once trained, the model is just a small formula and the training data can be thrown away. Prediction = plug the new input into the formula.
• Instance-based learning keeps the entire dataset as "the model." To predict a new input, it looks at the training examples most similar to it. The classic example is k-Nearest Neighbors (k-NN): to label a new point, find its $k$ nearest training points and take a majority vote of their labels.

	Model-based	Instance-based
What it stores	a small set of parameters	the whole dataset
Prediction cost	cheap (one formula)	costlier (compare to all data)
Boundary shape	fixed by the model (e.g. a line)	flexible / local

Analogy: model-based is a student who studies, writes a short cheat-sheet, then throws away the notes. Instance-based is a student who brings the whole textbook to the exam and looks up the closest similar problem.

from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_moons

# A dataset that is NOT linearly separable: two interleaving moons
X, y = make_moons(n_samples=200, noise=0.20, random_state=5)

# Model-based: logistic regression (keeps only w, b at predict time)
log = LogisticRegression(max_iter=200)
log.fit(X, y)

# Instance-based: k-NN (keeps the whole dataset)
knn = KNeighborsClassifier(n_neighbors=5, n_jobs=1)
knn.fit(X, y)

# Build a grid so we can color the decision regions
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 150),
                     np.linspace(y_min, y_max, 150))
grid = np.c_[xx.ravel(), yy.ravel()]

fig, axes = plt.subplots(1, 2, figsize=(11, 4))
for ax, model, title in [(axes[0], log, "Model-based: LogisticRegression"),
                         (axes[1], knn, "Instance-based: k-NN (k=5)")]:
    Z = model.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.3, cmap="coolwarm")
    ax.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue",
               edgecolor="k", s=30, label="class 0")
    ax.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:red",
               edgecolor="k", s=30, label="class 1")
    ax.set_title(title)
    ax.set_xlabel("feature 1")
    ax.set_ylabel("feature 2")
    ax.legend()
plt.tight_layout()
plt.show()

Reading the plots

Logistic regression (left) can only draw a straight boundary, so it misclassifies the curved moons. k-NN (right) needs no formula — it just votes among nearby points — so its boundary bends and wiggle to follow the moons. The trade-off: k-NN must carry the entire dataset to every prediction, while logistic regression only needs its handful of learned weights.

2.8 Shallow vs Deep Learning

• Shallow learning learns its parameters directly from the input features. Logistic regression, SVMs, decision trees, k-NN — almost everything we've seen so far — are shallow. The model is essentially one step: features → parameters → output.
• Deep learning stacks layers of tiny models (called neurons). Each layer transforms the previous layer's output into new, more abstract features. Most parameters are learned not from the raw features but from the outputs of the layers below. A network with more than one hidden layer between input and output is a deep neural network.

Analogy: shallow learning is a one-step recipe (mix ingredients → dish). Deep learning is an assembly line (raw parts → sub-assemblies → sub-sub-assemblies → final product), where each station learns to improve what the previous station handed it.

Don't worry if "layers" and "neurons" feel fuzzy now — we go hands-on with neural networks in Chapter 6. For now, let's just see the difference: a shallow model stuck with a straight boundary versus a deep model that can bend it.

from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons

# Same two-moons dataset (curved, not linearly separable)
X, y = make_moons(n_samples=200, noise=0.20, random_state=5)

# Shallow: logistic regression -> straight-line boundary
shallow = LogisticRegression(max_iter=200)
shallow.fit(X, y)

# Deep: a small multi-layer perceptron (2 hidden layers) -> can bend the boundary
deep = MLPClassifier(hidden_layer_sizes=(10, 10), max_iter=500,
                     random_state=5)
deep.fit(X, y)

x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 150),
                     np.linspace(y_min, y_max, 150))
grid = np.c_[xx.ravel(), yy.ravel()]

fig, axes = plt.subplots(1, 2, figsize=(11, 4))
for ax, model, title in [(axes[0], shallow, "Shallow: LogisticRegression (linear)"),
                         (axes[1], deep, "Deep: MLPClassifier (2 hidden layers)")]:
    Z = model.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.3, cmap="coolwarm")
    ax.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue",
               edgecolor="k", s=30, label="class 0")
    ax.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:red",
               edgecolor="k", s=30, label="class 1")
    ax.set_title(title)
    ax.set_xlabel("feature 1")
    ax.set_ylabel("feature 2")
    ax.legend()
plt.tight_layout()
plt.show()

print(f"Shallow training accuracy: {shallow.score(X, y):.2f}")
print(f"Deep training accuracy   : {deep.score(X, y):.2f}")

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (500) reached and the optimization hasn't converged yet.
  warnings.warn(

Shallow training accuracy: 0.88
Deep training accuracy   : 0.94

Reading the plots

The shallow logistic model is stuck with a straight boundary, so it slices through the moons and gets many points wrong. The deep MLP bends its boundary into a curve that hugs the two moons — and you can see the payoff in the accuracy numbers. That ability to learn features from previous layers is exactly what makes deep learning powerful for complex data like images and text.

Key Takeaways

• A scalar is one number, a vector is an ordered list, a matrix is a grid, a set is unordered, and $\mathbb{R}$ is all real numbers; the feature vector $\mathbf{x}$ and label $y$ are supervised learning's basic objects.
• The dot product combines two same-length vectors into one number; a norm measures length (2-norm = straight-line distance, 1-norm = taxicab distance).
• A random variable has a PMF (discrete) or PDF (continuous); its mean $\mu$ and variance $\sigma^2$ summarize it, and the sample mean converges to the true mean as the dataset grows.
• An unbiased estimator is right on average — the sample mean is unbiased, and sample variance needs the $N-1$ correction to be unbiased.
• Bayes' rule flips conditionals (posterior $\propto$ likelihood × prior); a rare disease plus a decent test still yields a low posterior — the base-rate fallacy.
• MLE maximizes data likelihood; MAP adds a prior and maximizes the posterior (MAP reduces to MLE under a flat prior, and the prior matters most when data is scarce).
• Classification predicts a category, regression predicts a number; model-based learning compresses data into parameters while instance-based (k-NN) keeps the data; shallow maps features → output directly while deep stacks layers that build features from previous layers.

What's Next

In Chapter 3 — Fundamental Algorithms, we'll meet our first real supervised learning algorithms in detail — including linear regression and logistic regression — and watch them turn these definitions into working, trainable models.

---

---

Chapter 3 — Fundamental Algorithms

Now that we have our vocabulary and notation from Chapter 2, it's time to meet the workhorses. This chapter introduces five classic learning algorithms that every machine learning practitioner should know. Some are powerful on their own; others are the building blocks behind the most effective modern methods.

In this chapter you will learn:

• How linear regression fits a line (or hyperplane) to data, and why it has a one-shot math solution
• Why logistic regression is actually a classification model built on the sigmoid curve
• How decision trees split data using impurity, and why deep trees overfit
• How support vector machines maximize the margin and use the kernel trick for non-linear data
• How k-nearest neighbors classifies by looking at the closest training examples, and how k controls overfitting

3.1 Linear Regression

The idea. We have a set of labeled examples, each one a pair (x, y) where x is a feature vector and y is a real number (not a class label this time). We want a model that predicts y from x. Linear regression assumes the prediction is a linear combination of the features:

f(x) = w · x + b

Here w is a vector of weights (one per feature), " · " means a weighted sum, and b is a bias (intercept) number. The model is "parametrized" by w and b — once we find good values for them, we have our predictor.

The goal is different from the SVM we met in Chapter 1. There, the hyperplane was a decision boundary placed as far as possible from both classes. Here, the hyperplane should sit as close as possible to all the training points, so that when we read off a prediction for a new x, it lands near the true y.

Everyday analogy. Plot people's heights against their shoe size and you'll see an upward trend. Linear regression draws the single straight line through the "middle" of that cloud of dots — the line that, on average, is closest to every point.

The squared-error loss and empirical risk

How do we measure "close to all points"? We need a loss function — a penalty for a wrong prediction. Linear regression uses the squared error loss: (f(x) − y)^2. We square the difference between the prediction and the true target. Squaring does two nice things: it makes every penalty positive (so over- and under-predictions don't cancel out), and it punishes big errors much more than small ones.

The overall cost function is the average loss over all training examples, also called the empirical risk: the mean of (f(x_i) − y_i)^2 across all i. We want the w and b that make this average as small as possible.

Why a square and not the plain absolute value? The square is smooth — it has a continuous derivative everywhere — which lets us solve for the best w and b with simple algebra (a "closed-form" solution) instead of a slow numerical search.

The normal equations: a closed-form solution

Here's the beautiful part. Because the squared-error cost is a smooth, bowl-shaped function of w and b, its minimum sits exactly where the slope (gradient) is zero. Setting the derivatives to zero gives a system of linear equations called the normal equations, which we can solve directly with linear algebra:

w = (X^T X)^(−1) X^T y

(X is the matrix of all feature vectors with a column of 1s appended for the bias, ^T means transpose, and ^(−1) means inverse.) This is a closed-form solution: plug in the data, do one matrix inversion and two multiplications, and you're done — no looping, no learning rate, no epochs. That's a luxury most algorithms don't have.

scikit-learn's LinearRegression uses exactly this math under the hood. Below we'll do it both ways — let sklearn fit the line, then recompute w and b ourselves with numpy — and confirm they agree.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression

# Consistent, compact figures for the whole notebook
plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# Step 1 -- a small 1-feature regression dataset (100 points, one target)
X, y = make_regression(n_samples=100, n_features=1, noise=15.0, random_state=42)

# Step 2 -- fit a linear regression model (uses the normal equations internally)
model = LinearRegression()
model.fit(X, y)

# The learned parameters: w (slope) and b (intercept)
w = model.coef_[0]
b = model.intercept_
print(f"Learned slope     w = {w:.3f}")
print(f"Learned intercept b = {b:.3f}")

# Step 3 -- plot the data and the fitted line
line_x = np.linspace(X.min(), X.max(), 100).reshape(-1, 1)
line_y = model.predict(line_x)

plt.scatter(X, y, s=20, color="tab:blue", label="data")
plt.plot(line_x, line_y, color="tab:orange", lw=2, label=f"fit: y = {w:.2f}x + {b:.2f}")
plt.title("Linear Regression — the line that best hugs the data")
plt.xlabel("feature x")
plt.ylabel("target y")
plt.legend()
plt.show()

Learned slope     w = 45.785
Learned intercept b = 1.748

Reading the plot

The orange line is the model f(x) = w · x + b. Notice it runs through the middle of the blue cloud — that's the line that minimizes the average squared vertical distance to all 100 points. For any new x, we just read up to the line to get our prediction.

The model printed its w (slope) and b (intercept). Now let's recompute those ourselves with the normal equation and see that we get the same numbers.

# Recompute w and b with the normal equation:  w = (X^T X)^(-1) X^T y
# We augment X with a column of 1s so the bias b is solved at the same time.
X_aug = np.hstack([X, np.ones((X.shape[0], 1))])      # shape (100, 2): [x, 1]
w_normal = np.linalg.inv(X_aug.T @ X_aug) @ X_aug.T @ y

w_ne, b_ne = w_normal[0], w_normal[1]
print(f"Normal-equation slope     w = {w_ne:.3f}")
print(f"Normal-equation intercept b = {b_ne:.3f}")
print()
print(f"Match with sklearn?  slope diff = {abs(w_ne - w):.2e},  intercept diff = {abs(b_ne - b):.2e}")

Normal-equation slope     w = 45.785
Normal-equation intercept b = 1.748

Match with sklearn?  slope diff = 4.26e-14,  intercept diff = 1.33e-15

They match — exactly

The numbers from our hand-rolled normal equation agree with scikit-learn's to many decimal places (the tiny difference is just floating-point round-off). That's the closed-form solution in action: one matrix formula, no looping, no tuning.

Because the model is so simple and has so few parameters, linear regression rarely overfits — it can't wiggle to chase every training point. (A degree-10 polynomial regression, by contrast, can bend wildly to fit the noise and then fail badly on new data — that's overfitting, which we'll tackle in Chapter 5.)

3.2 Logistic Regression

Surprise: despite the name, logistic regression is a classification algorithm, not a regression. The name comes from statistics because its math resembles linear regression's. But its job is to sort examples into classes (we'll look at the binary case: two classes).

The problem. We still compute the linear combination w · x + b. The trouble is that w · x + b ranges from minus infinity to plus infinity, while a class label is just 0 or 1. We need to squeeze that unbounded score into the range (0, 1) so we can read it as a probability.

The fix is the sigmoid (a.k.a. logistic) function: σ(z) = 1 / (1 + e^(−z)). It takes any number z and maps it to a value between 0 and 1 with an S-shape. The full model is f(x) = 1 / (1 + e^(−(w·x+b))). If f(x) ≥ 0.5 we predict class 1; otherwise class 0. (The 0.5 threshold can be tuned — we'll revisit this in Chapter 5.)

Everyday analogy. Think of the sigmoid as a "soft switch." Instead of snapping abruptly from OFF (0) to ON (1), it smoothly ramps around z = 0. A very negative score is almost surely OFF; a very positive score is almost surely ON; near zero it's a coin flip.

# Plot the sigmoid function to build intuition
z = np.linspace(-7, 7, 200)
sigmoid = 1 / (1 + np.exp(-z))

plt.plot(z, sigmoid, color="tab:green", lw=2)
plt.axhline(0.5, color="gray", ls="--", lw=1)
plt.axvline(0, color="gray", ls="--", lw=1)
plt.title("The sigmoid (logistic) function — a soft switch")
plt.xlabel("z = w·x + b  (the raw score)")
plt.ylabel("σ(z)  (probability of class 1)")
plt.ylim(-0.05, 1.05)
plt.show()

Maximum likelihood, log-loss, and the decision boundary

How do we find the best w and b? Logistic regression doesn't minimize squared error. Instead it maximizes the likelihood of the training labels — it picks parameters that make the observed labels most plausible under the model.

The likelihood of all N labels is a product (because we treat the examples as independent): for each example we take f(x_i) when y_i = 1, or (1 − f(x_i)) when y_i = 0, and multiply them all together. That f^y · (1−f)^(1−y) trick just selects the right term depending on the label.

In practice we maximize the log-likelihood (a log turns the product into a sum, which is easier to work with): the sum over all examples of [ y_i · log f(x_i) + (1 − y_i) · log(1 − f(x_i)) ]. This is equivalent to minimizing the log-loss (a.k.a. cross-entropy).

Unlike linear regression, there is no closed-form solution here — we solve it iteratively with gradient descent, which we'll meet properly in Chapter 4.

The decision boundary is where f(x) = 0.5, which is exactly where w · x + b = 0 — a straight line (or hyperplane). Let's fit a logistic regression and visualize both the boundary and the probability landscape.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

# A small 2-feature, 2-class dataset
X, y = make_classification(n_samples=200, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           class_sep=1.5, random_state=42)

# Fit logistic regression (gradient descent under the hood)
clf = LogisticRegression(max_iter=200)
clf.fit(X, y)
print(f"Training accuracy: {clf.score(X, y):.3f}")

# Build a meshgrid over the feature space to show decision boundary + probabilities
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))
grid = np.c_[xx.ravel(), yy.ravel()]
proba = clf.predict_proba(grid)[:, 1].reshape(xx.shape)

# Contour plot: color = predicted probability of class 1
plt.contourf(xx, yy, proba, levels=20, cmap="RdBu", alpha=0.7)
plt.colorbar(label="P(class 1)")
# Draw the 0.5 decision boundary in black
plt.contour(xx, yy, proba, levels=[0.5], colors="black", linestyles="--", linewidths=2)

plt.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue", edgecolor="k", label="class 0")
plt.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:orange", edgecolor="k", label="class 1")
plt.title("Logistic Regression — decision boundary & probability map")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

Training accuracy: 0.940

Reading the plot

• The dashed black line is the decision boundary — where the model is exactly 50/50. Points on the blue side are predicted class 0; on the red side, class 1.
• The color gradient shows the predicted probability of class 1. Near the orange cluster the probability is close to 1 (deep red); near the blue cluster it's close to 0 (deep blue). Right on the boundary it's around 0.5 — the model is least confident there.

So logistic regression doesn't just give a hard label; it gives a probability, which is useful when you need to weigh risks (e.g., "85% chance this email is spam" vs. just "spam").

3.3 Decision Tree Learning

A decision tree is a flowchart-like graph used to make decisions. At each branching node you inspect one feature: if its value is below a threshold you go left, otherwise you go right. You keep walking until you hit a leaf, which announces the predicted class. The neat part: a tree can be learned from data automatically.

How splits are chosen. At each node the algorithm searches every feature and every possible threshold, splits the data, and asks: did this split make the children "purer" than the parent? Impurity measures how mixed the labels are in a node. A node containing only one class is perfectly pure (impurity 0); a node split 50/50 is maximally impure.

Two common impurity measures, where p is the proportion of class 1 in the node:

• Entropy = −p · log(p) − (1−p) · log(1−p) — from information theory; measures uncertainty. It's 0 when pure and largest when 50/50.
• Gini impurity = 1 − p^2 − (1−p)^2 — the chance a randomly guessed label (using the node's class frequencies) is wrong. Also 0 when pure and largest at 50/50.

The algorithm picks the split with the biggest information gain (parent impurity minus the weighted child impurity).

Everyday analogy. Twenty Questions. Each question (split) tries to narrow the possibilities as fast as possible — you ask the question that best separates the remaining candidates.

# Compare entropy and Gini impurity as a function of the class-1 proportion p
p = np.linspace(0.001, 0.999, 200)

# Entropy (base-2 log, so it peaks at 1.0)
entropy = -p * np.log2(p) - (1 - p) * np.log2(1 - p)
# Gini impurity
gini = 1 - p**2 - (1 - p)**2

plt.plot(p, entropy, color="tab:red", lw=2, label="Entropy (base-2)")
plt.plot(p, gini, color="tab:purple", lw=2, label="Gini impurity")
plt.axvline(0.5, color="gray", ls="--", lw=1)
plt.title("Impurity peaks at 50/50 and is zero at the extremes")
plt.xlabel("proportion of class 1  (p)")
plt.ylabel("impurity")
plt.legend()
plt.show()

Reading the impurity plot + tree structure

Both curves are 0 at the extremes (a node of all one class is perfectly pure) and peak in the middle (a 50/50 node is the most mixed, the most useless for classification). The tree-growing algorithm greedily picks splits that push each node toward the pure extremes.

Why decision trees overfit. Nothing in the basic algorithm says "stop." Left unchecked, the tree keeps splitting until every leaf is pure — which means it can carve out a tiny box for every single training point, even the noisy ones. Such a tree memorizes the training set and usually fails on new data.

We control this with stopping rules (hyperparameters): limit the maximum depth d, require a minimum impurity decrease ε, or require a minimum number of samples per leaf. Restricting the depth is the simplest form of pruning. Let's see overfitting versus pruning side by side.

from sklearn.datasets import make_moons
from sklearn.tree import DecisionTreeClassifier

# A noisy, non-linear "two moons" dataset
X, y = make_moons(n_samples=200, noise=0.25, random_state=42)

# Two trees: one allowed to grow fully, one restricted to depth 3
tree_deep = DecisionTreeClassifier(random_state=42)              # no depth limit -> overfit
tree_pruned = DecisionTreeClassifier(max_depth=3, random_state=42)  # pruned

tree_deep.fit(X, y)
tree_pruned.fit(X, y)

print(f"Deep tree  : depth={tree_deep.get_depth()}, train acc={tree_deep.score(X, y):.3f}")
print(f"Pruned tree: depth={tree_pruned.get_depth()}, train acc={tree_pruned.score(X, y):.3f}")

# Meshgrid for decision boundaries
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 250),
                     np.linspace(y_min, y_max, 250))
grid = np.c_[xx.ravel(), yy.ravel()]

fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
for ax, tree, title in [(axes[0], tree_deep, "Deep tree (no limit) — overfits"),
                        (axes[1], tree_pruned, "Pruned tree (max_depth=3)")]:
    Z = tree.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, cmap="coolwarm", alpha=0.4)
    ax.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue", edgecolor="k", s=20)
    ax.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:red", edgecolor="k", s=20)
    ax.set_title(title)
    ax.set_xlabel("feature 1")
    ax.set_ylabel("feature 2")
plt.tight_layout()
plt.show()

Deep tree  : depth=8, train acc=1.000
Pruned tree: depth=3, train acc=0.910

Reading the plot

• The deep tree (left) carves the space into many small, jagged regions to surround every training point — including little islands around noisy points that are really the "wrong" class. Its training accuracy is ~1.00, but those wiggly boundaries won't generalize.
• The pruned tree (right, max_depth=3) produces a few clean, blocky cuts that capture the two-moon shape without chasing the noise. Its training accuracy is a bit lower, but it will usually do better on unseen test data.

This is the classic bias–variance trade-off: a deeper tree has lower bias (it can represent complex shapes) but higher variance (it fits the noise). Pruning trades a little bias for a big drop in variance.

3.4 Support Vector Machine

We met SVM briefly in Chapter 1. Here we fill in the two hard questions:

1. What if the data is noisy and no straight line can perfectly separate the classes?
2. What if the data is inherently non-linear (e.g., the boundary should be a circle)?

The margin. Recall that an SVM doesn't just draw any separating line — it draws the line with the widest gap (margin) between the two classes, placing it as far as possible from the nearest points (the support vectors). A bigger margin tends to generalize better.

Hard margin vs. soft margin. The original SVM (hard margin) requires perfect separation. To handle noise we introduce the hinge loss max(0, 1 − y_i (w·x_i − b)), which is 0 for points correctly classified outside the margin and grows for points on the wrong side. The soft-margin SVM minimizes a combination of a small-margin penalty and the average hinge loss:

C · ||w||^2 + (1/N) · Σ max(0, 1 − y_i (w·x_i − b))

The hyperparameter C balances the two goals:

• Large C: penalize misclassifications heavily → narrow margin, fewer training errors (risk overfitting).
• Small C: tolerate some errors → wider margin, smoother boundary (better generalization, maybe a few training mistakes).

Everyday analogy. Building a fence down the middle of a field with a few sheep that have wandered onto the wrong side. A small C says "leave a wide aisle and don't worry about a couple of stray sheep"; a large C says "build the fence tight around every sheep, even if the aisle gets skinny."

The kernel trick: separating non-linear data

What if no straight line works at all — say the classes form concentric rings? The trick: map the data into a higher-dimensional space where a flat hyperplane can separate them. A 2-D circle problem can become linearly separable in 3-D.

Doing that mapping explicitly would be expensive. The kernel trick avoids it: instead of transforming the points and then taking their dot product, we use a kernel function k(x, x') that computes the result of that dot product directly in the original space. Two favorites:

• Linear kernel k(x, x') = x · x' — no mapping; just a straight boundary.
• RBF (Gaussian) kernel k(x, x') = exp(−||x − x'||^2 / (2 σ^2)) — an effectively infinite-dimensional mapping that yields smooth, curvy boundaries. The width σ (scikit-learn's gamma) controls how curvy.

Let's compare a linear and an RBF SVM on a "circles" dataset, and see how C changes the RBF boundary.

from sklearn.datasets import make_circles
from sklearn.svm import SVC

# Concentric circles: NO straight line can separate these
X, y = make_circles(n_samples=200, noise=0.08, factor=0.5, random_state=42)

# Four models to compare: linear vs RBF, and two values of C
models = [
    ("linear, C=1", SVC(kernel="linear", C=1.0)),
    ("RBF, C=1",    SVC(kernel="rbf", C=1.0)),
    ("RBF, C=0.1",  SVC(kernel="rbf", C=0.1)),
    ("RBF, C=100",  SVC(kernel="rbf", C=100.0)),
]

x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))
grid = np.c_[xx.ravel(), yy.ravel()]

fig, axes = plt.subplots(2, 2, figsize=(11, 9))
for ax, (label, clf) in zip(axes.ravel(), models):
    clf.fit(X, y)
    Z = clf.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, cmap="coolwarm", alpha=0.4)
    ax.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue", edgecolor="k", s=18)
    ax.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:red", edgecolor="k", s=18)
    ax.set_title(f"{label}  (train acc={clf.score(X, y):.2f})")
    ax.set_xlabel("feature 1")
    ax.set_ylabel("feature 2")
plt.tight_layout()
plt.show()

Reading the plot

• The linear kernel (top-left) is hopeless on concentric circles — it can only draw a straight cut, so it gets ~50% accuracy, basically guessing.
• The RBF kernel carves out a curved (here roughly ring-shaped) boundary that hugs the inner circle.
• C = 0.1 (small) gives a smooth, generous boundary; C = 100 (large) tightens the boundary around the training points, risking overfitting to noise. The middle value, C = 1, is usually a sensible default.

So the kernel gives SVM its non-linear superpower, and C dials how aggressively it fits the training data.

3.5 k-Nearest Neighbors

kNN is delightfully simple and non-parametric: it doesn't build a compact formula and throw away the data — it keeps all the training examples in memory. When a new example x arrives, it finds the k closest training examples and takes a majority vote (for classification) or an average (for regression).

Closeness needs a distance metric. The usual choice is Euclidean distance — the straight-line distance, the square root of the summed squared differences of each feature. Other options: cosine similarity (cares about direction, not magnitude — popular for text), Chebyshev, Mahalanobis, and Hamming. The distance metric and k are hyperparameters you choose before running the algorithm.

The role of k:

• k = 1: every point gets the label of its single nearest neighbor → the boundary is super jagged and the model overfits (one noisy neighbor can flip a prediction).
• Large k: voting over many neighbors smooths the boundary, but too large and you underfit (the model just predicts the majority class everywhere).
• Odd k avoids tie votes in binary classification.

Everyday analogy. You move to a new city and want to guess whether a random house is "expensive." kNN says: look at the k houses nearest to it and take the majority opinion. Look at only one neighbor and a single oddball throws you off; look at 30 and you get the general vibe of the neighborhood.

from sklearn.neighbors import KNeighborsClassifier

# Two moons again, noisy
X, y = make_moons(n_samples=200, noise=0.25, random_state=42)

# k=1 (very flexible) vs k=15 (smooth)
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 250),
                     np.linspace(y_min, y_max, 250))
grid = np.c_[xx.ravel(), yy.ravel()]

fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
for ax, k in [(axes[0], 1), (axes[1], 15)]:
    clf = KNeighborsClassifier(n_neighbors=k)
    clf.fit(X, y)
    Z = clf.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, cmap="coolwarm", alpha=0.4)
    ax.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue", edgecolor="k", s=20)
    ax.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:red", edgecolor="k", s=20)
    ax.set_title(f"kNN  k={k}  (train acc={clf.score(X, y):.2f})")
    ax.set_xlabel("feature 1")
    ax.set_ylabel("feature 2")
plt.tight_layout()
plt.show()

Reading the plot

• k = 1 (left): each training point is surrounded by its own little colored island. The boundary is fragmented and clearly chases individual noisy points — classic overfitting.
• k = 15 (right): the boundary is smooth and captures the two-moon shape cleanly.

The catch: k = 1 has perfect training accuracy (every point is its own nearest neighbor), which is misleading. To judge fairly we must check held-out test data, not the data we trained on.

from sklearn.model_selection import train_test_split

# Split the moons into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

ks = range(1, 40, 2)            # odd k from 1 to 39
train_acc = []
test_acc = []
for k in ks:
    clf = KNeighborsClassifier(n_neighbors=k)
    clf.fit(X_train, y_train)
    train_acc.append(clf.score(X_train, y_train))
    test_acc.append(clf.score(X_test, y_test))

plt.plot(ks, train_acc, "o-", color="tab:blue", label="training accuracy")
plt.plot(ks, test_acc, "s-", color="tab:orange", label="test accuracy")
plt.title("kNN: the bias–variance trade-off as k grows")
plt.xlabel("k (number of neighbors)")
plt.ylabel("accuracy")
plt.legend()
plt.show()

Reading the plot

This curve is a textbook bias–variance picture:

• Small k (left): training accuracy is ~100%, but test accuracy is poor — the model is too flexible and overfits the noise (high variance).
• Large k (right): both accuracies drop as the model averages over too many neighbors and underfits (high bias) — eventually it just predicts the majority class.
• The sweet spot is somewhere in the middle, where test accuracy peaks. This is why k is a hyperparameter we tune (Chapter 5) rather than guess.

Notice the training and test curves separate: training accuracy is an optimistic measure. The number we actually care about is the test accuracy — how the model does on data it has never seen.

The five algorithms at a glance

Algorithm	Task	Model shape	How it's trained	Key hyperparameters
Linear regression	regression	straight line / hyperplane	closed form (normal equations)	(almost none)
Logistic regression	classification	linear boundary + sigmoid	gradient descent on log-loss	(regularization C)
Decision tree	both	axis-aligned splits	greedy impurity reduction	max depth, min split
SVM	classification	max-margin; linear or kernel	quadratic programming	C, kernel, gamma / σ
kNN	both	memory-based, no formula	none — just store data	k, distance metric

A useful rule of thumb: linear / logistic regression are your simple, fast, low-overfit baselines; decision trees are interpretable but overfit easily; SVMs with an RBF kernel are powerful for non-linear boundaries; kNN is dead-simple and needs no training, but slows down at prediction time on large datasets.

Key Takeaways

• Linear regression fits f(x) = w·x + b by minimizing the average squared error; thanks to the smooth square, it has a closed-form solution (the normal equations) — no iteration needed.
• Logistic regression is classification, not regression: the sigmoid squeezes a linear score into a probability, and training maximizes likelihood (equivalently minimizes log-loss) via gradient descent.
• Decision trees split data to reduce impurity (entropy or Gini); left uncontrolled they overfit by memorizing noise, so we prune with depth limits.
• SVM maximizes the margin; the soft-margin version with hinge loss and hyperparameter C handles noise, and the kernel trick lets it separate non-linear data without explicit high-dimensional computation.
• kNN classifies by majority vote of the k nearest training points; small k overfits, large k underfits, and we pick k by checking held-out test accuracy.
• All five illustrate the same tension: fit the training data well (low bias) without chasing the noise (low variance).

What's Next

In Chapter 4 — Anatomy of a Learning Algorithm, we'll open up the "engine" inside these models — how an algorithm is really just an objective function plus an optimizer (like gradient descent) — and see how the same recipe powers them all.

---

---

Chapter 4 — Anatomy of a Learning Algorithm

In the last chapter you met several ready-made models — linear regression, logistic regression, SVM, decision trees, and kNN. Under the hood, almost every one of them is assembled from the same handful of parts. This chapter opens up that "black box" and shows you what is inside, so that when you use a library later you actually understand what it is doing.

In this chapter you will learn:

• The four building blocks every learning algorithm is made of: a loss function, an optimization algorithm, a parameterized model, and an output.
• How gradient descent works — the single most important optimization method in machine learning — including the update rule $w \leftarrow w - \eta \nabla L$ and why the learning rate $\eta$ matters.
• How to code gradient descent from scratch with NumPy and watch it converge (and explode) on real plots.
• The typical workflow a machine learning engineer follows, from raw data to a deployed, monitored model.
• Practical differences between algorithms — speed, memory, interpretability, and the kinds of data they accept.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rcParams

# Consistent, readable figures for every plot in this notebook.
rcParams['figure.figsize'] = (7, 4)
rcParams['figure.dpi'] = 110
rcParams['axes.grid'] = True
rcParams['grid.alpha'] = 0.3

print("imports ready")

imports ready

4.1 Building Blocks of a Learning Algorithm

Pick any learning algorithm and you will find the same four ingredients inside it. Think of them as the organs that keep the algorithm alive:

1. A loss function — a way to score "how wrong is the model right now?"
2. An optimization algorithm — the engine that tweaks the model's knobs to make that score smaller.
3. A parameterized model $f(x)$ — the recipe that turns an input $x$ into a prediction, controlled by adjustable numbers called parameters (like $w$ and $b$).
4. An output (prediction) — the answer the trained model gives for a new input.

They fit together like this:

   training data  -->  (3) model f(x)  -->  prediction
        |                  ^                     |
        v                  |                     v
  (1) loss function  ------+               (4) final output
        |
        v
  (2) optimizer: tweak the parameters to shrink the loss

Everyday analogy: Imagine tuning a shower faucet. The model is "how far I turn the hot and cold handles." The loss is "how far the water temperature is from what I want." The optimizer is your hand adjusting the handles a little at a time. The output is the water that comes out.

The four ingredients in plain words

(1) The loss function. This is a single number that measures the gap between what the model predicts and what the data actually says. For regression the usual choice is the mean squared error (MSE) — the average of the squared differences between predictions and true values. A smaller loss means a better fit. Without a loss function the algorithm has no way to know whether a change is an improvement.

(2) The optimization algorithm. Once we can measure "wrongness," we need a procedure that adjusts the parameters to make it smaller. The star of this chapter is gradient descent. Some algorithms optimize an explicit criterion (linear/logistic regression, SVM); others, like decision trees and kNN, were built from intuition first and only later shown to be optimizing something.

(3) The parameterized model $f(x)$. This is the shape of the prediction rule, with numbers we get to learn. For linear regression it is $f(x) = wx + b$, where $w$ (the slope, or weight) and $b$ (the intercept, or bias) are the parameters. Different parameter values give different lines; learning means finding the values that minimize the loss.

(4) The output (prediction). After training, we feed the model a new input $x$ and it returns $f(x)$ — a number for regression, a class label (and sometimes a probability) for classification. This is the part the user actually sees.

# A toy look at the four building blocks, written out explicitly.

# (3) A parameterized model f(x) with parameters w and b
def f(x, w, b):
    return w * x + b

# (1) A loss function: how wrong is the model on some data?
def mse_loss(x, y, w, b):
    return ((y - f(x, w, b)) ** 2).mean()

x_demo = np.array([1.0, 2.0, 3.0])
y_demo = np.array([2.0, 4.1, 5.9])   # roughly y = 2x

# (2) An optimization algorithm tweaks w to reduce the loss.
#     (Here we simply try several values and see which scores lowest.)
print("Trying different values of w (with b fixed at 0.5):")
for w_try in [0.5, 1.0, 1.5, 2.0, 2.5]:
    print(f"  w={w_try}: loss = {mse_loss(x_demo, y_demo, w_try, 0.5):.3f}")

# (4) An output: once we believe w=2 is best, the model predicts for new x
print("Prediction for x=4 with w=2, b=0.5:", f(4.0, 2.0, 0.5))

Trying different values of w (with b fixed at 0.5):
  w=0.5: loss = 7.657
  w=1.0: loss = 2.857
  w=1.5: loss = 0.390
  w=2.0: loss = 0.257
  w=2.5: loss = 2.457
Prediction for x=4 with w=2, b=0.5: 8.5

Notice how the loss changes as we try different values of $w$: it drops, reaches a lowest point around $w=2$, then rises again. That bowl shape is exactly what an optimizer exploits — it walks downhill until it can't go any lower. All four building blocks are present in the cell above: the model f, the loss mse_loss, the (very simple) optimizer loop, and the prediction at the end.

4.2 Gradient Descent

Gradient descent is the optimization algorithm behind linear regression, logistic regression, SVM, and — as you'll see later — neural networks. The idea is beautifully simple.

• The gradient $\nabla L$ is a vector of partial derivatives. Each entry tells you how steeply the loss increases if you nudge one parameter a tiny bit. It points in the steepest uphill direction.
• To go downhill, we move in the opposite direction: the negative gradient.
• The learning rate $\eta$ (pronounced "eta") controls how big each step is.

Putting it together gives the update rule, applied again and again:

$$w \leftarrow w - \eta \,\nabla L$$

Read it as: "new parameter = old parameter minus a small step in the uphill direction." Because we subtract the uphill direction, we head downhill.

Everyday analogy: You are blindfolded on a hill and want to reach the bottom. You feel the slope under your feet (the gradient), then take a step in the downhill direction. A bigger learning rate means longer strides. Too small and you crawl forever; too large and you leap clear across the valley and end up higher than before.

# Gradient descent from scratch on a simple convex (bowl-shaped) loss.
# Loss: L(w, b) = (w - 3)^2 + (b + 1)^2
# Its minimum is at w = 3, b = -1, where the loss is 0.

def loss(w, b):
    return (w - 3) ** 2 + (b + 1) ** 2

# The gradient: a vector of partial derivatives (points uphill).
def grad(w, b):
    dL_dw = 2 * (w - 3)   # dL/dw
    dL_db = 2 * (b + 1)   # dL/db
    return dL_dw, dL_db

# The gradient-descent loop: start somewhere, step downhill repeatedly.
def gradient_descent(eta, epochs, w0=0.0, b0=0.0):
    w, b = w0, b0
    history = [loss(w, b)]
    for _ in range(epochs):
        gw, gb = grad(w, b)
        w = w - eta * gw    # update rule:  w <- w - eta * gradient
        b = b - eta * gb
        history.append(loss(w, b))
    return w, b, history

# Run with a sensible learning rate.
w_opt, b_opt, hist = gradient_descent(eta=0.1, epochs=60)
print(f"Converged to w={w_opt:.4f}, b={b_opt:.4f}  (true minimum: w=3, b=-1)")
print(f"Final loss: {hist[-1]:.2e}")

fig, ax = plt.subplots()
ax.plot(hist, marker='o', ms=3)
ax.set_title("Gradient descent on $(w-3)^2 + (b+1)^2$  (learning rate $\eta=0.1$)")
ax.set_xlabel("Iteration")
ax.set_ylabel("Loss")
ax.set_yscale('log')
plt.show()

Converged to w=3.0000, b=-1.0000  (true minimum: w=3, b=-1)
Final loss: 2.35e-11

The loss falls quickly at first, then slows as we approach the flat bottom of the bowl — exactly what we want. After 60 steps we land at $w \approx 3$, $b \approx -1$, the true minimum. Because this function is convex (a single bowl with one bottom), gradient descent is guaranteed to find that global minimum from any starting point, as long as $\eta$ is not too big.

# Now use a learning rate that is TOO LARGE.
# For this quadratic, eta > 1 makes each step overshoot and climb the
# opposite side -- the loss grows instead of shrinks.

w_div, b_div, hist_div = gradient_descent(eta=1.1, epochs=30)

fig, ax = plt.subplots()
ax.plot(hist_div, marker='o', color='crimson', ms=4)
ax.set_title("Too-large learning rate ($\eta=1.1$): loss EXPLODES")
ax.set_xlabel("Iteration")
ax.set_ylabel("Loss")
plt.show()

print(f"Loss went from {hist_div[0]:.2e} to {hist_div[-1]:.2e} in 30 steps")

Loss went from 1.00e+01 to 5.63e+05 in 30 steps

With $\eta = 1.1$ the steps are so large that each one overshoots the bottom and lands higher up the other side. The loss grows instead of shrinks — the algorithm has diverged. This is the central practical lesson: the learning rate is the most important knob. Pick it too small and training crawls; pick it too large and training blows up.

Gradient descent for linear regression

Now let's use the same idea to actually learn a model. Our model is $f(x) = wx + b$ and our loss is the MSE:

$$L = \frac{1}{N}\sum_{i=1}^{N}\big(y_i - (wx_i + b)\big)^2$$

Using the chain rule from calculus, the partial derivatives (the gradient entries) are:

$$\frac{\partial L}{\partial w} = \frac{1}{N}\sum_{i=1}^{N} -2x_i\big(y_i - (wx_i + b)\big), \qquad \frac{\partial L}{\partial b} = \frac{1}{N}\sum_{i=1}^{N} -2\big(y_i - (wx_i + b)\big)$$

In words: $\partial L/\partial w$ tells us how the loss changes when we wiggle the slope $w$, and $\partial L/\partial b$ does the same for the intercept $b$. We update both with the same rule, $w \leftarrow w - \eta\,\partial L/\partial w$ and $b \leftarrow b - \eta\,\partial L/\partial b$, looping over the whole dataset many times. One full pass over the data is called an epoch.

Let's code it from scratch and watch it learn.

# A tiny regression dataset: y = 1.8*x + 0.4 + small noise
np.random.seed(42)
x = np.linspace(0, 3, 30)
y = 1.8 * x + 0.4 + np.random.randn(30) * 0.15

N = len(x)
w, b = 0.0, 0.0          # start the parameters at zero
eta = 0.05               # learning rate
epochs = 200             # cap on passes over the data
losses = []

for epoch in range(epochs):
    preds = w * x + b
    error = y - preds
    # Gradients of the Mean Squared Error
    grad_w = (-2.0 / N) * np.sum(x * error)
    grad_b = (-2.0 / N) * np.sum(error)
    # Update rule
    w = w - eta * grad_w
    b = b - eta * grad_b
    # Record the loss AFTER this epoch's update
    losses.append(np.mean((y - (w * x + b)) ** 2))

print(f"Learned w={w:.3f}, b={b:.3f}  (true w=1.8, b=0.4)")
print(f"Final MSE: {losses[-1]:.4f}")

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(losses)
ax1.set_title("Training loss (MSE) vs. epoch")
ax1.set_xlabel("Epoch"); ax1.set_ylabel("MSE")
ax2.scatter(x, y, s=25, label="data")
ax2.plot(x, w * x + b, 'r-', label=f"fit: y={w:.2f}x+{b:.2f}")
ax2.set_title("Data and learned regression line")
ax2.set_xlabel("x"); ax2.set_ylabel("y"); ax2.legend()
plt.show()

Learned w=1.746, b=0.454  (true w=1.8, b=0.4)
Final MSE: 0.0155

Two things happened: the MSE curve on the left drops and levels off — that is convergence. On the right, the red line learned by gradient descent sits right on top of the data, recovering $w \approx 1.8$ and $b \approx 0.4$, which are (up to noise) the true values we baked in. We built a working learning algorithm from nothing but NumPy and the update rule.

from sklearn.linear_model import SGDRegressor, LinearRegression

X = x.reshape(-1, 1)

# Practical version 1: scikit-learn's SGD-based regressor (same idea,
# library-grade, uses mini-batches internally).
sgd = SGDRegressor(learning_rate='constant', eta0=0.05, max_iter=200,
                   penalty=None, random_state=42)
sgd.fit(X, y)

# Practical version 2: the closed-form (normal equation) solution.
lr = LinearRegression().fit(X, y)

print("From-scratch GD  :  w = {:.3f}, b = {:.3f}".format(w, b))
print("SGDRegressor     :  w = {:.3f}, b = {:.3f}".format(
    sgd.coef_[0], sgd.intercept_[0]))
print("LinearRegression :  w = {:.3f}, b = {:.3f}".format(
    lr.coef_[0], lr.intercept_))

From-scratch GD  :  w = 1.746, b = 0.454
SGDRegressor     :  w = 1.753, b = 0.462
LinearRegression :  w = 1.749, b = 0.448

All three methods agree closely — our hand-written gradient descent matches scikit-learn's SGDRegressor and the closed-form LinearRegression. That is the point: the libraries are doing the same math you just wrote, only faster and with more polish.

Stochastic gradient descent (SGD) speeds things up on large datasets by estimating the gradient from a small random batch of examples instead of the whole dataset. Several upgrades exist:

Variant	What it adds
SGD	Approximates the gradient with mini-batches — faster, slightly noisy
Momentum	Builds up velocity in a consistent direction to dampen oscillation
Adagrad	Adapts the learning rate per parameter based on gradient history
RMSprop	Adapts the rate using a moving average of squared gradients
Adam	Combines momentum + adaptive rates; the default choice for neural nets

One important reminder: gradient descent and its variants are not machine learning algorithms — they are general solvers for any minimization problem where the function has a gradient.

4.3 How Machine Learning Engineers Work

In practice you rarely code gradient descent by hand. You reach for libraries (scikit-learn being the most common) and follow a workflow that looks roughly like this:

  raw data
     |
     v
[ clean & explore ] --> [ extract / pick features ]
     |                            |
     |                            v
     |                    [ choose a model ]
     |                            |
     +<---------------------------+
     |
     v
[ split: train / validation / test ]
     |
     v
[ train  ->  tune hyperparameters on validation ]
     |
     v
[ evaluate on the test set ]
     |
     v
[ deploy to production ]
     |
     v
[ monitor performance, retrain on new data ]

A few plain-language notes on each stage:

• Clean & explore: look at the data, fix errors, handle missing values, spot obvious patterns.
• Features: decide which columns the model is allowed to see; convert categories to numbers, scale values (coming in Chapter 5).
• Choose a model: pick a family — linear model, tree, kNN, etc. — often starting simple.
• Split the data: train on one portion, tune on another (validation), and keep a third (test) untouched until the very end to get an honest score.
• Train & tune: fit the model, then adjust its hyperparameters (settings chosen by you, not learned from data) using the validation set.
• Evaluate: report the test-set score. If it is good enough, proceed.
• Deploy & monitor: put the model where it can make real predictions, then watch it. If the world changes, retrain.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

# A miniature end-to-end workflow on synthetic data.
X, y = make_classification(n_samples=300, n_features=4, n_informative=3,
                           n_redundant=0, random_state=42)

# 1) Split into train / test
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          random_state=42)

# 2) Build a pipeline: scale features, then fit a model
pipe = make_pipeline(StandardScaler(),
                     LogisticRegression(max_iter=200, n_jobs=1))

# 3) Train (fit) on the training split
pipe.fit(X_tr, y_tr)

# 4) Evaluate on the held-out test split
print(f"Train accuracy: {pipe.score(X_tr, y_tr):.3f}")
print(f"Test accuracy : {pipe.score(X_te, y_te):.3f}")

Train accuracy: 0.849
Test accuracy : 0.867

That snippet is the workflow in miniature: make data, split it, build a Pipeline that scales then classifies, fit on the train split, and score on the held-out test split. The test accuracy is the honest number we report; the gap between train and test accuracy hints at whether we are overfitting (a topic for later chapters).

4.4 Learning Algorithms' Particularities

Not all algorithms are interchangeable. In practice they differ in ways that matter when you pick one:

Property	What it means	Example
Hyperparameters	Settings you choose before training	$C$ in SVM, tree depth, learning rate $\eta$
Input data type	Categorical vs. numerical features	Trees accept "red/yellow/green"; SVM & kNN need numbers
Class weighting	Let you care more about rare classes	SVM can up-weight a minority class
Output type	Label only vs. label + probability/score	SVM/kNN give a class; logistic regression gives a probability
Retraining	Rebuild from scratch vs. update incrementally	Trees/SVM/kNN rebuild; SGD models & Naive Bayes update online
Task types	Classification only, regression only, or both	Trees, SVM, kNN do both; logistic regression classifies
Speed & memory	Training/prediction time and RAM footprint	kNN is slow at predict time; trees are fast

Two extra points worth memorizing:

• Interpretability matters when a human must justify a decision. A small decision tree or linear model is easy to read; a neural network is not.
• Data scaling is required by distance- and gradient-based methods (kNN, SVM, linear/logistic regression, neural nets). Tree-based methods mostly do not care.

Every library documents these properties, so reading the docs for an algorithm before using it is time well spent.

Key Takeaways

• Every learning algorithm is built from four parts: a loss function, an optimization algorithm, a parameterized model $f(x)$, and an output (prediction).
• Gradient descent repeatedly applies $w \leftarrow w - \eta\,\nabla L$ to walk downhill on the loss surface. The gradient points uphill, so subtracting it goes downhill.
• The learning rate $\eta$ is the most important knob: too small crawls, too large diverges. On a convex loss, a reasonable $\eta$ finds the global minimum.
• You can implement gradient descent from scratch in a few lines of NumPy and recover the same answer scikit-learn's SGDRegressor gives.
• One full pass over the data is an epoch; SGD estimates the gradient with mini-batches, and variants like Momentum, Adagrad, RMSprop, and Adam refine the idea.
• Gradient descent is an optimizer, not a learning algorithm itself — it is a general minimization tool.
• The engineer's workflow is: clean data → choose features → pick a model → split into train/validation/test → train & tune → evaluate → deploy → monitor.
• Algorithms differ in hyperparameters, accepted data types, class weighting, output scores, retraining style, speed, and interpretability — read the docs before choosing.

What's Next

Now that you know what is inside a learning algorithm, Chapter 5 — "Basic Practice" — shows you what it looks like to actually apply one to real, messy data: how to turn categories into numbers, scale features, split data properly, and pick the right model for the job.

---

---

Chapter 5 — Basic Practice

Until now we have looked at learning algorithms one at a time, mostly in their clean, mathematical form. Real projects are messier: the data arrives as raw logs, some values are missing, and you have to make dozens of practical choices before you can write model.fit(X, y). This chapter is about those choices — the everyday craft of applied machine learning.

In this chapter you will learn:

• How to turn raw data into features a model can use (one-hot encoding, binning, scaling)
• How to pick a reasonable learning algorithm for your problem
• Why you need three separate datasets: train, validation, and test
• What underfitting and overfitting look like, and how to find the "sweet spot"
• How regularization (L1 and L2) keeps models from overfitting
• How to measure a model's quality with confusion matrices, precision/recall, and cross-validation
• How to tune hyperparameters with grid search and random search

5.1 Feature Engineering

When someone hands you five years of user-interaction logs and asks "will this customer stay?", you cannot just pour the logs into a library. First you have to build a dataset: a table where each row is one example and each column is a feature — a number that describes that example.

Feature engineering is the art of transforming raw data into that table of informative features. A feature is informative (it has high predictive power) when it genuinely helps predict the label. For predicting whether a user keeps using an app, "average session length" is informative; "user's shoe size" probably is not.

A model has low bias when it predicts the training labels well. Good features are what let a model achieve low bias without cheating. The rest of this section walks through the most common feature-engineering moves.

One-Hot Encoding

Most learning algorithms want numbers, not words. If you have a categorical feature like color with values red / yellow / green, you need to convert it.

The tempting shortcut is to map red→1, yellow→2, green→3. Don't. That invents a fake ordering ("green is three times red") and most algorithms will take that order seriously and overfit to it.

Instead, split the one categorical column into one 0/1 column per category — a 1 says "this example is red", 0 says it is not:

red    -> [1, 0, 0]
yellow -> [0, 1, 0]
green  -> [0, 0, 1]

This is one-hot encoding. You add columns, but you avoid lying to the model about an order that does not exist.

A close cousin is a feature cross: combine two categorical features into one so the model can learn a rule that depends on their combination (for example, "red on a weekend" might behave differently from "red on a weekday"). You make it by concatenating the two values into a new category, then one-hot encoding that new category.

Everyday analogy: Think of a light-switch panel where each color gets its own on/off switch, instead of one dial with the numbers 1–3 painted on it.

%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# A tiny dataset: a few customers with a categorical 'color' and two numeric features.
df = pd.DataFrame({
    "color": ["red", "yellow", "green", "red", "green", "yellow"],
    "age":   [22, 31, 45, 19, 55, 38],
    "spend": [120, 80, 300, 60, 500, 200],
})
print("Before (raw data, with a categorical column):")
print(df)
print()

# One-hot encoding: turn 'color' into one binary column per category.
# BAD idea: mapping red=1, yellow=2, green=3 would invent a fake order.
# GOOD idea: give each color its own 0/1 column.
df_oh = pd.get_dummies(df, columns=["color"], prefix="is")
print("After one-hot encoding:")
print(df_oh)
print()

# A feature cross: combine 'color' with a 'is_weekend' flag so the model can
# learn rules that depend on the COMBINATION, not just each piece alone.
df2 = pd.DataFrame({
    "color":      ["red", "yellow", "green", "red", "green"],
    "is_weekend": [1, 0, 1, 0, 1],
})
df2["color_x_weekend"] = df2["color"] + "_" + df2["is_weekend"].astype(str)
crossed = pd.get_dummies(df2, columns=["color_x_weekend"], prefix="cross")
print("Feature cross: a new 0/1 column for each (color, weekend) combination:")
print(crossed)

Before (raw data, with a categorical column):
    color  age  spend
0     red   22    120
1  yellow   31     80
2   green   45    300
3     red   19     60
4   green   55    500
5  yellow   38    200

After one-hot encoding:
   age  spend  is_green  is_red  is_yellow
0   22    120     False    True      False
1   31     80     False   False       True
2   45    300      True   False      False
3   19     60     False    True      False
4   55    500      True   False      False
5   38    200     False   False       True

Feature cross: a new 0/1 column for each (color, weekend) combination:
    color  is_weekend  cross_green_1  cross_red_0  cross_red_1  cross_yellow_0
0     red           1          False        False         True           False
1  yellow           0          False        False        False            True
2   green           1           True        False        False           False
3     red           0          False         True        False           False
4   green           1           True        False        False           False

Binning

Sometimes you want to go the other way: you have a continuous number but you would rather treat it as a few categories. This is binning (or bucketing): chop the range into bins and record which bin each value falls into.

For example, instead of storing exact age as one number, you could create bins "0–5", "6–10", "11–15", and so on. Each becomes its own 0/1 indicator feature.

Why bother? It gives the model a hint that inside a bin the exact value does not matter, which can let it learn the same pattern from fewer examples.

Everyday analogy: A wine menu that groups bottles into "under $20", "$20–50", and "over $50" instead of listing every exact price — coarse buckets are often all you need.

# Binning (bucketing): turn a continuous 'age' into a few categorical bins.
ages = pd.DataFrame({"age": [3, 7, 13, 22, 31, 45, 19, 55, 38, 60]})

# Define bin edges and human-readable labels.
bins   = [0, 5, 10, 15, 25, 40, 100]
labels = ["0-5", "6-10", "11-15", "16-25", "26-40", "41+"]

ages["age_bin"] = pd.cut(ages["age"], bins=bins, labels=labels, right=True)
print("Age with its bin label:")
print(ages)
print()

# One-hot encode the bins so a learning algorithm gets numeric 0/1 features.
age_oh = pd.get_dummies(ages, columns=["age_bin"], prefix="age")
print("Binned age turned into 0/1 indicator features:")
print(age_oh)

Age with its bin label:
   age age_bin
0    3     0-5
1    7    6-10
2   13   11-15
3   22   16-25
4   31   26-40
5   45     41+
6   19   16-25
7   55     41+
8   38   26-40
9   60     41+

Binned age turned into 0/1 indicator features:
   age  age_0-5  age_6-10  age_11-15  age_16-25  age_26-40  age_41+
0    3     True     False      False      False      False    False
1    7    False      True      False      False      False    False
2   13    False     False       True      False      False    False
3   22    False     False      False       True      False    False
4   31    False     False      False      False       True    False
5   45    False     False      False      False      False     True
6   19    False     False      False       True      False    False
7   55    False     False      False      False      False     True
8   38    False     False      False      False       True    False
9   60    False     False      False      False      False     True

Normalization and Standardization

Numeric features often live on wildly different scales: age might range 0–100 while income ranges 0–1,000,000. Many algorithms train faster and more stably when all features sit in a similar small range. Two standard recipes:

• Normalization (min-max scaling): squeeze every feature into [0, 1].
  • x_norm = (x - min) / (max - min)
  • Gives a fixed bounded range, but a single outlier can crush the other values into a tiny slice.
• Standardization (z-score): shift to mean 0 and standard deviation 1.
  • x_std = (x - mean) / std
  • Handles outliers more gracefully and is the default choice for most algorithms, especially unsupervised ones.

Which to use? There is no universal answer. A common rule of thumb: prefer standardization for data near a bell curve or with outliers; use normalization otherwise. When in doubt and you have time, try both.

Everyday analogy: Normalization is like grading every test onto a 0–100 scale; standardization is like reporting each score as "how many standard deviations above the class average." Both put different things on a common footing.

from sklearn.preprocessing import MinMaxScaler

# Reuse the numeric columns from the one-hot encoded table (cell above).
numeric = df_oh[["age", "spend"]].copy()

print("Before scaling (note the very different ranges):")
print(numeric.describe().loc[["min", "max", "mean", "std"]].round(2))
print()

# 1) Normalization (min-max): squeeze every feature into [0, 1].
mm = MinMaxScaler()
normed = pd.DataFrame(mm.fit_transform(numeric), columns=numeric.columns)
print("After MinMaxScaler (everything in [0, 1]):")
print(normed.describe().loc[["min", "max", "mean"]].round(2))
print()

# 2) Standardization (z-score): mean 0, std 1.
scaler = StandardScaler()
scaled = pd.DataFrame(scaler.fit_transform(numeric), columns=numeric.columns)
print("After StandardScaler (mean ~ 0, std ~ 1):")
print(scaled.describe().loc[["min", "max", "mean", "std"]].round(2))

Before scaling (note the very different ranges):
        age   spend
min   19.00   60.00
max   55.00  500.00
mean  35.00  210.00
std   13.78  167.21

After MinMaxScaler (everything in [0, 1]):
       age  spend
min   0.00   0.00
max   1.00   1.00
mean  0.44   0.34

After StandardScaler (mean ~ 0, std ~ 1):
       age  spend
min  -1.27  -0.98
max   1.59   1.90
mean  0.00  -0.00
std   1.10   1.10

Dealing with Missing Features

Real tables have holes: someone forgot to log a value, or a sensor failed. You generally have three options:

1. Drop the rows (or columns) with missing values — fine if you have plenty of data.
2. Use an algorithm that handles missing values natively (some tree implementations do).
3. Impute — fill the holes with a sensible stand-in value.

Common imputation tricks:

• Replace a missing value with the column mean (or median).
• Replace it with a value outside the normal range, so the model can learn "missing means something unusual."
• Replace it with the midpoint of the range, so a missing value barely nudges the prediction.
• Get fancy: train a small model to predict the missing feature from the other features.

Whatever you choose, apply the same imputation at prediction time that you used during training.

from sklearn.impute import SimpleImputer

# Real data often has holes. Here a couple of values are missing (NaN).
messy = pd.DataFrame({
    "age":   [22, 31, np.nan, 19, 55, 38],
    "spend": [120, np.nan, 300, 60, 500, 200],
})
print("Data with missing values:")
print(messy)
print()

# Strategy 'mean': replace each hole with that column's average.
imp = SimpleImputer(strategy="mean")
filled = pd.DataFrame(imp.fit_transform(messy), columns=messy.columns)
print("After mean imputation (holes filled with column averages):")
print(filled.round(1))

Data with missing values:
    age  spend
0  22.0  120.0
1  31.0    NaN
2   NaN  300.0
3  19.0   60.0
4  55.0  500.0
5  38.0  200.0

After mean imputation (holes filled with column averages):
    age  spend
0  22.0  120.0
1  31.0  236.0
2  33.0  300.0
3  19.0   60.0
4  55.0  500.0
5  38.0  200.0

5.2 Learning Algorithm Selection

There is no single "best" algorithm — the right choice depends on your problem. Before reaching for a model, ask yourself a few questions:

• Does the model need to be explainable? If a non-technical audience must understand predictions, prefer simple, transparent models (linear/logistic regression, kNN, a single decision tree). If pure accuracy matters more, "black box" models (neural nets, ensembles) are often stronger.
• Does the data fit in memory? If yes, almost any algorithm is on the table. If not, look for incremental / out-of-core learners.
• How many examples and features? Neural networks and gradient boosting scale to millions of rows and features; SVMs are more modest.
• Categorical or numerical features? Some algorithms need everything numeric, so you one-hot encode first.
• Is the relationship linear? Linear/logistic regression and a linear-SVM suit linear data; nonlinear data may need kernels, ensembles, or neural nets.
• How fast must training and prediction be? Linear models predict almost instantly; kNN is slower at prediction time.

A quick cheat-sheet by problem type:

Situation	Typical first choice
Small data, need explainability	Linear / logistic regression, decision tree
Medium tabular data, max accuracy	Random forest, gradient boosting
Very large data, roughly linear	Linear / logistic regression (scaled features)
Nonlinear, a few thousand examples	SVM with RBF kernel, small neural net
Images, text, audio	(Ch. 6) neural networks / deep learning

When you cannot decide, the practical answer is: try a few on a validation set and let the numbers decide.

5.3 Three Sets: Train, Validation, Test

This is one of the most important habits in all of machine learning. Once your dataset is ready, shuffle it and split it into three pieces:

1. Training set — the biggest piece. The learning algorithm uses it to fit the model.
2. Validation set — used to choose the algorithm and tune hyperparameters. The algorithm never trains on it.
3. Test set — used once, at the very end, to estimate how the final model will do in the real world.

Why three, not one? A model that simply memorized the training data would score perfectly on the training set but be useless on new data — so we need a hold-out set to measure generalization.

Why two hold-out sets, not one? Because every time you use a hold-out set to make a decision (pick an algorithm, tweak a hyperparameter), you leak a little information into it. The validation set is allowed to be "used up" by tuning; the test set must stay untouched until the final, single assessment. A common split is 70% / 15% / 15%, though huge datasets can get away with 95% / 2.5% / 2.5%.

Everyday analogy: Training set = the practice problems you study from. Validation set = the mock exams you use to adjust your strategy. Test set = the real final exam, taken once.

from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

# A small labeled dataset: 200 examples, 2 classes, 6 features.
X, y = make_classification(n_samples=200, n_features=6, n_informative=3,
                           n_redundant=1, random_state=42)

# Step 1: peel off the TEST set first (15%) -- touch only once, at the very end.
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y)

# Step 2: split what is left into TRAIN (~70% of original) and VAL (~15%).
# 0.15 / 0.85 ~= 0.1765, so val gets ~15% of the full dataset.
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.1765, random_state=42, stratify=y_temp)

print("Three sets created from 200 examples:")
print(f"  train: {X_train.shape[0]} examples  (used to FIT the model)")
print(f"  val:   {X_val.shape[0]} examples    (used to CHOOSE model / hyperparameters)")
print(f"  test:  {X_test.shape[0]} examples   (used ONCE to assess the final model)")

Three sets created from 200 examples:
  train: 139 examples  (used to FIT the model)
  val:   31 examples    (used to CHOOSE model / hyperparameters)
  test:  30 examples   (used ONCE to assess the final model)

5.4 Underfitting and Overfitting

Two things can go wrong with a model, and they are opposite extremes:

• Underfitting (high bias): the model is too simple for the data, so it does poorly even on the training set. A straight line trying to follow a curvy sine wave is a classic underfit. Fix: use a more powerful model, or engineer better features.
• Overfitting (high variance): the model is too complex — it memorizes the training data, noise and all, and does great on it but poorly on the validation/test sets. A wiggly degree-15 polynomial through a noisy sine is a classic overfit. Fix: simplify the model, get more data, reduce features, or regularize.

The name "variance" comes from statistics: if you had drawn a different training set, an overfit model would change dramatically. That sensitivity to the exact training sample is exactly why it fails on new data.

Our goal is the sweet spot in between — complex enough to capture the real pattern, simple enough to ignore the noise. The next cell makes this visible by fitting polynomials of increasing degree to a noisy sine wave and watching the training and validation errors.

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error

# A noisy sine wave: the TRUE relationship is y = sin(x), hidden under noise.
rng = np.random.RandomState(42)
x = np.sort(rng.rand(40) * 2 * np.pi)            # 40 points spread over [0, 2*pi)
y = np.sin(x) + rng.normal(0, 0.2, size=x.size)  # signal + noise

# Split: 25 for training, 15 for validation.
x_tr, x_va, y_tr, y_va = train_test_split(x, y, test_size=15, random_state=42)
x_tr2 = x_tr[:, None]   # sklearn wants a 2-D X
x_va2 = x_va[:, None]

# Try polynomials of degree 0 (constant) up to 15 (very wiggly).
degrees = range(0, 16)
train_err, val_err = [], []
for d in degrees:
    m = make_pipeline(PolynomialFeatures(d), LinearRegression())
    m.fit(x_tr2, y_tr)
    train_err.append(mean_squared_error(y_tr, m.predict(x_tr2)))
    val_err.append(mean_squared_error(y_va, m.predict(x_va2)))

# --- Plot 1: error vs degree (the bias-variance picture) ---
plt.figure(figsize=(7, 4))
plt.plot(list(degrees), train_err, "o-", label="train error", color="tab:blue")
plt.plot(list(degrees), val_err, "s-", label="validation error", color="tab:orange")
plt.xlabel("polynomial degree  (model complexity -->)")
plt.ylabel("mean squared error")
plt.title("Underfitting vs. Overfitting: find the sweet spot")
plt.legend()
plt.show()

best_d = list(degrees)[int(np.argmin(val_err))]
print(f"Sweet spot (lowest validation error): degree {best_d}")
print(f"  train MSE = {train_err[best_d]:.4f}   val MSE = {val_err[best_d]:.4f}")

# --- Plot 2: three concrete fits -- underfit, good, overfit ---
grid = np.linspace(0, 2 * np.pi, 200)[:, None]
fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))
for ax, d, title in zip(axes, [1, best_d, 15],
                        ["Underfit (deg 1)", f"Good fit (deg {best_d})", "Overfit (deg 15)"]):
    m = make_pipeline(PolynomialFeatures(d), LinearRegression())
    m.fit(x_tr2, y_tr)
    ax.scatter(x_tr, y_tr, s=20, color="tab:blue", label="train")
    ax.scatter(x_va, y_va, s=20, color="tab:orange", label="val", marker="s")
    ax.plot(grid, m.predict(grid), color="black", lw=1.5, label="model")
    ax.plot(grid, np.sin(grid), color="green", ls="--", lw=1, label="true sin(x)")
    ax.set_title(title); ax.set_xlabel("x"); ax.set_ylim(-2, 2)
axes[0].set_ylabel("y")
axes[0].legend(fontsize=7)
plt.tight_layout()
plt.show()

Sweet spot (lowest validation error): degree 4
  train MSE = 0.0344   val MSE = 0.0332

Reading the plots

• In the error-vs-degree plot, training error (blue) keeps dropping as the polynomial gets more flexible — more complexity always fits the training data better.
• Validation error (orange) drops at first, then turns back up. The bottom of that orange curve is the sweet spot: the model is just complex enough.
• To the left of the sweet spot the model underfits (too stiff). To the right it overfits (too wiggly, chasing noise).
• The three fit plots show this concretely: degree 1 is a near-flat line that misses the wave; the best degree traces the sine nicely; degree 15 contorts itself through every noisy point and would fail badly on new data.

5.5 Regularization

Even a simple linear model can overfit when you have many features but few examples: it assigns non-zero weights to almost every feature, hunting for spurious patterns. Regularization adds a penalty to the learning objective so the model is nudged toward smaller, simpler weights — a bit more bias, a lot less variance. This trade-off is the famous bias–variance tradeoff.

For linear regression, instead of only minimizing the mean squared error, we minimize:

error + C * penalty

where C controls how strongly we penalize complexity. Two flavors of penalty:

• L2 (Ridge): penalize the sum of squared weights, sum of w_j^2. Weights shrink toward zero but rarely hit exactly zero. Usually gives the best hold-out performance.
• L1 (Lasso): penalize the sum of absolute weights, sum of |w_j|. Many weights are driven to exactly zero — so L1 also performs feature selection, automatically dropping useless features. Great for explainability.

Combine both and you get elastic net. Regularization is not just for linear models — it shows up in neural networks too (plus tricks like dropout, covered in Chapter 8).

Everyday analogy: L2 is like gently turning down the volume on every instrument in a band so none dominates; L1 is like muting the instruments that are not really contributing to the song.

from sklearn.linear_model import Ridge, Lasso
from sklearn.datasets import make_regression

# A regression problem with 30 features, but only 5 actually matter.
# The rest are noise -- perfect for seeing how L1 zeros out useless features.
X, y = make_regression(n_samples=200, n_features=30, n_informative=5,
                       noise=10, random_state=42)

# Same data, two regularized models. 'alpha' is the strength of the penalty.
ridge = Ridge(alpha=10.0).fit(X, y)
lasso = Lasso(alpha=1.0, max_iter=10000).fit(X, y)

ridge_coef = ridge.coef_
lasso_coef = lasso.coef_

print(f"Ridge (L2): {int(np.sum(ridge_coef != 0))} non-zero coefficients "
      f"out of {len(ridge_coef)}")
print(f"Lasso (L1): {int(np.sum(lasso_coef != 0))} non-zero coefficients "
      f"out of {len(lasso_coef)}  --> it dropped {int(np.sum(lasso_coef == 0))} noise features")

# Plot coefficient magnitudes side by side.
fig, axes = plt.subplots(1, 2, figsize=(11, 4))
axes[0].bar(range(len(ridge_coef)), np.abs(ridge_coef), color="tab:blue")
axes[0].set_title("Ridge (L2): shrunk but rarely zero")
axes[0].set_xlabel("feature index"); axes[0].set_ylabel("|coefficient|")

axes[1].bar(range(len(lasso_coef)), np.abs(lasso_coef), color="tab:orange")
axes[1].set_title("Lasso (L1): many exactly zero")
axes[1].set_xlabel("feature index"); axes[1].set_ylabel("|coefficient|")
plt.tight_layout()
plt.show()

Ridge (L2): 30 non-zero coefficients out of 30
Lasso (L1): 11 non-zero coefficients out of 30  --> it dropped 19 noise features

Reading the coefficient plot

Both models saw the same 30-feature dataset where only 5 features truly mattered. Look at the difference:

• Ridge (L2) shrinks every coefficient but keeps almost all of them non-zero — it hushes the noise features rather than removing them.
• Lasso (L1) zeroes out most of the 25 noise features and keeps only a handful. The bars that remain point to the features that actually carry signal.

That automatic feature selection is why L1 is handy when you want a model you can explain: "these few inputs are what really drive the prediction."

5.6 Model Performance Assessment

You have a model. How good is it, really? You measure it on the test set — data the model has never seen.

For regression, it is straightforward: compare predictions to true values with mean squared error (MSE). A sanity check: your model should beat the "mean model" (the trivial model that always predicts the training average). If test MSE is much higher than training MSE, you are overfitting.

For classification, the toolbox is richer:

• Confusion matrix — a table of predicted vs. actual labels, showing where the model gets confused.
• Accuracy — fraction of examples classified correctly. Simple, but misleading when classes are imbalanced.
• Precision — of everything the model called positive, how much really was? TP / (TP + FP).
• Recall — of everything that really is positive, how much did the model catch? TP / (TP + FN).
• F1-score — the harmonic mean of precision and recall, a single number balancing both.
• ROC / AUC — a curve and its area that summarize the trade-off between true-positive and false-positive rates across thresholds.

You usually have to trade precision for recall (or vice versa). For a spam filter, you want high precision (don't bury a real email) and accept lower recall (some spam gets through). For a cancer screen, you want high recall (don't miss a sick patient) and accept lower precision (some false alarms).

Cross-validation is the trick for when you don't have enough data for a separate validation set: split the training data into k folds (typically 5), train on k−1 folds and validate on the held-out fold, rotate so each fold is validated once, and average the k scores. That gives a much more reliable estimate than a single split.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.metrics import (confusion_matrix, classification_report,
                             accuracy_score)

# A binary classification dataset, slightly imbalanced (70/30).
X, y = make_classification(n_samples=300, n_features=8, n_informative=5,
                           n_redundant=1, weights=[0.7, 0.3], random_state=42)

# Train/test split: train a model, then assess on the test set.
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25,
                                          random_state=42, stratify=y)

clf = RandomForestClassifier(n_estimators=50, max_depth=4,
                             random_state=42, n_jobs=1)
clf.fit(X_tr, y_tr)
y_pred = clf.predict(X_te)

# 5-fold cross-validation on the training set: a robust, less jittery score.
cv_scores = cross_val_score(clf, X_tr, y_tr, cv=5, n_jobs=1)
print("5-fold CV accuracy: %.3f +/- %.3f" % (cv_scores.mean(), cv_scores.std()))
print("Test-set accuracy:  %.3f" % accuracy_score(y_te, y_pred))
print()

# Confusion matrix: rows = actual, columns = predicted.
cm = confusion_matrix(y_te, y_pred)
print("Confusion matrix (rows = actual, cols = predicted):")
print(cm)
print()

# Full report: precision, recall, f1-score per class.
print("Classification report:")
print(classification_report(y_te, y_pred, target_names=["class 0", "class 1"]))

5-fold CV accuracy: 0.849 +/- 0.026
Test-set accuracy:  0.880

Confusion matrix (rows = actual, cols = predicted):
[[50  2]
 [ 7 16]]

Classification report:
              precision    recall  f1-score   support

     class 0       0.88      0.96      0.92        52
     class 1       0.89      0.70      0.78        23

    accuracy                           0.88        75
   macro avg       0.88      0.83      0.85        75
weighted avg       0.88      0.88      0.88        75

Reading the assessment output

• The 5-fold CV accuracy is averaged over five different train/validation splits, so it is far less jittery than a single split. Its standard deviation tells you how sensitive the score is to the exact split.
• The confusion matrix shows the four counts for a binary problem: true positives, false positives, false negatives, true negatives. The off-diagonal numbers are the mistakes.
• The classification report gives precision, recall, and F1 per class. For the minority class, precision and recall usually tell you more than raw accuracy.
• Notice the test accuracy can differ from the CV accuracy — that gap is exactly why we keep a held-out test set.

ROC Curves and AUC

Some classifiers (logistic regression, random forests, neural nets) output a confidence score or probability rather than a hard yes/no. By sliding the decision threshold from 0 to 1, you get a family of (false-positive-rate, true-positive-rate) points. Plot them and you get the ROC curve.

The area under the curve (AUC) boils that curve down to one number:

• AUC = 1.0 — a perfect classifier.
• AUC = 0.5 — no better than a coin flip (the diagonal line).
• AUC < 0.5 — something is wrong (your labels may be flipped).

AUC is handy because it summarizes performance across all thresholds at once, so you can compare models without first committing to a threshold.

from sklearn.metrics import roc_curve, roc_auc_score

# Use the classifier from the previous cell. Many models can output a
# probability per example -- we use that "confidence score" to draw a ROC curve.
y_score = clf.predict_proba(X_te)[:, 1]   # probability of the positive class

fpr, tpr, thresholds = roc_curve(y_te, y_score)
auc = roc_auc_score(y_te, y_score)

plt.figure(figsize=(5, 4))
plt.plot(fpr, tpr, color="tab:blue", lw=2, label=f"ROC curve (AUC = {auc:.3f})")
plt.plot([0, 1], [0, 1], color="gray", ls="--", label="random (AUC = 0.5)")
plt.xlabel("false positive rate")
plt.ylabel("true positive rate (recall)")
plt.title("ROC curve")
plt.legend(loc="lower right")
plt.show()

print(f"AUC = {auc:.3f}  (>0.5 beats random; 1.0 is perfect)")

AUC = 0.901  (>0.5 beats random; 1.0 is perfect)

5.7 Hyperparameter Tuning

A hyperparameter is a setting you choose before training — not something the algorithm learns. Examples: C for an SVM, max_depth for a tree, the learning rate for gradient descent. Picking good values is called tuning, and two simple strategies dominate:

• Grid search: list a discrete set of values for each hyperparameter, then try every combination. Thorough, but the number of combinations explodes quickly (7 values × 2 kernels = 14 models; add a third hyperparameter and it multiplies again).
• Random search: give a range/distribution for each hyperparameter and sample a fixed number of random combinations. Surprisingly often, random search finds a model that is nearly as good in far fewer trials, because it explores each hyperparameter more broadly.

When you don't have enough data for a validation set, grid search + cross-validation is the standard combo: GridSearchCV tries each combination and scores it with k-fold CV, then picks the best. (Bayesian and other smarter methods exist too — they use past trials to choose the next ones to evaluate.)

A practical tip: for a hyperparameter like C that can span many orders of magnitude, search on a logarithmic scale (0.001, 0.01, 0.1, 1, 10, 100, 1000) rather than evenly spaced values.

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Grid search = try EVERY combination in a small table of hyperparameters.
# Here we tune an SVM: the penalty C (log-spaced) and the kernel.
# GridSearchCV scores each combination with 5-fold cross-validation.
param_grid = {
    "C":      [0.1, 1, 10, 100],
    "kernel": ["linear", "rbf"],
}
# 4 x 2 = 8 combinations, each scored with 5-fold CV -> 40 fits.
# n_jobs=1 keeps it single-threaded and quick on this small dataset.
svc = SVC(random_state=42)
grid = GridSearchCV(svc, param_grid, cv=5, n_jobs=1)
grid.fit(X_tr, y_tr)

print("Grid search tried", len(grid.cv_results_['params']), "combinations")
print("Best parameters:    ", grid.best_params_)
print("Best CV accuracy:    %.3f" % grid.best_score_)
print("Test-set accuracy:   %.3f (best model on held-out test)"
      % accuracy_score(y_te, grid.predict(X_te)))
print()

# Random search = sample a fixed number of combos from given value lists.
# Same search space, but we only try 6 random combinations.
rand = RandomizedSearchCV(svc,
                          {"C": [0.1, 1, 10, 100, 1000], "kernel": ["linear", "rbf"]},
                          n_iter=6, cv=5, random_state=42, n_jobs=1)
rand.fit(X_tr, y_tr)
print("Random search (6 samples) best: ", rand.best_params_,
      " CV=%.3f" % rand.best_score_)

Grid search tried 8 combinations
Best parameters:     {'C': 1, 'kernel': 'rbf'}
Best CV accuracy:    0.902
Test-set accuracy:   0.893 (best model on held-out test)

Random search (6 samples) best:  {'kernel': 'rbf', 'C': 10}  CV=0.884

Reading the tuning output

• GridSearchCV tried all 8 combinations (4 values of C × 2 kernels), each scored with 5-fold CV — 40 model fits in all — and reported the best.
• The best CV accuracy is the cross-validated score of the winning combination; the test accuracy is how that winning model does on the untouched test set.
• Random search sampled fewer combinations yet landed on a comparably good setting — that is the appeal: less compute, similar result.
• In a real project you would then zoom in around the best values with a finer grid, and finally lock in the model with one test-set evaluation.

Key Takeaways

• Feature engineering turns raw data into a table of informative features; one-hot encode categories, bin continuous values when it helps, scale numbers, and impute missing values so the model sees clean numeric input.
• Pick an algorithm by asking about explainability, data size, feature types, linearity, and speed — when unsure, let a validation set decide.
• Always split into train / validation / test. Train to fit, validation to choose, test once to assess.
• Underfitting = too simple (high bias); overfitting = too complex (high variance). Find the sweet spot by watching validation error as model complexity grows.
• Regularization (L1/L2) adds a penalty for complexity. L1 (lasso) zeros out useless features; L2 (ridge) gently shrinks all weights.
• Measure classification with a confusion matrix, precision/recall/F1, and ROC/AUC — not accuracy alone, especially for imbalanced data. Use cross-validation when data is scarce.
• Tune hyperparameters with grid or random search, searching sensitive parameters on a log scale, and combine with cross-validation (GridSearchCV).

What's Next

We have stayed in the world of "shallow" models and practical recipes. In Chapter 6 — Neural Networks and Deep Learning, we open the black box that powers modern image, text, and speech systems: artificial neurons, layers, backpropagation, and how to train them.

---

---

Chapter 6 — Neural Networks and Deep Learning

If you understood linear and logistic regression, you already understand the building block of a neural network. In this chapter we take that building block — the humble weighted-sum-plus-activation — stack it into layers, and see how the whole stack learns. Then we go "deep": we look at why adding more layers helps, and at the two special architectures that dominate modern deep learning, convolutional networks for images and recurrent networks for sequences.

In this chapter you will learn:

• What a neuron is (weighted sum + activation) and how neurons stack into layers
• How a feed-forward pass turns an input into a prediction
• The three most common activation functions: ReLU, sigmoid, and tanh
• How backpropagation trains a network using gradient descent
• How to choose layer sizes, and the idea of universal approximation
• What makes a network "deep" and why depth helps
• How convolutional neural networks (CNNs) process images
• How recurrent neural networks (RNNs) handle sequences

6.1 Neural Networks

Here is a nice surprise: you already know what a neural network is. Logistic regression — the model you saw in earlier chapters — is, in fact, a single neuron. Its generalization to multiple classes (softmax regression) is a standard unit inside a neural network.

A neural network, just like a regression or SVM model, is a mathematical function: y = f_NN(x). The special thing about f_NN is that it is a nested function — one function wrapped inside another, layer by layer. For a 3-layer network that outputs a single number, it looks like this:

y = f_NN(x) = f3(f2(f1(x)))

Each layer function has the same shape:

fl(z) = gl(Wl · z + bl)

where:

• Wl is a matrix of weights (the "knobs" the network learns),
• bl is a vector of biases (one per unit),
• gl is a fixed, usually nonlinear activation function chosen by you before training starts.

Compare that to logistic regression: it is exactly sigmoid(w·x + b). A neural network just chains several of these together, with each layer's output becoming the next layer's input.

Everyday analogy: Think of a neural network as an assembly line. Each workstation (layer) takes in some parts, does a small calculation (weighted sum plus a nonlinear squashing), and passes the result to the next workstation. The final workstation outputs the finished product — your prediction.

The Neuron: Weighted Sum + Activation

The smallest unit inside a neural network is a neuron (also called a unit). A neuron does two simple things:

1. Weighted sum: it takes its input vector z, multiplies each entry by a weight, adds them up, and adds a bias. In math: a = w·z + b. This is exactly what linear regression does.
2. Activation: it applies a nonlinear function g to that sum: output = g(a).

Why the nonlinear step? Without it, stacking many layers would be pointless — a linear function of a linear function is still a linear function, so a deep stack of purely-linear layers collapses into one boring linear model. The nonlinearity is what gives a neural network the power to model curved, complicated relationships.

In a multilayer perceptron (MLP) — the most common "vanilla" architecture — every unit in one layer sends its output to every unit in the next layer. That is called a fully-connected (or dense) layer.

Everyday analogy: A neuron is like a tiny voting committee member. Each input is a piece of evidence, the weight is how much the member trusts that evidence, the bias is the member's personal leaning, and the activation is how strongly they react to the total — maybe they stay silent unless the evidence is strong enough (ReLU), or they always give a measured "confidence" between 0 and 1 (sigmoid).

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# --- A tiny neural network by hand, in pure numpy ---
# Architecture: 2 inputs -> 3 hidden neurons (ReLU) -> 1 output (sigmoid)
# The whole network is just: y = sigmoid( W2 @ relu(W1 @ x + b1) + b2 )

def relu(z):
    return np.maximum(0, z)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

# Two input examples, each a 2-dimensional feature vector
X = np.array([[0.5, -1.0],
              [2.0,  0.5]])

# Layer 1 (hidden): W1 has shape (3 neurons, 2 inputs); b1 has 3 entries
W1 = np.array([[ 0.2,  0.8],
               [-0.5,  1.0],
               [ 0.9, -0.3]])
b1 = np.array([0.0, 0.1, -0.2])

# Layer 2 (output): W2 has shape (1 output, 3 hidden); b2 has 1 entry
W2 = np.array([[1.0, -0.5, 0.3]])
b2 = np.array([0.0])

# --- Feed-forward pass, step by step ---

# 1) Hidden layer: linear part (weighted sum), then ReLU activation
z1 = X @ W1.T + b1          # pre-activation: weighted sum for each hidden neuron
h1 = relu(z1)               # activation: squash negatives to zero
print("Hidden layer pre-activations (z1):")
print(z1)
print("Hidden layer activations (h1, after ReLU):")
print(h1)

# 2) Output layer: linear part, then sigmoid to get a probability
z2 = h1 @ W2.T + b2         # weighted sum for the output neuron
y_hat = sigmoid(z2)         # squash to (0, 1) -> a probability
print("\nOutput pre-activations (z2):")
print(z2)
print("Final predictions (y_hat, after sigmoid):")
print(y_hat)
print("\nInterpretation: y_hat > 0.5 -> predict class 1, else class 0.")

Hidden layer pre-activations (z1):
[[-0.7  -1.15  0.55]
 [ 0.8  -0.4   1.45]]
Hidden layer activations (h1, after ReLU):
[[0.   0.   0.55]
 [0.8  0.   1.45]]

Output pre-activations (z2):
[[0.165]
 [1.235]]
Final predictions (y_hat, after sigmoid):
[[0.54115667]
 [0.77469249]]

Interpretation: y_hat > 0.5 -> predict class 1, else class 0.

Reading the output

That little script did a complete feed-forward pass: data flowed from the input, through the hidden layer, to the output, with no learning involved — we just picked the weights by hand. Notice:

• The hidden layer's z1 is just a weighted sum; relu then zeros out the negative entries. Those zeros are the neuron "staying silent."
• The output neuron combines the three hidden signals and sigmoid squashes the result into a number between 0 and 1, which we can read as a probability.

In a real network the weights W1, b1, W2, b2 are not hand-picked — they are learned by gradient descent, which we get to shortly.

Activation Functions

The activation function g is the nonlinear "spark" inside each neuron. Three choices cover most of what you will see:

Function	Formula	Output range	Typical use
Sigmoid	1 / (1 + e^-z)	(0, 1)	output layer for binary classification (a probability)
Tanh	(e^z - e^-z) / (e^z + e^-z)	(-1, 1)	hidden layers (older networks); centered around 0
ReLU	max(0, z)	[0, infinity)	hidden layers (modern default); fast and simple

• Sigmoid squashes anything into a 0-to-1 probability. The downside: its gradient is tiny for very large or very negative inputs, which slows learning in deep networks.
• Tanh is shaped like a softer S, but ranges from -1 to 1. Being centered at zero often helps it learn faster than sigmoid.
• ReLU is the simplest: if the input is negative, output 0; otherwise, pass the input through. It is cheap to compute and, importantly, its gradient does not vanish for large positive inputs — which is a big reason deep networks became practical.

Let's plot all three so you can see their shapes.

# Plot the three most common activation functions side by side
z = np.linspace(-6, 6, 300)

fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))

axes[0].plot(z, sigmoid(z), color="tab:blue")
axes[0].set_title("Sigmoid")
axes[0].set_ylim(-1.2, 1.2); axes[0].axhline(0, color="gray", linewidth=0.8)

axes[1].plot(z, np.tanh(z), color="tab:orange")
axes[1].set_title("Tanh")
axes[1].set_ylim(-1.2, 1.2); axes[1].axhline(0, color="gray", linewidth=0.8)

axes[2].plot(z, relu(z), color="tab:green")
axes[2].set_title("ReLU")
axes[2].set_ylim(-1.2, 2.5); axes[2].axhline(0, color="gray", linewidth=0.8)

for ax in axes:
    ax.set_xlabel("input z")
    ax.set_ylabel("output g(z)")

plt.suptitle("Common activation functions", y=1.02)
plt.tight_layout()
plt.show()

Reading the plot

• Sigmoid flattens out at 0 on the far left and 1 on the far right — that flatness is exactly the vanishing gradient problem: where the curve is flat, the slope (gradient) is near zero, so learning barely happens.
• Tanh is the same S-shape but stretched to go from -1 to 1. Its steepest part is around zero, so it learns well when inputs are small.
• ReLU is a hockey stick: dead flat at zero for negative inputs, then a straight 45-degree ramp for positive inputs. That ramp has a constant gradient of 1, which is why ReLU keeps deep networks trainable.

Layers and the Feed-Forward Pass

In a multilayer perceptron, neurons are organized into layers:

• The input layer is just your feature vector x — it does no computation, it only holds the data.
• The hidden layers do the real work: each is a set of neurons that all receive the previous layer's outputs.
• The output layer produces the final prediction. For regression it uses a linear (identity) activation; for binary classification it uses sigmoid; for multiclass it uses softmax.

Because every neuron in layer l connects to every neuron in layer l+1, these are fully-connected layers. Passing data from input to output is the feed-forward pass — exactly what our numpy code did above, just generalized to any number of layers.

A handy way to picture it: the number of neurons in the output layer matches your task — one for regression or binary classification, k for k-class classification. The hidden layers can be any size you choose.

Backpropagation: How the Network Learns

So far our weights were fixed. In reality they are learned from data. The workhorse algorithm is backpropagation, and it is not a new kind of magic — it is gradient descent (which you already know) applied to the network's weights.

Here is the idea in plain words:

1. Forward pass: push a training example through the network and get a prediction.
2. Measure the error: compare the prediction to the true label y using a cost function (mean squared error for regression, cross-entropy for classification).
3. Backward pass: figure out how much each weight contributed to the error. This is where the chain rule from calculus comes in: because the network is a nesting of functions, you can work out the derivative of the cost with respect to any weight by multiplying the derivatives of the functions it sits inside. Backpropagation is just an efficient way to do that chain-rule bookkeeping from the output layer back to the input.
4. Update: nudge each weight a tiny step in the direction that reduces the error (gradient descent).

The name "backpropagation" comes from step 3: the error signal flows backward through the network, opposite to the forward data flow.

Everyday analogy: You bake a cake (forward pass), taste it, and it is too salty. You work backward: "the saltiness came from the salt I added at the seasoning step, which depended on the recipe I chose at the planning step." You adjust each earlier decision a little so the next cake is better. Repeat many times and the cake improves.

6.1.2 Choosing the Architecture

A practical question: how many layers, and how many neurons per layer?

• The output layer size is fixed by your problem: 1 for regression / binary classification, k for k-class classification.
• The hidden layers are your choice. A common starting point is one or two hidden layers with a moderate number of neurons (say 16 to 128). You tune this like any hyperparameter.
• Bigger is not always better: adding a 1000-neuron layer adds roughly one million parameters, which makes training slower and raises the risk of overfitting.

There is a famous theoretical result called the universal approximation theorem: a feed-forward network with just one hidden layer (and enough neurons, and a suitable nonlinearity) can approximate essentially any continuous function. In practice, though, a deep network with several smaller layers usually needs far fewer total parameters than one giant hidden layer to model complicated patterns — which is one reason depth is so popular.

from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons

# Build a classic non-linearly-separable dataset: two interleaving moons
X, y = make_moons(n_samples=200, noise=0.25, random_state=42)

# Train a small MLP: 2 hidden layers of 16 neurons each, ReLU activation
clf = MLPClassifier(hidden_layer_sizes=(16, 16), activation="relu",
                    solver="adam", max_iter=400, random_state=42)
clf.fit(X, y)

# Plot the data and the learned decision boundary
plt.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue", label="class 0")
plt.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:orange", label="class 1")

# Build a grid and predict the class of every grid point -> decision boundary
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))
grid = np.c_[xx.ravel(), yy.ravel()]
zz = clf.predict(grid).reshape(xx.shape)
plt.contourf(xx, yy, zz, alpha=0.15, cmap="coolwarm")

plt.title("MLP on make_moons - accuracy {:.2f}".format(clf.score(X, y)))
plt.xlabel("feature 1"); plt.ylabel("feature 2")
plt.legend()
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (400) reached and the optimization hasn't converged yet.
  warnings.warn(

Reading the plot

The two-moons dataset is not linearly separable — no straight line can split the classes. Yet the MLP carves out a smooth, curved decision boundary that wraps right around each moon. That curved boundary is the nonlinearity at work: stacked ReLU layers let the model bend the decision surface to fit the data's shape.

This is the key advantage of neural networks over plain linear models: with hidden layers and nonlinear activations, they can model arbitrarily complicated boundaries. Try this with a linear classifier and you would see a straight line cutting awkwardly through the moons.

6.2 Deep Learning

What makes a network "deep"? Deep learning refers to training neural networks with more than one or two hidden layers. Historically, stacking many layers was hard: as you added layers, gradient descent struggled with two problems:

• Vanishing gradient: when you backpropagate through many layers, you multiply many small partial derivatives together (courtesy of the chain rule). Multiply a dozen numbers smaller than 1 and you get something close to 0 — so the earliest layers barely update and never learn. This plagued deep networks for decades.
• Exploding gradient: the opposite — gradients can grow huge, making updates jump wildly and training unstable. This was easier to tame with tricks like gradient clipping and L1/L2 regularization.

Modern deep learning overcame these hurdles: ReLU suffers far less from vanishing gradients (its gradient is a constant 1 for positive inputs), and special architectures like LSTM networks and skip connections (used in residual networks) let gradients flow through hundreds or even thousands of layers. Today "deep learning" is used broadly to mean training neural networks with the modern toolkit, regardless of exact depth — and in practice, many business problems are solved just fine with 2-3 hidden layers.

Why does depth help? Each layer can learn a hierarchy of features: the first layer detects simple edges, the next combines edges into textures or parts, the next combines parts into objects, and so on. Depth lets the model build complicated concepts out of simpler ones, which is both more parameter-efficient and more natural for structured data like images and text.

Everyday analogy: You don't learn to recognize a face all at once. You first learn to see edges, then shapes, then features like eyes and noses, then the whole face. A deep network does the same — each layer builds on the previous one's work.

6.2.1 Convolutional Neural Networks (CNN)

When the input is an image, a fully-connected MLP runs into trouble: even a small 100x100 image has 10,000 pixels, so the first layer alone would need millions of weights. Convolutional neural networks (CNNs) solve this with a simple but powerful idea inspired by how we look at images.

The key insight: in an image, nearby pixels usually belong to the same thing — sky, fur, bricks, an edge. So instead of connecting every pixel to every neuron, a CNN uses a small filter (a p x p matrix of weights) that slides across the image, one patch at a time. At each position, the filter computes a convolution: it takes the dot product of the filter with the patch under it, sums the result, adds a bias, and applies an activation (usually ReLU). The output is a feature map — a grid showing where that pattern appears.

• Each filter acts like a little pattern detector: one might fire on vertical edges, another on horizontal edges, another on corners.
• A convolution layer has many filters, so it produces many feature maps at once.
• Pooling then shrinks each feature map (e.g., by taking the max of every 2x2 block), which cuts the number of parameters and makes the network robust to small shifts.
• Stacking several conv+pool layers builds the hierarchy: edges -> textures -> parts -> objects.

Because the same filter is reused at every position, a CNN has far fewer parameters than an equivalent MLP — and it naturally respects the 2D structure of images. The filter weights are still learned by gradient descent with backpropagation.

Let's see a convolution in action on a tiny synthetic image.

# --- A single convolution, by hand, in numpy ---
# This shows exactly what one Conv2D filter does to one small image.

# A tiny 8x8 grayscale "image": a white square on a black background
img = np.zeros((8, 8))
img[2:6, 2:6] = 1.0

# A 3x3 filter that detects VERTICAL edges
F_vert = np.array([[ 1, 0, -1],
                   [ 1, 0, -1],
                   [ 1, 0, -1]])

# A 3x3 filter that detects HORIZONTAL edges
F_horiz = np.array([[ 1,  1,  1],
                    [ 0,  0,  0],
                    [-1, -1, -1]])

def conv2d(image, filt):
    # Slide `filt` across `image`, computing one convolution per position.
    h, w = image.shape
    fh, fw = filt.shape
    out_h, out_w = h - fh + 1, w - fw + 1
    out = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            patch = image[i:i+fh, j:j+fw]    # the p x p window under the filter
            out[i, j] = np.sum(patch * filt)  # dot product -> one number
    return out

fm_vert  = conv2d(img, F_vert)
fm_horiz = conv2d(img, F_horiz)

fig, axes = plt.subplots(1, 4, figsize=(14, 3.5))
axes[0].imshow(img, cmap="gray_r");        axes[0].set_title("input image")
axes[1].imshow(F_vert, cmap="gray_r");     axes[1].set_title("vertical-edge filter")
axes[2].imshow(fm_vert, cmap="gray_r");    axes[2].set_title("vertical feature map")
axes[3].imshow(fm_horiz, cmap="gray_r");   axes[3].set_title("horizontal feature map")
for ax in axes:
    ax.set_xticks([]); ax.set_yticks([])
plt.suptitle("One convolution layer: a filter slides across an image", y=1.02)
plt.tight_layout()
plt.show()

print("The vertical-edge feature map lights up on the LEFT and RIGHT edges")
print("of the square, and is ~0 in the flat interior. That is the filter")
print("'recognizing' vertical boundaries.")

The vertical-edge feature map lights up on the LEFT and RIGHT edges
of the square, and is ~0 in the flat interior. That is the filter
'recognizing' vertical boundaries.

Reading the feature maps

• The vertical-edge filter produces large positive values on the left edge of the square and large negative values on the right edge — exactly where vertical boundaries are. In the flat interior the filter sees no contrast, so the output is near zero.
• The horizontal-edge filter does the same for the top and bottom edges.

That is the whole magic of a convolution layer: each filter is a reusable pattern detector, and the feature map tells you where in the image that pattern appears. A real CNN stacks dozens of these filters across many layers, with pooling in between, so later filters can detect increasingly complex patterns (edges -> shapes -> objects) while keeping the parameter count manageable.

6.2.2 Recurrent Neural Networks (RNN)

Some data is sequential — the order of the elements matters. A sentence is a sequence of words; a stock price is a sequence of daily values; an audio clip is a sequence of sound samples. A feed-forward network has no notion of order: shuffle the inputs and it gives the same answer. Recurrent neural networks (RNNs) are built for sequences.

The core idea is the hidden state — a kind of running memory. At each time step t, an RNN unit receives two things:

1. the current input x(t), and
2. its own hidden state h(t-1) from the previous time step.

It combines them into a new hidden state:

h(t) = g(W · x(t) + U · h(t-1) + b)

where W weights the new input, U weights the memory, and g is usually tanh. The same weights W, U, b are reused at every time step, which is what lets the RNN handle sequences of any length.

A convenient way to understand an RNN is to unroll it across time: instead of one unit with a loop, imagine one copy of the unit for each time step, with the hidden state flowing from step to step. Unrolling turns the recurrent network into a deep feed-forward network where "depth" equals the sequence length — which is exactly why RNNs also suffer from vanishing gradients on long sequences.

Everyday analogy: Reading a sentence word by word, you carry along a mental summary of what you have read so far (the hidden state). Each new word updates that summary. By the end, your summary captures the meaning of the whole sentence. That is what an RNN does.

# --- A tiny RNN forward pass, by hand, in numpy ---
# One recurrent unit processes a sequence of 5 input vectors,
# carrying a hidden state from step to step.

np.random.seed(0)
seq_len, in_dim, hid_dim = 5, 2, 4   # 5 timesteps, 2-dim input, 4-dim hidden

# The recurrent weights (shared across ALL time steps)
W_x = np.random.randn(hid_dim, in_dim) * 0.5   # input -> hidden
W_h = np.random.randn(hid_dim, hid_dim) * 0.5  # hidden -> hidden (the "memory" weights)
b   = np.zeros(hid_dim)

# A made-up input sequence: 5 vectors of length 2
x_seq = np.random.randn(seq_len, in_dim)

h = np.zeros(hid_dim)   # initial hidden state (start with "no memory")

print("Processing the sequence one time step at a time:\n")
for t in range(seq_len):
    # The recurrent equation: new state = tanh( W_x @ input + W_h @ old_state + b )
    h = np.tanh(W_x @ x_seq[t] + W_h @ h + b)
    print("step {}:  input {}  ->  hidden state {}".format(
        t + 1, np.round(x_seq[t], 2), np.round(h, 2)))

print("\nNotice how each hidden state depends on the CURRENT input AND the")
print("previous hidden state -- that is the 'memory' of the sequence at work.")

Processing the sequence one time step at a time:

step 1:  input [ 2.27 -1.45]  ->  hidden state [ 0.94 -0.48  0.99  0.83]
step 2:  input [ 0.05 -0.19]  ->  hidden state [ 0.49  0.46  0.59 -0.83]
step 3:  input [1.53 1.47]  ->  hidden state [0.82 0.99 0.9  0.61]
step 4:  input [0.15 0.38]  ->  hidden state [ 0.71  0.83  0.34 -0.47]
step 5:  input [-0.89 -1.98]  ->  hidden state [-0.88 -0.98  0.68 -0.53]

Notice how each hidden state depends on the CURRENT input AND the
previous hidden state -- that is the 'memory' of the sequence at work.

Reading the RNN output

Watch the hidden state evolve: at each step it is a blend of the new input and everything that came before (compressed into the previous state). By the final step, h holds a summary of the entire sequence. You could feed that final hidden state into a simple classifier to label the whole sequence (e.g., "is this sentence positive or negative?"), or take the hidden state at every step to label each word.

Gated RNNs: LSTM and GRU

Plain RNNs have a memory problem: on long sequences, early inputs get "forgotten" because the state keeps getting overwritten. The fix is gated units — Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. These add gates (sigmoid-controlled switches between 0 and 1) that learn what to store, what to forget, and what to output. A gate near 0 blocks information; a gate near 1 lets it through. Because storing information acts like the identity function (whose derivative is a constant 1), gradients no longer vanish — so gated RNNs can learn dependencies across many time steps. Modern sequence models (including the attention-based Transformers you may hear about) build on these ideas.

Key Takeaways

• A neuron is a weighted sum plus a nonlinear activation; logistic regression is a single neuron.
• A neural network is a nested function, y = f3(f2(f1(x))), where each layer computes gl(Wl · z + bl).
• ReLU, sigmoid, and tanh are the three workhorse activations; ReLU's constant gradient is a key reason deep networks train well.
• Backpropagation is gradient descent on the network's weights, using the chain rule to send the error signal backward through the layers.
• A network with enough hidden neurons can approximate essentially any function (universal approximation), but depth does so more efficiently by building a hierarchy of features.
• Deep learning = neural networks with several hidden layers, made practical by ReLU and tricks like skip connections that fight vanishing/exploding gradients.
• CNNs use sliding filters to detect local patterns in images, sharing weights across positions to stay parameter-efficient.
• RNNs use a hidden state carried across time steps to handle sequences; gated variants (LSTM, GRU) fix the long-memory problem.

What's Next

In Chapter 7 — Problems and Solutions, we'll see how the models from this and earlier chapters get applied to real problems: classification, regression, and more, along with practical advice on choosing and tuning the right model for the task.

---

---

Chapter 7 — Problems and Solutions

Most of this book so far has focused on the tools — linear models, SVMs, decision trees, neural nets. This chapter flips the view and looks at common problems you'll meet in practice, and the solutions (families of models) that fit each one. Think of it as a field guide: "I have data shaped like this and a goal like that — what do I reach for?"

In this chapter you will learn:

• How to handle more than two classes at once (multiclass classification)
• How to spot anomalies when you only have "normal" examples (one-class classification)
• How to predict several labels per example instead of just one (multi-label classification)
• Why combining many weak models beats one strong model (ensemble learning: bagging, boosting, stacking)
• How to learn from mostly unlabeled data (semi-supervised learning)
• The ideas behind one-shot and zero-shot learning, plus quick notes on a few other problem types

7.1 Kernel Regression

Sometimes a straight line is a bad fit for your data. You could bend it with polynomial features, but picking the right polynomial gets hard once your input has more than a couple of features.

Kernel regression takes a different, non-parametric route: there are no weights to learn. Instead, to predict the value at a new point x, you look at all training points and take a weighted average of their labels. Points close to x get a lot of weight; far-away points get almost none. The weight comes from a kernel function (a bell-shaped Gaussian is the usual choice), and a single knob — the kernel's width b — controls how local the average is. A tiny b chases every wiggly point (overfit); a huge b flattens everything into a near-line (underfit). You tune b on a validation set, just like other hyperparameters.

Everyday analogy: estimating tomorrow's temperature by averaging recent days, but weighting the most recent days much more heavily than days from a month ago.

7.2 Multiclass Classification

In multiclass classification the label is one of C classes (not just two): y ∈ {1, 2, …, C}. For example, sorting handwritten digits into 0–9.

Some algorithms handle many classes natively — decision trees, k-NN, and logistic regression all extend naturally. But many workhorses (like the basic SVM) are fundamentally binary. When you have a binary-only algorithm but a multiclass problem, the standard fix is one-vs-rest (OvR): build C separate binary classifiers, each one asking "is this class c or not?". To predict, run all C classifiers and pick the class whose model is most confident.

When an algorithm can be multiclass natively, the cleanest extension of logistic regression is to swap the sigmoid for the softmax function. Softmax turns a vector of raw scores into a probability distribution: it exponentiates each score and normalizes so the numbers are positive and sum to 1. The predicted class is simply the one with the largest probability.

Everyday analogy: one-vs-rest is like asking C yes/no questions ("is it a cat?", "is it a dog?", …) and going with the loudest "yes". Softmax is like one ranked ballot that spreads 100% of your confidence across all the options at once.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# --- Softmax from scratch, just to see what it does ---
def softmax(scores):
    # Subtract the max for numerical stability (keeps exp() from blowing up)
    s = scores - np.max(scores)
    exp_s = np.exp(s)
    return exp_s / np.sum(exp_s)

# Raw scores a model might produce for 3 classes (cat / dog / bird)
scores = np.array([2.0, 1.0, 0.1])
probs  = softmax(scores)

print("Raw scores      :", scores)
print("Softmax probs   :", np.round(probs, 4))
print("Sum of probs    :", round(probs.sum(), 6))   # always 1.0
print("Predicted class :", np.argmax(probs))

Raw scores      : [2.  1.  0.1]
Softmax probs   : [0.659  0.2424 0.0986]
Sum of probs    : 1.0
Predicted class : 0

Reading the output

The three raw scores {2.0, 1.0, 0.1} became probabilities {0.659, 0.242, 0.099}. Two things always hold:

• every probability is positive (because exp is always positive), and
• they sum to exactly 1 — softmax's whole job is to produce a valid probability distribution.

The biggest score (2.0) became the biggest probability (≈0.66), so we'd predict class 0 ("cat"). Softmax is soft: it doesn't just pick a winner, it tells you how confident it is about every option.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# 3 classes, 2 informative features so we can plot in 2D
X, y = make_classification(n_samples=300, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           n_classes=3, random_state=42)

# Modern LogisticRegression uses softmax (multinomial) by default for >2 classes
clf = LogisticRegression(max_iter=500, random_state=42)
clf.fit(X, y)

# Build a fine grid and ask the model to label every point -> decision regions
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))
grid = np.c_[xx.ravel(), yy.ravel()]
Z = clf.predict(grid).reshape(xx.shape)

plt.contourf(xx, yy, Z, alpha=0.3, cmap="tab10")
plt.scatter(X[:, 0], X[:, 1], c=y, cmap="tab10", edgecolors="k", s=30)
plt.title("Multiclass logistic regression (softmax) — 3 classes")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.show()

# Predicted probabilities for one example should also sum to 1
proba = clf.predict_proba(X[:1])
print("Probabilities for first example:", np.round(proba[0], 4))
print("Sum:", round(proba[0].sum(), 6))

Probabilities for first example: [1.20e-03 9.98e-01 7.00e-04]
Sum: 1.0

Reading the plot

Each colored region is the area the model "owns" for one class; the colored dots are the actual training examples. The boundaries between regions are linear because logistic regression draws straight-line borders.

Notice the predicted probabilities for the first example again sum to 1 — the same softmax behavior we built by hand, now happening inside the model.

One-vs-rest in disguise: if you'd used a plain SVM instead, scikit-learn would have quietly trained three binary SVMs (one per class) under the hood and let them vote — that's the OvR strategy from the intro, handled automatically via decision_function_shape='ovr'.

7.3 One-Class Classification (Anomaly & Novelty Detection)

Sometimes you have piles of "normal" examples and almost no examples of the "bad" ones. Think of network traffic: you have endless normal traffic, but attacks are rare and constantly changing. You can't train a normal classifier to separate "attack" from "normal" if you barely have any attack examples.

One-class classification flips the problem: instead of learning the boundary between two classes, you learn a tight description of what "normal" looks like, and anything that falls outside it gets flagged as an outlier. Common approaches fit a probability density to the normal data (a one-class Gaussian), or draw a boundary around it (one-class SVM), or isolate points by randomly splitting the feature space (Isolation Forest). You then set a threshold: inside the description = normal, outside = anomaly.

Everyday analogy: you've never seen a counterfeit $100 bill, but you've handled thousands of real ones — so a bill that feels off (wrong paper, odd color) stands out because it doesn't match your mental model of "normal bill".

from sklearn.ensemble import IsolationForest

# A tight cluster of "normal" points, plus a few scattered outliers
rng = np.random.RandomState(42)
X_normal   = rng.randn(200, 2) * 0.8                       # normal data near origin
X_outliers = rng.uniform(low=-6, high=6, size=(15, 2))     # 15 random outliers
X_all = np.vstack([X_normal, X_outliers])

# IsolationForest: "isolate" points by random splits; outliers are easiest to isolate
iso = IsolationForest(contamination=15 / 215, random_state=42, n_jobs=1)
pred = iso.fit_predict(X_all)   # +1 = normal, -1 = outlier

normal   = X_all[pred == 1]
outliers = X_all[pred == -1]

plt.scatter(normal[:, 0], normal[:, 1], c="tab:blue", s=20, label="normal")
plt.scatter(outliers[:, 0], outliers[:, 1], c="tab:red", s=60,
            marker="x", label="flagged outliers")
plt.title("One-class detection with Isolation Forest")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

print(f"Flagged {len(outliers)} points as outliers (we planted 15).")

Flagged 15 points as outliers (we planted 15).

Reading the plot

The blue cluster is the model's idea of "normal". The red x's are the points the forest decided were too easy to isolate — they sit far from the dense region, exactly where we secretly placed the outliers. contamination is the key hyperparameter: it's your guess at what fraction of the data is anomalous, and you tune it on a validation set (or set it from domain knowledge).

7.4 Multi-Label Classification

Don't confuse this with multiclass. In multiclass each example has one label out of many. In multi-label each example can have several labels at once. A photo might be tagged people, concert, and nature simultaneously.

The common trick is to turn one multi-label problem into L independent binary problems — one per label ("does this image contain people? yes/no", "…concert? yes/no", …). Any classifier that gives a score per class (trees, logistic regression, neural nets, k-NN) can be applied this way: predict every label, then keep the ones whose score clears a threshold you choose on a validation set. Neural nets do this naturally with one sigmoid output per label and a binary cross-entropy loss.

Everyday analogy: multiclass is a single-choice quiz (pick one answer). Multi-label is "tick all that apply".

scikit-learn stores multi-label targets as an indicator matrix — a 0/1 grid with one column per label. The MultiLabelBinarizer converts human-readable label lists into this matrix and back.

from sklearn.datasets import make_multilabel_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MultiLabelBinarizer

# First, see how MultiLabelBinarizer turns label LISTS into a 0/1 grid.
# Each row is one example; each column is one possible label; 1 = label present.
raw_labels = [["people", "concert"], ["nature"], ["people", "nature", "concert"]]
mlb = MultiLabelBinarizer()
Y_demo = mlb.fit_transform(raw_labels)
print("Indicator matrix:\n", Y_demo)
print("Label columns:", list(mlb.classes_))
print("Back to lists:", mlb.inverse_transform(Y_demo))
print()

# Now build a real (tiny) multi-label dataset directly as an indicator matrix.
X, Y = make_multilabel_classification(n_samples=120, n_features=6, n_classes=3,
                                      n_labels=2, random_state=42)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.25,
                                                    random_state=42)

# A RandomForest can train on the 0/1 matrix directly: it effectively learns
# one binary model per label column.
clf = RandomForestClassifier(n_estimators=50, random_state=42, n_jobs=1)
clf.fit(X_train, Y_train)

print("True labels (first 3 test rows):\n", Y_test[:3])
print("Predicted labels (first 3 test rows):\n", clf.predict(X_test[:3]))
print("Subset accuracy on test set:", round(clf.score(X_test, Y_test), 3))

Indicator matrix:
 [[1 0 1]
 [0 1 0]
 [1 1 1]]
Label columns: ['concert', 'nature', 'people']
Back to lists: [('concert', 'people'), ('nature',), ('concert', 'nature', 'people')]

True labels (first 3 test rows):
 [[0 1 0]
 [0 1 1]
 [1 0 0]]
Predicted labels (first 3 test rows):
 [[0 1 0]
 [0 1 1]
 [1 0 0]]
Subset accuracy on test set: 0.667

Reading the output

The indicator matrix is just a tidy 0/1 table: row 0 [1 1 0] means "people yes, concert yes, nature no". inverse_transform flips it back to readable label lists.

The RandomForest trains on this matrix directly and predicts a full 0/1 row per example — so one photo can come out with several 1's. The score shown is subset accuracy: it only counts a prediction correct when every label is right, which is harsh, so don't be alarmed if it looks modest. In practice people report per-label accuracy or F1 instead.

7.5 Ensemble Learning

So far we've tried to build one accurate model. Ensemble learning takes the opposite bet: train a crowd of cheap, individually-weak models and combine their votes into one strong "meta-model". The crowd is right more often than any single member — if the members disagree with each other in different ways.

Why does this work? Every model makes errors, but if the errors are uncorrelated, they cancel out when you average or vote. Two main strategies dominate:

• Bagging builds many models in parallel on random resamples of the data, then averages. It tames variance (overfitting). Random Forest is the star example.
• Boosting builds models sequentially: each new model focuses on fixing the previous one's mistakes. It tames bias (underfitting), but can overfit if pushed too far. Gradient Boosting is the star example.

A third, stacking, trains several different models and then a meta-learner that learns whose advice to trust.

Everyday analogy: bagging asks 100 people the same trivia question and takes the majority. Boosting has each person study only the questions the previous people got wrong. Stacking is a manager who learns which employee to trust on which topic.

7.5.1 Bagging & Random Forest

Bagging ("bootstrap aggregating") goes like this: draw B random samples with replacement from your training set (each sample the same size as the original, so some rows repeat and some are missing), train a shallow decision tree on each sample, then average their predictions (or majority-vote for classification).

Random Forest adds one twist: at every split, each tree only looks at a random subset of the features. Why? If one feature is a super-predictor, every tree would latch onto it and become nearly identical — and identical trees can't cancel each other's errors. Forcing each tree to see different features keeps the forest diverse, and diversity is what makes ensembling pay off.

The big dials to tune are the number of trees B and the feature-subset size.

7.5.2 Boosting & Gradient Boosting

Boosting builds its crowd in sequence. For regression you start with a constant prediction (the average label), then repeatedly:

1. Compute each example's residual = true label − current prediction (how far off the model is).
2. Train a small tree to predict those residuals — i.e., to fix the current model's errors.
3. Add the new tree to the ensemble, scaled by a small learning rate.

Repeat for M trees. Each tree patches the mistakes of the ones before it. It's called "gradient" boosting because the residuals act as a stand-in for the gradient of the error — they point in the direction that reduces the loss, just like gradient descent did for linear regression in Chapter 4.

Key dials: number of trees M, learning rate, and tree depth. Boosting reduces bias and often beats random forest on accuracy, but its sequential nature makes it slower to train, and with too many deep trees it can overfit (unlike bagging, which rarely does).

from sklearn.datasets import make_moons
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Two interleaving half-circles -- a classic non-linear problem
X, y = make_moons(n_samples=200, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=42)

# Bagging family: Random Forest (many trees in parallel, averaged)
rf = RandomForestClassifier(n_estimators=50, random_state=42, n_jobs=1)
rf.fit(X_train, y_train)

# Boosting family: Gradient Boosting (trees in sequence, each fixes the last)
gb = GradientBoostingClassifier(n_estimators=50, max_depth=3,
                                learning_rate=0.1, random_state=42)
gb.fit(X_train, y_train)

# Shared grid for decision-boundary plots
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 250),
                     np.linspace(y_min, y_max, 250))
grid = np.c_[xx.ravel(), yy.ravel()]

fig, axes = plt.subplots(1, 2, figsize=(11, 4), sharey=True)
for ax, model, name in [(axes[0], rf, "Random Forest (bagging)"),
                        (axes[1], gb, "Gradient Boosting (boosting)")]:
    Z = model.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.3, cmap="coolwarm")
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap="coolwarm",
               edgecolors="k", s=30)
    acc = accuracy_score(y_test, model.predict(X_test))
    ax.set_title(f"{name}\ntest accuracy = {acc:.2f}")
    ax.set_xlabel("feature 1")
axes[0].set_ylabel("feature 2")
plt.tight_layout()
plt.show()

print(f"Random Forest test accuracy     : {accuracy_score(y_test, rf.predict(X_test)):.3f}")
print(f"Gradient Boosting test accuracy : {accuracy_score(y_test, gb.predict(X_test)):.3f}")

Random Forest test accuracy     : 0.900
Gradient Boosting test accuracy : 0.833

Reading the hero plot

The two half-moons can't be separated by a straight line, but both ensembles carve out a curved, flexible boundary that hugs the moons — something a single shallow tree couldn't do. On this small dataset the two accuracies are usually close; the story is in how they got there:

• Random Forest averaged 50 independently-grown trees, smoothing away each tree's overfitting (lower variance).
• Gradient Boosting grew 50 trees in a chain, each one cleaning up the previous one's errors (lower bias).

Both beat what a lone tree would do. That's the whole appeal of ensembling: many weak, diverse models combine into one strong one.

7.5.3 Stacking

Bagging and boosting combine copies of the same weak learner. Stacking combines different kinds of models — say a tree, a k-NN, and a logistic regression — and then trains a meta-learner on their predictions. The meta-learner learns which base model to trust for a given input, rather than just averaging everyone equally.

The honest way to train it: split your data, train the base models on one part, have them predict on the other part, and use those predictions (plus the true labels) to train the meta-learner. Otherwise the meta-learner just trusts whichever base model memorized the training data. scikit-learn's StackingClassifier handles this cross-fitting for you.

from sklearn.ensemble import StackingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Same moons data as the hero demo (regenerated so this cell stands alone)
X, y = make_moons(n_samples=200, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=42)

# Two DIFFERENT base learners + a logistic-regression meta-learner on top
base = [("tree", DecisionTreeClassifier(max_depth=3, random_state=42)),
        ("knn",  KNeighborsClassifier(n_neighbors=5))]
stack = StackingClassifier(estimators=base,
                           final_estimator=LogisticRegression(max_iter=300,
                                                               random_state=42),
                           cv=3, n_jobs=1)
stack.fit(X_train, y_train)

# Compare: a lone shallow tree vs k-NN vs the stacked ensemble
tree_only = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)
knn_only  = KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train)

print(f"Lone shallow tree  : {accuracy_score(y_test, tree_only.predict(X_test)):.3f}")
print(f"k-NN (k=5)         : {accuracy_score(y_test, knn_only.predict(X_test)):.3f}")
print(f"Stacked ensemble   : {accuracy_score(y_test, stack.predict(X_test)):.3f}")

Lone shallow tree  : 0.850
k-NN (k=5)         : 0.917
Stacked ensemble   : 0.883

Reading the output

The lone shallow tree does worst (one small tree can't capture both moons). The stacked ensemble usually matches or beats its best base model, because the meta-learner can lean on the tree where it's strong and on k-NN where the tree is weak. Stacking pays off most when your base models make different kinds of mistakes.

Other Problem Types (Brief Notes)

A few more supervised flavors you should know exist — no demos here, just the idea.

7.6 Learning to Label Sequences

A sequence is structured data where order matters: words in a sentence, daily stock prices, notes in a melody. In sequence labeling each position in the input gets its own label — e.g., tagging each word with its part of speech ("big"→adjective, "car"→noun). Recurrent neural nets (RNNs) handle this naturally by reading one step at a time. A classic non-neural alternative is the Conditional Random Field (CRF), which you can think of as logistic regression generalized to whole sequences — it scores how well a sequence of labels fits a sequence of inputs.

7.7 Sequence-to-Sequence Learning

Seq2seq generalizes sequence labeling: now the input sequence and output sequence can have different lengths. The poster child is machine translation (English sentence in, French sentence out). The standard architecture has two parts — an encoder that reads the input into a compact vector called an embedding, and a decoder that generates the output sequence from that embedding, feeding each output back in as the next input. Adding an attention mechanism lets the decoder focus on the most relevant input positions, which dramatically helps with longer sequences.

7.8 Active Learning

Labels can be expensive — a doctor must read each scan, an analyst each transaction. Active learning starts with a few labeled examples and a pile of unlabeled ones, then chooses which unlabeled example to ask the expert about next — picking the one the current model is most uncertain about (or that sits in a dense, confusing region). Each new label is added, the model is retrained, and the cycle repeats until the budget runs out or accuracy plateaus. It's supervised learning that shops for labels wisely.

7.9 Semi-Supervised Learning

What if you have a handful of labeled examples and a mountain of unlabeled ones? Throwing the unlabeled data away seems wasteful, but you can't train a supervised model on blanks. Semi-supervised learning (SSL) tries to squeeze useful signal out of the unlabeled mass too.

One simple idea is self-learning: train on the labeled few, predict the unlabeled many, and "promote" the high-confidence predictions into the training set, then repeat. It can help, but it can also drift off course if the early model is confidently wrong. A gentler, graph-based idea is label propagation: treat every example (labeled or not) as a node, connect nearby examples, and let the known labels bleed across the graph to their neighbors. Unlabeled points that sit inside a labeled region inherit that region's label. sklearn gives us LabelPropagation for exactly this.

from sklearn.semi_supervised import LabelPropagation
from sklearn.datasets import make_blobs

# 3 blobs; we'll reveal only a handful of labels and hide the rest
X, y_true = make_blobs(n_samples=150, centers=3, cluster_std=1.2,
                       random_state=42)

# Copy the labels, then hide ~90% of them (mark as -1 = "unlabeled")
rng = np.random.RandomState(1)
y_hidden = y_true.copy()
unlabeled_idx = rng.choice(len(y_true), size=135, replace=False)
y_hidden[unlabeled_idx] = -1

# LabelPropagation spreads the known labels across the graph of nearby points
lp = LabelPropagation(kernel="knn", n_neighbors=7)
lp.fit(X, y_hidden)            # entries equal to -1 are treated as unlabeled
y_learned = lp.transduction_   # the labels the model assigned to every point

fig, axes = plt.subplots(1, 2, figsize=(11, 4), sharey=True)
# Left: what we actually gave the model (only 15 labeled dots)
axes[0].scatter(X[:, 0], X[:, 1], c="lightgray", s=20)
labeled = y_hidden != -1
axes[0].scatter(X[labeled, 0], X[labeled, 1], c=y_hidden[labeled],
                cmap="viridis", edgecolors="k", s=80)
axes[0].set_title("Given to model: 15 labels, 135 unknown (gray)")
axes[0].set_xlabel("feature 1")
axes[0].set_ylabel("feature 2")

# Right: labels propagated to everyone
axes[1].scatter(X[:, 0], X[:, 1], c=y_learned, cmap="viridis",
                edgecolors="k", s=30)
axes[1].set_title("After label propagation: everyone gets a label")
axes[1].set_xlabel("feature 1")
plt.tight_layout()
plt.show()

acc = np.mean(y_learned == y_true)
print(f"Recovered label accuracy vs. truth: {acc:.3f} (from only 15 seeds)")

Recovered label accuracy vs. truth: 1.000 (from only 15 seeds)

Reading the plot

On the left, the model saw mostly gray (unknown) dots plus just 15 colored seeds. On the right, label propagation has spread those seeds across the graph: each unlabeled point adopted the label most common among its 7 nearest neighbors, so the three blobs get painted in even though we never labeled 90% of them. The recovered accuracy tells you how often those propagated labels match the hidden ground truth — usually quite good when the clusters are clean and well-separated like here.

7.10 One-Shot Learning

The name is slightly misleading: it doesn't mean you train on one example. It means the deployed model only needs one reference example of a new identity to recognize it — the classic case is face unlock: you enroll with a single selfie, and your phone can later tell "you" from "not you".

The usual approach is a siamese network: the same neural net processes two images and produces an embedding (a compact vector) for each. The network is trained with a triplet loss using triplets (anchor, positive, negative) — two photos of the same person and one of a different person — pushing same-person embeddings close together and different-person embeddings far apart. At inference time you compare a new photo's embedding to the stored one using a distance threshold.

Everyday analogy: you don't need to see a hundred photos of a new coworker to recognize them tomorrow — one good look is enough, because you've already learned generally what "faces" and "differences between faces" look like.

7.11 Zero-Shot Learning

Zero-shot learning (ZSL) asks the impossible-sounding question: can a model predict a label it has never seen in training? The trick is to represent not just the inputs but also the labels as vectors (think word embeddings, where words with similar meanings sit close together in space).

Instead of predicting a class index, the model predicts an embedding vector. To name the prediction, you find the word in your dictionary whose embedding is closest (by cosine similarity) to the predicted one. Because the model learns features that transfer — "stripiness", "mammalness", "orangeness" — it can land on a label like tiger even if no tiger appeared in training, as long as related concepts (zebra, clownfish) did.

Everyday analogy: you've never tasted a "pluot", but you know plums and apricots, so when you taste one you can guess its name from the two familiar flavors it combines.

Key Takeaways

• Multiclass = one label out of many; solve it natively with softmax (logistic regression) or with one-vs-rest when your algorithm is binary-only.
• Softmax turns raw scores into a probability distribution that is positive and sums to 1.
• One-class classification learns what "normal" looks like and flags the rest as outliers — handy when anomalous examples are rare (Isolation Forest, one-class SVM).
• Multi-label = several labels per example; usually modeled as one binary problem per label, stored as a 0/1 indicator matrix.
• Ensembles beat single models by combining many diverse weak learners: bagging (Random Forest) lowers variance; boosting (Gradient Boosting) lowers bias; stacking lets a meta-learner choose which base model to trust.
• Semi-supervised learning extracts signal from unlabeled data — label propagation spreads known labels across a graph of nearby points.
• One-shot and zero-shot learning extend supervised ideas to settings with almost no labeled data, using embeddings and similarity.

What's Next

In Chapter 8 — Advanced Practice, we'll move beyond individual problem types and look at the craft of putting machine learning to work for real: feature engineering, model evaluation, hyperparameter tuning, and avoiding the practical traps that sink naive projects.

---

---

Chapter 8 — Advanced Practice

Some techniques aren't harder than the basics — they're just extra tools you reach for in specific situations. This chapter collects the most useful "advanced practice" tricks: dealing with lopsided datasets, blending several models together, training neural networks without losing your mind, regularizing in cleverer ways, handling several inputs or outputs at once, borrowing knowledge from one model to solve a different problem, and thinking about how fast your code runs.

In this chapter you will learn:

• Why a 99% accuracy score can be terrible news, and how to fix imbalanced datasets with class weights, resampling, and threshold tuning.
• How to combine several models (averaging, voting, stacking) so they're smarter together than alone.
• Practical tips for training neural networks: learning-rate schedules, batch size, and early stopping.
• Advanced regularization ideas: dropout, batch normalization, and data augmentation.
• How to feed a model several kinds of input at once, or ask it for several outputs.
• What transfer learning is and why it's a superpower of neural networks.
• How to reason about algorithm speed with big-O notation.

8.1 Handling Imbalanced Datasets

Imagine you're building a fraud detector for an online store. Out of every 1,000 transactions, maybe 5 are fraud and 995 are honest. If your model simply shouted "honest!" every single time, it would be right 99.5% of the time. That sounds great — until you realize it catches exactly zero fraud.

This is the imbalanced dataset trap: when one class vastly outnumbers the others, a model can rack up a high accuracy (fraction of correct predictions) while being useless for the class you actually care about. The majority class drowns out the minority.

The fix in one line: make the minority class "count more" during training, or change how you measure success.

A few standard techniques:

• Class weights — tell the learning algorithm "a mistake on a minority example is N times more expensive than a mistake on a majority example." Most sklearn classifiers accept class_weight='balanced', which sets weights inversely proportional to class frequency.
• Resampling — oversample the minority class (duplicate its examples, or invent synthetic ones) so both classes are equally common, or undersample the majority class (randomly drop some of its examples).
• Threshold tuning — many classifiers output a probability; the default decision threshold is 0.5. With imbalanced data, a different threshold often gives far better recall on the minority class.

Everyday analogy: it's like a teacher grading a class where 30 students are fluent in the topic and 1 student is struggling. If the teacher only looks at "class average score," everything looks fine and the struggling student vanishes from the statistics. You need a metric (and a strategy) that sees the minority.

Let's see the problem and the fixes in code.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, balanced_accuracy_score,
                             f1_score, confusion_matrix, ConfusionMatrixDisplay)

# Consistent figure size for the whole notebook
plt.rcParams['figure.figsize'] = (6, 4)

# --- Build a lopsided dataset: 5% minority (class 1), 95% majority (class 0) ---
X, y = make_classification(n_samples=1000, n_features=8, n_informative=5,
                           n_redundant=2, n_classes=2, weights=[0.95, 0.05],
                           flip_y=0.02, random_state=7)

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=7,
                                      stratify=y)

print("Training class counts:", np.bincount(ytr))

# Train a plain logistic regression (no special handling of imbalance)
clf = LogisticRegression(max_iter=1000, random_state=7)
clf.fit(Xtr, ytr)
pred = clf.predict(Xte)

print("\n--- Plain model (ignores imbalance) ---")
print("Accuracy            : %.3f" % accuracy_score(yte, pred))
print("Balanced accuracy   : %.3f" % balanced_accuracy_score(yte, pred))
print("F1 (minority class) : %.3f" % f1_score(yte, pred))
print("Confusion matrix (rows=true, cols=pred):\n", confusion_matrix(yte, pred))

Training class counts: [659  41]

--- Plain model (ignores imbalance) ---
Accuracy            : 0.963
Balanced accuracy   : 0.676
F1 (minority class) : 0.522
Confusion matrix (rows=true, cols=pred):
 [[283   0]
 [ 11   6]]

Reading those numbers

The accuracy looks impressive (well above 90%), but the balanced accuracy and F1 tell a different, sadder story. The confusion matrix usually shows a big block of correct majority predictions and a worrying number of missed minority examples (false negatives).

• Balanced accuracy averages the recall of each class, so a model that ignores the minority class can't hide.
• F1 combines precision and recall for the minority class — exactly what you care about in fraud detection.

When the classes are lopsided, never trust accuracy alone.

# Now flip on class weighting: sklearn will up-weight the rare class
clf_bal = LogisticRegression(max_iter=1000, class_weight='balanced', random_state=7)
clf_bal.fit(Xtr, ytr)
pred_bal = clf_bal.predict(Xte)

print("--- Weighted model (class_weight='balanced') ---")
print("Accuracy            : %.3f" % accuracy_score(yte, pred_bal))
print("Balanced accuracy   : %.3f" % balanced_accuracy_score(yte, pred_bal))
print("F1 (minority class) : %.3f" % f1_score(yte, pred_bal))

# Side-by-side confusion matrices: before vs after weighting
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
ConfusionMatrixDisplay.from_predictions(yte, pred, ax=axes[0], colorbar=False)
axes[0].set_title("Plain model\n(ignores imbalance)")
ConfusionMatrixDisplay.from_predictions(yte, pred_bal, ax=axes[1], colorbar=False)
axes[1].set_title("Weighted model\n(class_weight='balanced')")
plt.tight_layout(); plt.show()

--- Weighted model (class_weight='balanced') ---
Accuracy            : 0.797
Balanced accuracy   : 0.754
F1 (minority class) : 0.282

What changed

Weighting the classes trades a little accuracy on the majority class for a lot more recall on the minority class. Look at the confusion matrix: the weighted model catches more fraud (fewer false negatives), at the cost of a few more false alarms (false positives). Whether that trade is worth it depends on your problem — in fraud or disease screening, missing a real case is usually far more expensive than a false alarm.

Now let's try resampling and threshold tuning, two more knobs you can turn.

# --- 1) Resampling by hand (no extra libraries needed) ---
# Oversample the minority class: duplicate its training rows until ~balanced
mask_min = (ytr == 1)
Xtr_over = np.vstack([Xtr, Xtr[mask_min].repeat(18, axis=0)])  # ~match majority
ytr_over = np.concatenate([ytr, ytr[mask_min].repeat(18)])
print("After oversampling, train class counts:", np.bincount(ytr_over))

clf_over = LogisticRegression(max_iter=1000, random_state=7)
clf_over.fit(Xtr_over, ytr_over)
print("Oversample F1 : %.3f" % f1_score(yte, clf_over.predict(Xte)))

# Undersample the majority class: randomly drop majority rows until balanced
rng = np.random.RandomState(7)
maj_idx = np.where(ytr == 0)[0]
keep = rng.choice(maj_idx, size=int(mask_min.sum()), replace=False)
idx_under = np.concatenate([keep, np.where(ytr == 1)[0]])
clf_under = LogisticRegression(max_iter=1000, random_state=7)
clf_under.fit(Xtr[idx_under], ytr[idx_under])
print("Undersample F1: %.3f" % f1_score(yte, clf_under.predict(Xte)))

# --- 2) Threshold tuning on the PROBABILITY output ---
proba = clf.predict_proba(Xte)[:, 1]   # P(class=1)
thresholds = np.linspace(0.05, 0.95, 19)
f1s = [f1_score(yte, proba >= t) for t in thresholds]

best_t = thresholds[int(np.argmax(f1s))]
plt.plot(thresholds, f1s, 'o-')
plt.axvline(best_t, color='r', ls='--', label="best threshold=%.2f" % best_t)
plt.xlabel("Decision threshold"); plt.ylabel("F1 (minority class)")
plt.title("Threshold tuning: F1 vs decision threshold")
plt.legend(); plt.tight_layout(); plt.show()
print("Default 0.5 F1: %.3f  |  best F1: %.3f" % (f1_score(yte, proba>=0.5), max(f1s)))

After oversampling, train class counts: [659 779]
Oversample F1 : 0.252
Undersample F1: 0.276

Default 0.5 F1: 0.522  |  best F1: 0.643

8.2 Combining Models

A Random Forest already combines many trees of the same kind. But you can also mix different kinds of models — a logistic regression, a support vector machine, and a random forest, say — and the mixture can beat any single one. The magic word is uncorrelated: if the models make different mistakes, those mistakes cancel out when they vote.

Three classic ways to combine:

• Averaging — each model outputs a score; you take the mean. Great for regression and for classifiers that return probabilities.
• Majority vote — each model outputs a class label; you pick the class that the most models agree on. (Ties are broken randomly or reported as "unsure.")
• Stacking — train a meta-model whose input features are the predictions of the base models. The meta-model learns which base model to trust in which situation.

Everyday analogy: asking three friends with different backgrounds for restaurant recommendations and going with the place two of them agree on. If they all have the same taste, you gain nothing; if their tastes differ, the consensus is usually safer than any single pick.

from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.svm import SVC
from sklearn.datasets import make_moons

# A small, slightly tricky 2D dataset
X, y = make_moons(n_samples=300, noise=0.25, random_state=4)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=4)

# Three different "base" models
log = LogisticRegression(max_iter=1000, random_state=4)
rf  = RandomForestClassifier(n_estimators=50, max_depth=5, random_state=4, n_jobs=1)
svm = SVC(probability=True, random_state=4)   # probability=True so soft voting works

# Hard voting = majority of predicted labels
# Soft voting = average of predicted probabilities
vote_hard = VotingClassifier([('log', log), ('rf', rf), ('svm', svm)], voting='hard')
vote_soft = VotingClassifier([('log', log), ('rf', rf), ('svm', svm)], voting='soft')

for name, m in [('LogReg', log), ('RandForest', rf), ('SVM', svm),
                ('Vote(hard)', vote_hard), ('Vote(soft)', vote_soft)]:
    m.fit(Xtr, ytr)
    print("%-12s test accuracy: %.3f" % (name, m.score(Xte, yte)))

LogReg       test accuracy: 0.878
RandForest   test accuracy: 0.956
SVM          test accuracy: 0.956
Vote(hard)   test accuracy: 0.944
Vote(soft)   test accuracy: 0.933

Reading the result

Often the ensemble matches or beats the best individual model, and — just as importantly — it's less jittery: it rarely collapses on a bad split the way a single model can. Soft voting (averaging probabilities) usually edges out hard voting because it uses more information than a flat label.

from sklearn.ensemble import StackingClassifier

# Stacking: base models feed a "meta" model that learns who to trust
stack = StackingClassifier(
    estimators=[('log', log), ('rf', rf), ('svm', svm)],
    final_estimator=LogisticRegression(max_iter=1000),  # the meta-model
    cv=5,           # use cross-validation to build the meta-features (avoids leakage)
    n_jobs=1,
)
stack.fit(Xtr, ytr)
print("Stacking     test accuracy: %.3f" % stack.score(Xte, yte))

Stacking     test accuracy: 0.933

Why stacking can help

The meta-model doesn't just average — it learns weights. If the SVM tends to be right where the logistic model is wrong, the meta-model can learn to lean on the SVM in that region. The cv=5 setting builds the meta-features with cross-validation so the meta-model never sees predictions made on data a base model was already trained on (that would be leakage — cheating).

Just remember the warning: stacking two near-identical models (say, two SVMs with similar settings) gains little. Combine models of different natures.

8.3 Training Neural Networks

Training a neural network has a few extra dials compared to a simple model. The most important practical ones:

• Learning rate — the step size for gradient descent. Too big and training bounces around; too small and it crawls. A learning-rate schedule starts large (to move fast) and decays over time (to settle in). sklearn's MLPClassifier supports learning_rate='invscaling', which shrinks the step each epoch.
• Batch size — how many examples you process before updating the weights. Small batches = noisier but cheaper updates; large batches = smoother but slower per epoch.
• Early stopping — instead of training for a fixed number of epochs, you watch a validation set after each epoch and stop the moment validation performance stops improving (or starts getting worse). This prevents overfitting and saves time.

Everyday analogy: tuning a guitar. You turn the peg boldly at first to get close, then make tiny adjustments to fine-tune (that's a learning-rate schedule). You stop when it sounds right, not after some fixed number of turns (that's early stopping) — otherwise you'll overshoot and snap the string.

from sklearn.neural_network import MLPClassifier

X, y = make_classification(n_samples=800, n_features=10, n_informative=6,
                           n_redundant=2, flip_y=0.1, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=3)

# Model A: train a fixed 300 epochs, no early stopping
mlp_fixed = MLPClassifier(hidden_layer_sizes=(16, 8), max_iter=300,
                          random_state=3, n_iter_no_change=10000)  # effectively no early stop
mlp_fixed.fit(Xtr, ytr)

# Model B: early stopping -- halt when validation score stops improving
mlp_early = MLPClassifier(hidden_layer_sizes=(16, 8), max_iter=300,
                          early_stopping=True, validation_fraction=0.2,
                          n_iter_no_change=10, random_state=3)
mlp_early.fit(Xtr, ytr)

print("Fixed  (no early stop): %d epochs, train acc %.3f, test acc %.3f" %
      (mlp_fixed.n_iter_, mlp_fixed.score(Xtr,ytr), mlp_fixed.score(Xte,yte)))
print("Early stop           : %d epochs, train acc %.3f, test acc %.3f" %
      (mlp_early.n_iter_, mlp_early.score(Xtr,ytr), mlp_early.score(Xte,yte)))

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (300) reached and the optimization hasn't converged yet.
  warnings.warn(

Fixed  (no early stop): 300 epochs, train acc 0.923, test acc 0.858
Early stop           : 48 epochs, train acc 0.761, test acc 0.787

Reading the result

The early-stopping model usually trains in fewer epochs and often holds a slightly better test accuracy, because it quits before it starts memorizing the training set's noise. The fixed model, left to run, can overfit: its training accuracy keeps climbing while its test accuracy flattens or drops.

Early stopping is one of the cheapest, most reliable forms of regularization — try it first.

8.4 Advanced Regularization

Beyond L1 and L2, neural networks have their own regularizers:

• Dropout — during each training step, randomly turn off a fraction of the neurons (set their output to zero). The network can't rely on any single neuron, so it spreads its knowledge around — like a sports team that trains with random players benched, so no one becomes a single point of failure. The dropout rate (0 to 1) is tuned on validation data.
• Batch normalization ("batch standardization") — standardize each layer's outputs (zero mean, unit variance) before passing them to the next layer. It's not officially a regularizer, but it makes training faster, more stable, and often slightly more generalizing. In libraries you insert a BatchNorm layer between two layers.
• Data augmentation — instead of only training on your original examples, create synthetic ones by transforming them (rotate, flip, zoom, darken an image; add jitter to numbers). The label stays the same. More diverse training data means less overfitting. This works for any model, not just neural networks.

Everyday analogy for dropout: studying for an exam by occasionally hiding random pages of your notes — it forces you to learn the material more robustly instead of memorizing one page's layout.

Let's see data augmentation in action on a tiny 2D dataset (no images needed).

from sklearn.datasets import make_moons
from sklearn.neighbors import KNeighborsClassifier

# A small training set (intentionally small, so it overfits easily)
X, y = make_moons(n_samples=60, noise=0.30, random_state=1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=1)

# Baseline: KNN on the original (tiny) training set
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(Xtr, ytr)
base_train = knn.score(Xtr, ytr); base_test = knn.score(Xte, yte)

# Data augmentation: add Gaussian jitter (noise) to copies of training points.
# Same labels -- we're inventing "nearby but slightly moved" moons.
rng = np.random.RandomState(1)
X_aug = [Xtr]; y_aug = [ytr]
for _ in range(20):                       # make 20 jittered copies
    X_aug.append(Xtr + rng.normal(0, 0.10, size=Xtr.shape))
    y_aug.append(ytr)
Xtr_aug = np.vstack(X_aug); ytr_aug = np.concatenate(y_aug)

knn_aug = KNeighborsClassifier(n_neighbors=3)
knn_aug.fit(Xtr_aug, ytr_aug)
aug_train = knn_aug.score(Xtr, ytr); aug_test = knn_aug.score(Xte, yte)

print("Original data : train %.3f, test %.3f" % (base_train, base_test))
print("Augmented data: train %.3f, test %.3f" % (aug_train, aug_test))

# Visualize the original vs augmented training points
plt.figure(figsize=(6,4))
plt.scatter(Xtr[:,0], Xtr[:,1], c=ytr, cmap='coolwarm', edgecolor='k', s=60, label='original')
aug_X = Xtr_aug[len(Xtr):]; aug_y = np.concatenate(y_aug)[len(ytr):]
plt.scatter(aug_X[:,0], aug_X[:,1], c=aug_y, cmap='coolwarm', alpha=0.15, s=15, label='augmented (jittered)')
plt.title("Data augmentation: jittered copies of the training points")
plt.xlabel("x1"); plt.ylabel("x2"); plt.legend(); plt.tight_layout(); plt.show()

Original data : train 0.944, test 0.917
Augmented data: train 1.000, test 0.917

What augmentation did

By adding jittered copies, we effectively told the model "the moon shape is fuzzy — points near these ones belong to the same class." The augmented model usually generalizes a bit better to the test set even though it's the same KNN with the same settings. That's the whole idea of augmentation: cheap, label-preserving diversity that fights overfitting.

Dropout and batch normalization aren't exposed in sklearn's MLPClassifier in a fine-grained way, but the concept is identical — and you'll meet them the moment you touch a deep-learning library like Keras.

8.5 Handling Multiple Inputs

Many real problems are multimodal: the input is several different kinds of thing at once. "Does this text describe this image?" combines an image and a sentence. How do you feed both into one model?

With shallow models, two options:

1. Train separate models on each input, then combine their predictions (averaging / voting / stacking — see the previous section).
2. Vectorize each input separately, then concatenate the feature vectors into one long vector. If the image gives features [i1, i2, i3] and the text gives [t1, t2, t3, t4], you feed the model [i1, i2, i3, t1, t2, t3, t4].

With neural networks you get more flexibility: build a subnetwork for each input (a CNN for the image, an RNN for the text), let each produce an embedding (a compact vector summary), then concatenate the embeddings and add a classification layer on top. Libraries like Keras make this "two towers, then merge, then classify" pattern easy.

Everyday analogy: judging a restaurant from both the menu (text) and the photos (images). You could score each separately and average, or read them together for a combined verdict — the combined view is usually richer.

Let's illustrate the simplest version — concatenating two feature sources.

from sklearn.tree import DecisionTreeClassifier

# Toy "multimodal" setup: the SAME samples described by two different feature blocks.
# "Modality A": 3 numeric features ; "Modality B": 4 numeric features.
# The target depends on information split across BOTH modalities.
rng = np.random.RandomState(5)
n = 300
A = rng.normal(size=(n, 3))          # modality A (e.g. "image features")
B = rng.normal(size=(n, 4))          # modality B (e.g. "text features")
# Label = 1 if a combo across A and B exceeds a threshold
y = (A[:,0] + B[:,2] > 0.5).astype(int)

# Same random_state + same row count => identical split indices for A and B
Xtr_A, Xte_A, ytr, yte = train_test_split(A, y, test_size=0.3, random_state=5)
Xtr_B, Xte_B, _, _ = train_test_split(B, y, test_size=0.3, random_state=5)

m_A = DecisionTreeClassifier(max_depth=3, random_state=5).fit(Xtr_A, ytr)
m_B = DecisionTreeClassifier(max_depth=3, random_state=5).fit(Xtr_B, ytr)

# Concatenated input: both modalities side by side
X_both = np.hstack([A, B])
Xtr_both, Xte_both, _, _ = train_test_split(X_both, y, test_size=0.3, random_state=5)
m_both = DecisionTreeClassifier(max_depth=3, random_state=5).fit(Xtr_both, ytr)

print("Only modality A : test acc %.3f" % m_A.score(Xte_A, yte))
print("Only modality B : test acc %.3f" % m_B.score(Xte_B, yte))
print("Both (concat)   : test acc %.3f" % m_both.score(Xte_both, yte))

Only modality A : test acc 0.744
Only modality B : test acc 0.767
Both (concat)   : test acc 0.900

Reading the result

Neither single modality has the full story (the label depends on both A[:,0] and B[:,2]), so each alone struggles. Concatenating them gives the model everything it needs in one vector — and accuracy jumps. That's the shallow-learning recipe for multimodal input in a nutshell.

For genuinely different types of data (pixels vs. words), the neural-network "two towers plus merge" approach scales better — but the underlying idea is the same: turn each input into a vector, then let the model see them together.

8.6 Handling Multiple Outputs

Sometimes one input should produce several outputs. Detect an object in a photo and return both its bounding-box coordinates and its category ("cat" / "dog" / "hamster").

Some multi-output problems can be flattened into multi-label classification (especially when the outputs are tags of the same kind). But when the outputs are different types — a vector of real numbers (coordinates) and a class label — flattening doesn't work.

The neural-network solution: one encoder subnetwork reads the input and produces an embedding. Then two heads branch off the embedding:

• Head 1 predicts coordinates (a regression head, often with ReLU output, trained with mean-squared-error cost C1).
• Head 2 predicts the class label (a softmax head, trained with cross-entropy cost C2).

You can't minimize both costs at once perfectly — improving one can hurt the other. The standard trick is a combined cost: C = alpha*C1 + (1-alpha)*C2, where alpha is a hyperparameter in (0, 1) you tune on validation data. It says "this much do I care about coordinates, the rest about the label."

Everyday analogy: a teacher grading both math correctness and writing neatness on the same exam. They combine the two sub-scores into one grade, weighting them by how much each matters.

Let's illustrate multi-output prediction with sklearn (one input, two targets).

from sklearn.multioutput import MultiOutputRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

# One input x -> TWO numeric outputs (a 2D regression target)
rng = np.random.RandomState(9)
X = rng.uniform(-3, 3, size=(120, 1))
# Output 1 ~ sin(x), Output 2 ~ 0.5*x^2 (very different shapes)
y_multi = np.column_stack([np.sin(X[:,0]) + rng.normal(0, 0.1, 120),
                           0.5 * X[:,0]**2 + rng.normal(0, 0.3, 120)])

Xtr, Xte, ytr, yte = train_test_split(X, y_multi, test_size=0.3, random_state=9)

# MultiOutputRegressor fits one independent model per output target
mor = MultiOutputRegressor(LinearRegression())
mor.fit(Xtr, ytr)
pred = mor.predict(Xte)

# r2_score with multioutput='raw_values' gives one R^2 per output target
per_target = r2_score(yte, pred, multioutput='raw_values')
print("Per-target R^2 on test set:", ["%.3f" % v for v in per_target])
print("Average R^2             : %.3f" % mor.score(Xte, yte))

# Plot both true targets and predictions
order = np.argsort(Xte[:,0])
plt.figure(figsize=(7,4))
plt.scatter(Xte[order,0], yte[order,0], s=20, label='true out 1 (sin)')
plt.plot(Xte[order,0], pred[order,0], 'r-', lw=1.5, label='pred out 1')
plt.scatter(Xte[order,0], yte[order,1], s=20, marker='s', label='true out 2 (0.5x^2)')
plt.plot(Xte[order,0], pred[order,1], 'g-', lw=1.5, label='pred out 2')
plt.title("Multiple outputs from one input")
plt.xlabel("x"); plt.ylabel("target values"); plt.legend(); plt.tight_layout(); plt.show()

Per-target R^2 on test set: ['0.717', '-0.026']
Average R^2             : 0.346

8.7 Transfer Learning

This is where neural networks shine brightest. Transfer learning means: take a model trained on one dataset, and adapt it to a different but related problem.

The classic recipe (for neural networks):

1. Train a deep model on a big "source" dataset (e.g. wild animals).
2. Gather a much smaller labeled dataset for your real "target" problem (e.g. domestic animals).
3. Remove the last layer(s) of the source model (the task-specific head — usually everything after the embedding layer).
4. Add fresh layers suited to your new problem.
5. Freeze the old layers (don't update their weights).
6. Train only the new layers on your small dataset with gradient descent.

The old layers have already learned generic, reusable features (edges, textures, shapes); you're just bolting a new "decision head" on top. This lets you get great results from surprisingly little labeled data.

Transfer learning isn't limited to reusing whole networks. A simpler, shallow version of the same idea: fit a transformer (like PCA or a scaler) on one dataset, then reuse its learned .transform() on a different dataset. Let's see that lightweight version.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# "Source" problem: a BIG labeled dataset (wild-animal analogue)
X_src, y_src = make_classification(n_samples=2000, n_features=20, n_informative=10,
                                   n_redundant=5, random_state=11)
# "Target" problem: a SMALL labeled dataset, different distribution (domestic-animal analogue)
X_tgt, y_tgt = make_classification(n_samples=150, n_features=20, n_informative=10,
                                   n_redundant=5, flip_y=0.05, random_state=99)

Xtr_t, Xte_t, ytr_t, yte_t = train_test_split(X_tgt, y_tgt, test_size=0.3,
                                              random_state=99, stratify=y_tgt)

# --- Baseline: train from scratch on the small target data only ---
from sklearn.pipeline import make_pipeline
scratch = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scratch.fit(Xtr_t, ytr_t)
print("From scratch (small data only) : test acc %.3f" % scratch.score(Xte_t, yte_t))

# --- Transfer: fit PCA on the BIG source data, REUSE it on the target data ---
# We "freeze" the PCA (fit on source) and only train the classifier on target.
pca = PCA(n_components=8, random_state=11).fit(StandardScaler().fit_transform(X_src))

scaler_t = StandardScaler().fit(Xtr_t)
Xtr_t_pca = pca.transform(scaler_t.transform(Xtr_t))
Xte_t_pca = pca.transform(scaler_t.transform(Xte_t))
transfer_clf = LogisticRegression(max_iter=1000).fit(Xtr_t_pca, ytr_t)
print("Transfer (PCA from source data) : test acc %.3f" % transfer_clf.score(Xte_t_pca, yte_t))

From scratch (small data only) : test acc 0.689
Transfer (PCA from source data) : test acc 0.622

Reading the result

The PCA was trained on the large source dataset, so it found good low-dimensional directions (features). We then reused those directions to compress the small target dataset before training a fresh classifier — a miniature of "freeze the feature extractor, retrain the head." Depending on how related the two problems are, this can match or beat training from scratch on limited data.

In real transfer learning with deep nets you'd reuse a whole pretrained network (downloadable online) instead of just a PCA, but the philosophy is identical: don't throw away learned features — repurpose them.

8.8 Algorithmic Efficiency

Not every algorithm that works is practical. As your dataset grows, some algorithms slow down gracefully and some blow up. Big-O notation describes how running time (or memory) grows as the input size N grows, ignoring constant factors:

• O(N) — time grows roughly linearly with N. Double the data, double the time.
• O(N squared) — time grows with the square of N. Double the data, four times the time.
• O(log N) — time grows with the logarithm of N. Double the data, add just one extra step. Very fast.
• O(N log N) — between linear and quadratic; the sweet spot for sorting-like problems.

An algorithm is usually called efficient if its complexity is polynomial in N (so O(N), O(N squared), O(N cubed) all count). But in the big-data era, even O(N squared) can be too slow, and people hunt for O(N log N) or O(log N) solutions.

Everyday analogy: looking up a word in a dictionary. Flipping every page one-by-one is O(N). Flipping to the middle and throwing away the wrong half each time (binary search) is O(log N) — a 1,000-page dictionary takes about 10 flips instead of up to 1,000.

Let's feel the difference with a tiny timing experiment.

import time

# Two ways to find the two most distant 1-D numbers in a list S of size N.
def find_max_distance_slow(S):           # O(N^2): compare every pair
    best = 0; result = None
    for x1 in S:
        for x2 in S:
            d = abs(x1 - x2)
            if d >= best:
                best = d; result = (x1, x2)
    return result

def find_max_distance_fast(S):           # O(N): just track min and max
    mn = float('inf'); mx = float('-inf')
    for x in S:
        if x < mn: mn = x
        if x > mx: mx = x
    return (mx, mn)

sizes = [500, 1000, 2000, 4000]
slow_t, fast_t = [], []
rng = np.random.RandomState(0)
for N in sizes:
    S = rng.normal(size=N)
    t0 = time.perf_counter(); find_max_distance_slow(S); slow_t.append(time.perf_counter()-t0)
    t0 = time.perf_counter(); find_max_distance_fast(S); fast_t.append(time.perf_counter()-t0)

plt.figure(figsize=(6,4))
plt.plot(sizes, slow_t, 'o-', label='O(N^2) slow')
plt.plot(sizes, fast_t, 's-', label='O(N) fast')
plt.xlabel("input size N"); plt.ylabel("time (seconds)")
plt.title("Big-O in action: same answer, very different speed")
plt.legend(); plt.tight_layout(); plt.show()
print("Slow times:", ["%.4fs" % t for t in slow_t])
print("Fast times:", ["%.4fs" % t for t in fast_t])

Slow times: ['1.6167s', '3.6168s', '8.9517s', '45.3968s']
Fast times: ['0.0011s', '0.0013s', '0.0010s', '0.0173s']

Practical efficiency tips

• Avoid Python loops for math — use numpy vector operations (numpy.dot(w, x)) instead of element-by-element loops.
• Pick the right data structure — use set for membership tests (fast) instead of list (slow); use dict for key-to-value lookups.
• Prefer battle-tested libraries (numpy, scipy, scikit-learn) — their core routines are written in C for speed.
• Use generators to stream huge collections one element at a time instead of loading them all into memory.
• Profile first with cProfile to find the actual bottleneck.
• When the algorithm itself can't be improved: parallelize with multiprocessing, or JIT-compile with Numba / PyPy.

Typical complexity of common ML algorithms (sklearn-style)

Algorithm	Training	Prediction (per sample)
k-NN	O(1) (just store data)	O(N * d) (scan all training points)
Logistic Regression	O(N * d * iters)	O(d)
Decision Tree	O(N * d * log N)	O(depth)
Random Forest (T trees)	O(T * N * d * log N)	O(T * depth)
SVM (kernel)	O(N^2 to N^3)	O(#support vectors * d)
k-Means	O(N * k * d * iters)	O(k * d)

Here N = number of training examples, d = number of features, k = number of clusters, T = number of trees, and "iters" = iterations of the optimizer. The takeaway: k-NN trains instantly but predicts slowly; SVMs train expensively; linear models stay cheap everywhere.

Key Takeaways

• Imbalanced data breaks accuracy. Use balanced accuracy / F1 to see the minority class, and fix it with class_weight='balanced', resampling, or threshold tuning.
• Combining uncorrelated models (averaging, voting, stacking) often beats any single model — diversity is the secret ingredient.
• Neural-network training dials: pick a learning-rate schedule, mind your batch size, and use early stopping as your first line of defense against overfitting.
• Advanced regularization = dropout (randomly silence neurons), batch normalization (standardize between layers), and data augmentation (invent label-preserving training examples).
• Multiple inputs: vectorize each modality and concatenate, or (for neural nets) build a subnetwork per input and merge their embeddings.
• Multiple outputs: branch several heads off one embedding; combine their costs as alpha*C1 + (1-alpha)*C2 and tune alpha.
• Transfer learning: reuse a pretrained model's features and retrain only a new head — a huge saver of labeled data, and a unique strength of deep nets.
• Big-O thinking: prefer O(N) or O(N log N) over O(N squared); vectorize, choose smart data structures, and profile before you optimize.

What's Next

We've spent two chapters on practical supervised learning. Next, in Chapter 9 — Unsupervised Learning, we drop the labels entirely and ask a different question: what structure hides in the data on its own? We'll meet clustering, dimensionality reduction, and anomaly detection.

---

---

Chapter 9 — Unsupervised Learning

Up to now, almost every dataset we touched came with labels — the "right answer" written next to each example. Unsupervised learning is what we do when those answers are missing. We hand the algorithm a pile of unlabeled examples and ask it to find structure on its own: groups, a simpler description, the shape of the distribution, or a guess about what a "typical" example looks like.

In this chapter you will learn:

• How to estimate the shape of a data distribution (density estimation), with and without assuming a bell curve
• How to group similar examples with k-means, Gaussian mixture models, hierarchical clustering, and DBSCAN
• How to choose the number of clusters with the elbow method
• How to squeeze many features into a few with Principal Component Analysis (PCA)
• How recommender systems guess missing ratings with collaborative filtering

9.1 Density Estimation

Density estimation asks: "I have a sample of data points — what curve did they probably come from?" The answer is an estimate of the probability density function (pdf), the function f whose value at any point x tells you how likely data is to appear near x.

Two families of approaches exist:

• Parametric: you assume a specific shape — most commonly the bell-shaped Gaussian (normal) distribution — and just fit its parameters: the mean μ and the variance σ² (or the covariance matrix Σ in higher dimensions). Fast and simple, but if the real data isn't bell-shaped, the fit is poor.
• Non-parametric: you make no strong shape assumption. Kernel density estimation (KDE) is the classic example: you drop a little smooth "bump" (a kernel, usually a tiny Gaussian) on top of every data point and add them all up. The result is a smooth curve that follows whatever shape the data actually has.

A bandwidth b controls how wide each bump is. Too small and every point becomes its own spike (overfitting); too large and everything melts into one fat blob (underfitting). It's the same bias–variance trade-off we keep meeting.

Everyday analogy: parametric estimation is like insisting "the crowd must be one normal blob, let me find its center." KDE is like pouring a handful of sand on every person in the crowd and looking at the resulting dune — it reveals whatever shape the crowd really has, even if it's two separate groups.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.neighbors import KernelDensity

plt.rcParams["figure.figsize"] = (7, 4)
plt.rcParams["axes.grid"] = True

# Build a 1-D dataset that is clearly NOT a single bell curve:
# two separate blobs stitched together (a bimodal distribution).
np.random.seed(42)
data = np.concatenate([
    np.random.normal(loc=-3.0, scale=0.8, size=120),
    np.random.normal(loc=3.0, scale=0.8, size=120),
])

# --- Parametric fit: assume ONE Gaussian, estimate mean & variance ---
mu_hat = data.mean()
sigma2_hat = data.var()

# --- Non-parametric fit: Kernel Density Estimation ---
kde = KernelDensity(bandwidth=0.5, kernel="gaussian").fit(data[:, None])

# Grid of x values where we will evaluate both estimates
x_grid = np.linspace(-7, 7, 400)

# Gaussian pdf:  1/sqrt(2*pi*sigma^2) * exp(-(x-mu)^2 / (2*sigma^2))
gaussian_pdf = (1.0 / np.sqrt(2 * np.pi * sigma2_hat)) * \
               np.exp(-(x_grid - mu_hat) ** 2 / (2 * sigma2_hat))

# KDE pdf: score_samples returns log-density, so we exponentiate
kde_pdf = np.exp(kde.score_samples(x_grid[:, None]))

# Plot the histogram of the data plus both density estimates
plt.hist(data, bins=30, density=True, color="lightgray", edgecolor="white",
         label="data histogram")
plt.plot(x_grid, gaussian_pdf, color="tab:red", lw=2,
         label=f"Gaussian fit (mu={mu_hat:.1f})")
plt.plot(x_grid, kde_pdf, color="tab:blue", lw=2, label="KDE (bandwidth=0.5)")
plt.title("Density Estimation: one Gaussian vs KDE")
plt.xlabel("value")
plt.ylabel("density")
plt.legend()
plt.show()

Reading the plot

• The gray histogram is the raw data — notice the two peaks around −3 and +3.
• The red Gaussian tries to describe all of it with a single bell centered at 0. It spreads wide to "cover" both bumps, but it dips where the data actually peaks and rises where the data is actually empty. That is the cost of a wrong assumption.
• The blue KDE places a small bump on each point and adds them up, so it naturally reproduces the two peaks. No shape assumption needed — just a bandwidth choice.

When you have no reason to believe the data is bell-shaped, KDE is the safer bet. When you do expect a single bell (many measurement errors are roughly Gaussian), the parametric fit is cheaper and good enough.

9.2 Clustering

Clustering is the unsupervised version of classification. We want to assign each example a cluster label — but nobody gave us the labels, and we don't even know how many groups there are. The algorithm has to discover the grouping from the geometry of the data alone.

This makes clustering hard to evaluate: with no true labels, "is this clustering good?" becomes a judgment call. Different algorithms make different geometric assumptions, so the "best" one depends on the unknown shape of your data. Let's meet the four most useful families.

9.2.1 K-Means

K-means is the workhorse of clustering. You choose k (the number of clusters) and the algorithm does this:

1. Initialize: place k points called centroids somewhere in the feature space (often randomly).
2. Assign: give every example the label of its nearest centroid (Euclidean distance).
3. Update: move each centroid to the mean (average position) of all the examples now assigned to it.
4. Repeat steps 2–3 until the assignments stop changing.

The quantity it minimizes is the sum of squared distances from each point to its centroid — scikit-learn calls this inertia. Lower inertia means tighter clusters.

$$\text{inertia} = \sum_{i=1}^{N} \min_{j}\ \lVert x_i - c_j \rVert^2$$

Two catches: you must pick k yourself, and because the start is random, two runs can give different answers (scikit-learn runs several starts and keeps the best by default).

Everyday analogy: k-means is like dropping k postmen in a city, having each house join its nearest postman, then moving each postman to the middle of their new round, and repeating until the postmen settle.

from sklearn.cluster import KMeans

# Three natural groups in 2-D (we pretend we don't know the labels)
X, y_true = make_blobs(n_samples=300, centers=3, cluster_std=0.9, random_state=7)

# Fit k-means with k=3. n_init=10 tries 10 random starts and keeps the best.
km = KMeans(n_clusters=3, n_init=10, random_state=7)
labels = km.fit_predict(X)

plt.scatter(X[:, 0], X[:, 1], c=labels, cmap="tab10", s=20, edgecolor="white")
# Mark the learned centroids with big red X's
plt.scatter(km.cluster_centers_[:, 0], km.cluster_centers_[:, 1],
            marker="X", s=200, color="red", edgecolor="black", label="centroids")
plt.title("K-Means clustering (k=3)")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(

Reading the plot

The three colored clouds are the clusters k-means found, and the big red X marks are the final centroids — each sits at the average of its cluster's points. Because k-means measures distance with straight lines, it produces round (hyperspherical) clusters. If your real groups are stretched, nested, or crescent-shaped, k-means will force them into circles and get it wrong — we'll see this clearly when we meet DBSCAN.

# Try k from 1 to 8 and record the inertia (within-cluster sum of squares)
ks = range(1, 9)
inertias = []
for k in ks:
    km = KMeans(n_clusters=k, n_init=10, random_state=7)
    km.fit(X)
    inertias.append(km.inertia_)

plt.plot(list(ks), inertias, marker="o", color="tab:purple")
plt.axvline(3, color="tab:red", linestyle="--", label="k=3 (the 'elbow')")
plt.title("Elbow method: inertia vs number of clusters")
plt.xlabel("number of clusters k")
plt.ylabel("inertia (sum of squared distances to centroid)")
plt.legend()
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(

Reading the elbow

Inertia always decreases as k grows — add more centroids and points get closer to one. So we don't look for the minimum; we look for the kink (the "elbow") where the drop suddenly flattens. Here inertia falls fast from k=1 to k=3, then barely improves after. That kink at k=3 matches the true number of blobs. The elbow method is subjective (the kink isn't always obvious), but it's a quick, practical first guess. The book also mentions more rigorous options: prediction strength and the gap statistic.

9.2.2 Gaussian Mixture Models (EM)

K-means gives a hard assignment: each point belongs to exactly one cluster. A Gaussian Mixture Model (GMM) is softer. It models the data as a weighted sum of k Gaussian blobs:

$$f(x) = \sum_{j=1}^{k} \pi_j\, \mathcal{N}(x \mid \mu_j, \Sigma_j)$$

Each cluster j has a mean μⱼ, a covariance Σⱼ (which lets the blob be an ellipse of any size, stretch, and rotation), and a weight πⱼ (how big that cluster is overall). The parameters are learned with the Expectation-Maximization (EM) algorithm, which alternates two steps:

• E-step: given the current Gaussians, compute for each point the probability it came from each cluster (a soft assignment).
• M-step: given those probabilities, update each Gaussian's mean, covariance, and weight (using probability-weighted averages).

Repeat until the parameters stop moving. It's k-means' cousin: k-means is the special case where every point is 100% in one cluster and the blobs are identical round spheres. GMM's ellipses and soft probabilities let it handle overlapping clusters that k-means would chop arbitrarily.

from sklearn.mixture import GaussianMixture
from matplotlib.patches import Ellipse

# Two overlapping blobs so soft assignment actually matters
Xg, _ = make_blobs(n_samples=220, centers=[(-2, 0), (2, 0)],
                   cluster_std=1.6, random_state=3)

gmm = GaussianMixture(n_components=2, covariance_type="full",
                      random_state=3, n_init=1, max_iter=100)
gmm.fit(Xg)
labels_g = gmm.predict(Xg)

plt.scatter(Xg[:, 0], Xg[:, 1], c=labels_g, cmap="coolwarm", s=20, edgecolor="white")

# Draw an ellipse for each component from its mean & covariance
ax = plt.gca()
for j in range(gmm.n_components):
    mean = gmm.means_[j]
    cov = gmm.covariances_[j]
    # Eigen-decomposition gives the ellipse's axes & rotation
    eigvals, eigvecs = np.linalg.eigh(cov)
    order = eigvals.argsort()[::-1]
    eigvals, eigvecs = eigvals[order], eigvecs[:, order]
    angle = np.degrees(np.arctan2(eigvecs[1, 0], eigvecs[0, 0]))
    # 2-sigma ellipse: full width/height = 2 * (2 * sigma) = 4 * sqrt(eigenvalue)
    width, height = 4 * np.sqrt(eigvals)
    ell = Ellipse(xy=mean, width=width, height=height, angle=angle,
                  edgecolor="black", facecolor="none", lw=2, linestyle="--")
    ax.add_patch(ell)

plt.title("Gaussian Mixture Model: soft clusters with 2-sigma ellipses")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=1.
  warnings.warn(

Reading the plot

The two colors are the most-likely cluster for each point, and the dashed ellipses show each Gaussian's shape (a 2σ contour). Notice the ellipses overlap in the middle — points there genuinely could belong to either cluster. With gmm.predict_proba(X) you'd get the actual soft probabilities (e.g. 0.7 / 0.3) instead of a forced single label. Because the covariance matrix can stretch and rotate, GMM captures elongated, tilted clusters that round k-means would misplace.

9.2.3 Hierarchical Clustering

Hierarchical clustering builds a tree (a dendrogram) of clusters. The common agglomerative (bottom-up) version starts with every point as its own cluster, then repeatedly merges the two closest clusters until only one giant cluster remains. You don't have to pick k up front: you build the whole tree once, then "cut" it at the height that gives you the number of clusters you want.

A dendrogram visualizes these merges: each horizontal line is a merge, and its height shows how dissimilar the two clusters were when they joined. Tall merges mean "these groups are far apart" — a natural place to cut.

Everyday analogy: like a family tree of species, built by joining the most similar organisms first. Cut the tree high to get a few big kingdoms; cut it low to get many small families.

from scipy.cluster.hierarchy import linkage, dendrogram

# Tiny dataset so the dendrogram stays readable
Xh, _ = make_blobs(n_samples=20, centers=3, cluster_std=0.8, random_state=11)

# linkage matrix: 'ward' merges the pair that least increases total variance
Z = linkage(Xh, method="ward")

dendrogram(Z, color_threshold=4.0)
plt.title("Hierarchical clustering dendrogram (20 points)")
plt.xlabel("example index")
plt.ylabel("merge distance")
plt.show()

Reading the dendrogram

The three colored branches near the bottom are the three tight clusters forming first. They then join each other higher up (around distance 4–6) — that big jump in height is the visual cue that three is a natural number of clusters. Cutting the tree with a horizontal line at height ≈ 4 would yield exactly three clusters. The dendrogram is a nice bonus: it shows the whole hierarchy, not just one flat answer.

9.2.4 DBSCAN

DBSCAN is density-based. Instead of asking for the number of clusters, you give it two numbers:

• ε (eps): the radius to look around each point.
• min_samples: how many points must sit within that radius for a spot to be "crowded."

It grows each cluster from a crowded point outward, absorbing neighbors, then their neighbors, and so on, until it hits a region that's too sparse. Points in sparse regions are labeled outliers (cluster −1). The payoff: DBSCAN finds clusters of arbitrary shape — rings, crescents, blobs — because it follows density, not distance to a center.

The catch: picking ε is fiddly, and one fixed ε struggles when clusters have very different densities (the book recommends HDBSCAN to fix that — it keeps DBSCAN's strengths but only needs min_samples).

Let's put DBSCAN head-to-head with k-means on two interleaved moons, a shape k-means fundamentally cannot handle.

from sklearn.cluster import DBSCAN
from sklearn.datasets import make_moons

# Two interleaved half-moons -- a classic non-convex shape
Xm, _ = make_moons(n_samples=250, noise=0.07, random_state=9)

# K-means insists on 2 round clusters -- it will slice each moon in half
km_m = KMeans(n_clusters=2, n_init=10, random_state=9)
labels_km = km_m.fit_predict(Xm)

# DBSCAN follows the density of each moon
db = DBSCAN(eps=0.2, min_samples=5)
labels_db = db.fit_predict(Xm)

fig, axes = plt.subplots(1, 2, figsize=(11, 4))

axes[0].scatter(Xm[:, 0], Xm[:, 1], c=labels_km, cmap="coolwarm", s=20, edgecolor="white")
axes[0].set_title("K-Means on two moons (FAILS)")
axes[0].set_xlabel("feature 1")
axes[0].set_ylabel("feature 2")

# Mark DBSCAN outliers (-1) in black
outlier = labels_db == -1
axes[1].scatter(Xm[~outlier, 0], Xm[~outlier, 1], c=labels_db[~outlier],
                cmap="coolwarm", s=20, edgecolor="white")
axes[1].scatter(Xm[outlier, 0], Xm[outlier, 1], color="black", s=30,
                marker="x", label="outlier")
axes[1].set_title("DBSCAN on two moons (follows the curves)")
axes[1].set_xlabel("feature 1")
axes[1].set_ylabel("feature 2")
axes[1].legend()

plt.tight_layout()
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=1.
  warnings.warn(

Reading the comparison

• Left (k-means): forced to produce two round regions, it draws a straight cut right through the middle of each moon. Each "cluster" contains half of the top moon and half of the bottom one — geometrically wrong even though k=2 was "correct."
• Right (DBSCAN): it simply walks along each dense crescent, labeling the whole top moon as one cluster and the whole bottom moon as another. A few sparse points (black ×) are flagged as outliers rather than forced into a group.

The lesson: match the algorithm's assumption to the data's shape. Round blobs → k-means. Overlapping tilted blobs → GMM. Weird shapes / unknown count → DBSCAN/HDBSCAN.

Algorithm	Assumes	Picks k?	Cluster shape	Outliers?
K-Means	round, similar-size blobs	yes (you set it)	spherical	no
GMM (EM)	Gaussian ellipses	yes (you set it)	elliptical	no
Hierarchical	a merge tree	cut later	flexible	no
DBSCAN	dense regions	no	arbitrary	yes

9.3 Dimensionality Reduction

Sometimes you have too many features — hundreds or thousands — and most are redundant, noisy, or correlated with each other. Dimensionality reduction rewrites each example using fewer new features while keeping as much information as possible. Two classic payoffs:

• Visualization: humans can read at most 3D plots, so we squash high-dimensional data to 2D/3D to "see" it.
• Simpler, cleaner models: fewer features mean faster training, less overfitting, and often better interpretability.

The book names three workhorses: Principal Component Analysis (PCA), UMAP, and autoencoders. We'll focus on PCA (the oldest and most transparent) and sketch the others.

9.3.1 Principal Component Analysis (PCA)

PCA finds a new set of axes — the principal components — that are just rotations of the original features. The first component points in the direction of the greatest variance (spread) in the data; the second is perpendicular to it and points in the direction of the next greatest variance; and so on. Each component comes with a number: how much variance it explains.

To reduce to D_new dimensions, you keep the top D_new components and project the data onto them. Because the early components capture most of the spread, dropping the later ones usually loses little. You can also reconstruct an approximation of the original point by going back up — the lost dimensions are exactly the low-variance ones, which mostly held noise.

Everyday analogy: photographing a 3D object from the angle that shows the most detail. The first photo captures the longest silhouette; a second photo from the side captures what's left. Throw away the blurry head-on shot and you've kept the essence in two pictures.

from sklearn.decomposition import PCA

# A synthetic 6-dimensional dataset with strong structure:
# 3 tight blobs live roughly in a 2-D plane, so 2 components should
# capture most of the variance.
Xp, yp = make_blobs(n_samples=200, centers=3, n_features=6,
                    cluster_std=0.5, random_state=5)

pca = PCA(n_components=2)
Xp_2d = pca.fit_transform(Xp)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# 2-D projection colored by the true (hidden) blob id
axes[0].scatter(Xp_2d[:, 0], Xp_2d[:, 1], c=yp, cmap="coolwarm",
                s=25, edgecolor="white")
axes[0].set_title("PCA: 6D data projected to 2D")
axes[0].set_xlabel("1st principal component")
axes[0].set_ylabel("2nd principal component")

# Explained variance ratio for ALL 6 components
pca_full = PCA(n_components=6).fit(Xp)
axes[1].bar(range(1, 7), pca_full.explained_variance_ratio_,
            color="tab:purple", label="per component")
axes[1].plot(range(1, 7), np.cumsum(pca_full.explained_variance_ratio_),
             marker="o", color="tab:red", label="cumulative")
axes[1].set_title("Explained variance ratio (6 features)")
axes[1].set_xlabel("principal component")
axes[1].set_ylabel("fraction of variance explained")
axes[1].legend()

plt.tight_layout()
plt.show()

Reading the PCA plots

• Left: even though each point really has 6 numbers, PCA found two new axes that separate the three blobs almost perfectly. We can now plot 6-dimensional data on a flat page.
• Right: the first two purple bars are tall and the red cumulative line jumps to near 1.0 by component 2 — meaning two components capture almost all the variance. The remaining four components explain almost nothing (mostly noise), so dropping them costs us little.

# Reconstruct the 6-D points from only 2 components and compare to originals.
pca2 = PCA(n_components=2).fit(Xp)
Xp_compressed = pca2.transform(Xp)               # 2-D code
Xp_reconstructed = pca2.inverse_transform(Xp_compressed)  # back to 6-D (approx)

print("Original      point 0:", np.round(Xp[0], 2))
print("Reconstructed point 0:", np.round(Xp_reconstructed[0], 2))
print()
print("Original      point 1:", np.round(Xp[1], 2))
print("Reconstructed point 1:", np.round(Xp_reconstructed[1], 2))
print()

# Mean squared reconstruction error across all points and features
err = np.mean((Xp - Xp_reconstructed) ** 2)
print(f"Mean squared reconstruction error: {err:.3f}")

Original      point 0: [-0.52 -7.49  7.42 -4.5  -2.24 -4.28]
Reconstructed point 0: [-0.68 -7.05  7.68 -4.93 -1.92 -4.04]

Original      point 1: [-0.35 -6.64  7.64 -4.6  -2.01 -4.27]
Reconstructed point 1: [-0.72 -6.88  7.51 -4.78 -1.91 -3.95]

Mean squared reconstruction error: 0.159

Reading the reconstruction

The reconstructed 6-number vectors are close to the originals but not exact — PCA kept the two "important" directions and threw away the four low-variance ones. The mean squared reconstruction error is small precisely because the discarded components held little variance. This is the core PCA trade: give up a little accuracy for a big drop in size.

9.3.2 UMAP and Autoencoders (conceptual)

PCA is fast and linear, but it can only capture straight-axis structure. Two more powerful alternatives:

• UMAP (and its cousin t-SNE) are non-linear methods built for visualization. They define a similarity between nearby points in the high-dimensional space, then place points in a 2D/3D space so those similarities are preserved as well as possible (the book writes this as a cross-entropy between the high- and low-dimensional similarity graphs). The result: clusters of weird shape often separate cleanly on a 2D plot — better than PCA for eyeballing. Slower than PCA, faster than an autoencoder.

• Autoencoders are tiny neural nets trained to copy their input to their output through a narrow bottleneck layer. Because the bottleneck has fewer neurons than the input, the network is forced to find a compact code. That bottleneck output is your reduced representation; the decoder half reconstructs the original. We met autoencoders in Chapter 7 — they also double as outlier detectors (an outlier reconstructs badly).

Reach for PCA when you want speed and interpretability, UMAP/t-SNE when you want an honest-to-the-eye 2D picture, and an autoencoder when the data is highly non-linear and you need a learned, reusable encoder.

9.4 Collaborative Filtering

How does a streaming service know you'll like a show you've never seen? Collaborative filtering is the classic recommender idea: use the ratings of many users to predict the rating of one user. The data lives in a user–item matrix R, where R[u, i] is the rating user u gave to item i — and most entries are missing (nobody has rated everything).

Two simple flavors:

• User-based: find users whose taste is most similar to yours (by comparing rating vectors with cosine similarity), then predict your missing rating as a weighted average of what those similar users gave the item.
• Item-based: find items most similar to the target item (again by cosine similarity over who-rated-what), then predict your rating as a weighted average of your own ratings on those similar items.

A more powerful approach is matrix factorization: approximate the big sparse matrix R as the product of two small matrices — a user matrix and an item matrix. Each user and item gets a short "taste vector," and a predicted rating is just the dot product of the two vectors. This is what won the Netflix Prize and still powers many real recommenders.

Everyday analogy: asking friends who share your taste to recommend a movie. The more a friend's past likes overlap with yours, the more you trust their suggestion — that weighting is exactly cosine similarity at work.

import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

# Tiny ratings matrix: 5 users x 5 movies, 0 means "not rated"
ratings = np.array([
    [5, 4, 0, 1, 0],   # user 0
    [4, 5, 0, 1, 0],   # user 1
    [1, 0, 5, 4, 4],   # user 2
    [0, 1, 4, 5, 5],   # user 3
    [5, 4, 1, 0, 0],   # user 4
])
users  = ["U0", "U1", "U2", "U3", "U4"]
movies = ["M0", "M1", "M2", "M3", "M4"]

df = pd.DataFrame(ratings, index=users, columns=movies)
print("Ratings matrix (0 = unrated):")
print(df)
print()

# Heatmap of the raw ratings
plt.imshow(ratings, cmap="YlGnBu", aspect="auto")
plt.colorbar(label="rating")
plt.xticks(range(len(movies)), movies)
plt.yticks(range(len(users)), users)
plt.title("User-Movie ratings matrix (0 = unrated)")
plt.xlabel("movie")
plt.ylabel("user")
plt.show()

# --- Predict U0's missing rating for M4 using USER-based CF ---
# For this small demo we treat unrated (0) as literal 0 in the cosine
# computation -- a simplification that real systems avoid.
sim = cosine_similarity(ratings.astype(float))

print("Cosine similarity of U0 to each user:")
for u, s in zip(users, sim[0]):
    print(f"  {u}: {s:.2f}")

# Predict U0 -> M4: similarity-weighted average of OTHER users' M4 ratings
target_user, target_movie = 0, 4
others = [u for u in range(len(users)) if u != target_user]
weights = sim[target_user, others]
other_ratings = ratings[others, target_movie]
rated = other_ratings > 0   # keep only users who actually rated M4
pred = np.average(other_ratings[rated], weights=weights[rated])
print(f"\nPredicted rating for U0 -> M4: {pred:.2f}")
print(f"(based on {int(rated.sum())} users who rated M4, similarity-weighted)")

Ratings matrix (0 = unrated):
    M0  M1  M2  M3  M4
U0   5   4   0   1   0
U1   4   5   0   1   0
U2   1   0   5   4   4
U3   0   1   4   5   5
U4   5   4   1   0   0

Cosine similarity of U0 to each user:
  U0: 1.00
  U1: 0.98
  U2: 0.18
  U3: 0.17
  U4: 0.98

Predicted rating for U0 -> M4: 4.48
(based on 2 users who rated M4, similarity-weighted)

Reading the demo

• The heatmap shows two clear taste groups: U0/U1/U4 love M0/M1 and rate M3 low; U2/U3 love M2/M3/M4 and rate M0 low. The zeros are the gaps a recommender must fill.
• U0 never rated M4. Their strongest matches are U1 and U4 (cosine similarity ≈ 0.98 — almost identical taste), while U2 and U3 are only weakly similar (≈ 0.18).
• To predict U0 → M4 we take a similarity-weighted average of the ratings other users gave M4. But here's the catch: U0's true taste-mates (U1, U4) didn't rate M4 either, so the prediction must lean on the weakly-similar U2/U3, who both gave M4 high marks (4 and 5). The result is a high predicted rating (~4.5) — yet we should distrust it, because it rests on users who barely share U0's taste.

This is the sparsity problem, and it's why real recommenders prefer matrix factorization: instead of comparing raw rating rows, it learns a short latent "taste vector" for every user and item, so a prediction can borrow strength across all users and items at once — even when a specific taste-mate hasn't rated the target item.

Key Takeaways

• Unsupervised learning finds structure in unlabeled data — there's no "right answer" to score against, so judging quality is harder than in supervised learning.
• Density estimation models the data's pdf: a single Gaussian is cheap but assumes a bell shape; KDE follows any shape by summing little bumps, with a bandwidth controlling smoothness.
• K-means iterates assign → update centroids and minimizes inertia; it makes round clusters and needs you to pick k (the elbow method is a practical first guess).
• Gaussian Mixture Models (fit by EM) give soft probabilities and ellipse-shaped clusters — ideal for overlapping data.
• Hierarchical clustering builds a merge dendrogram you cut at the height that gives the cluster count you want.
• DBSCAN is density-based: no k needed, handles arbitrary shapes, and flags outliers — but its ε parameter is fiddly (HDBSCAN helps).
• PCA rotates data onto axes of maximum variance, letting you project high-dimensional data to 2D/3D and reconstruct it with small error; UMAP and autoencoders extend the idea to non-linear structure.
• Collaborative filtering predicts missing user–item ratings via cosine similarity (user-/item-based) or matrix factorization, powering recommender systems.

What's Next

Next we move beyond the standard supervised/unsupervised split in Chapter 10 — Other Forms of Learning, covering semi-supervised learning, active learning, transfer learning, one-shot learning, and more.

---

---

Chapter 10 — Other Forms of Learning

Most of this book has focused on the "big three" supervised tasks — classification, regression — plus a few unsupervised ones like clustering. But not every learning problem fits neatly into those boxes. Sometimes what we want to learn is a distance, a ranking, a recommendation, or even word meanings, and the methods for those problems have their own flavor. This chapter tours four of these "other" forms of learning so you recognize them when you meet them in the wild.

In this chapter you will learn:

• How metric learning turns "find a good distance" into a learning problem, and why it helps k-NN and clustering
• Why learning to rank cares about the order of items instead of absolute scores, plus how NDCG measures ranking quality
• How recommender systems work via content-based and collaborative filtering (factorization machines and denoising autoencoders)
• How self-supervised learning creates its own labels from raw text to learn word embeddings like word2vec

10.1 Metric Learning

A metric is just a rule that says how "far apart" two things are. The Euclidean distance you know from geometry is the most common metric for feature vectors, and cosine similarity is the most common for text. These choices are reasonable, but they are also a bit arbitrary — and the fact that one works better than another on a given dataset is a hint that no single fixed metric is perfect for every problem.

The key idea of metric learning is refreshingly simple: instead of guessing the distance formula, learn it from data. Once you have a good metric, you can plug it into any algorithm that needs a distance — k-NN, k-means, hierarchical clustering — and they all get better.

Everyday analogy: A friend who has never cooked might judge "how similar are these two recipes?" by weighing raw ingredient lists (plain Euclidean over counts). A trained chef weights which ingredients matter — a pinch of saffron counts far more than a pinch of water. The chef has, in effect, learned a metric.

Making Euclidean distance learnable

Recall the plain Euclidean distance between two vectors x and x':

$$d(\mathbf{x}, \mathbf{x'}) = \sqrt{(\mathbf{x}-\mathbf{x'})^\top (\mathbf{x}-\mathbf{x'})}.$$

We make this parametrizable by slipping a matrix A into the middle:

$$d_A(\mathbf{x}, \mathbf{x'}) = \sqrt{(\mathbf{x}-\mathbf{x'})^\top \, A \, (\mathbf{x}-\mathbf{x'})}.$$

• If A is the identity matrix, this collapses back to ordinary Euclidean distance.
• If A is diagonal, each feature gets its own weight — the bigger the diagonal entry, the more that feature "counts." (This is exactly the chef weighting ingredients.)
• If A is a full matrix, it can also rotate and rescale the axes, so distance is measured in a direction that separates your classes well.

For this to behave like a genuine distance, A must be positive semidefinite (the matrix version of "non-negative"): for any vector z, z^T A z >= 0. That guarantees the distance is never negative and respects the triangle inequality. The quantity sqrt(x^T A x) is also called the Mahalanobis distance governed by A.

A classic algorithm here is LMNN (Large-Margin Nearest Neighbor): it chooses A so that each example's nearest neighbors are same-class, while examples of other classes are pushed a large margin away. We won't implement LMNN from scratch, but we can capture its spirit with a built-in linear transform that learns a separating direction from the labels.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.neighbors import NearestNeighbors

plt.rcParams["figure.figsize"] = (11, 4)
plt.rcParams["axes.grid"] = True

# --- A small 2D dataset where the feature SCALES are bad for plain Euclidean -
# Feature 1 is informative but on a tiny scale; feature 2 is useless noise but
# stretched to a huge scale. Plain Euclidean distance is dominated by feature 2,
# so k-NN mixes up the classes. A *learned* metric should fix this.
X, y = make_classification(n_samples=300, n_features=2, n_informative=1,
                           n_redundant=0, n_classes=2, n_clusters_per_class=1,
                           class_sep=2.5, random_state=7)
X = X.copy()
X[:, 0] *= 0.08    # squeeze the informative axis
X[:, 1] *= 12.0    # stretch the noise axis

# --- "Before": plain Euclidean space -----------------------------------------
query = np.array([[0.0, 0.0]])           # a point we'll classify with k-NN
nn_orig = NearestNeighbors(n_neighbors=5).fit(X)
_, idx_orig = nn_orig.kneighbors(query)

# --- "After": learn a linear transform with LDA, measure distance there -------
# LDA finds the direction that best separates the classes -- a stand-in for
# learning the matrix A in d_A(x, x'). With 2 classes it produces ONE axis.
lda = LinearDiscriminantAnalysis()
lda.fit(X, y)
Xt = lda.transform(X)                    # data in the learned metric's axes
query_t = lda.transform(query)
nn_new = NearestNeighbors(n_neighbors=5).fit(Xt)
_, idx_new = nn_new.kneighbors(query_t)

# --- Leave-one-out 5-NN accuracy in each space (the headline number) ----------
def loo_5nn_acc(features):
    nn6 = NearestNeighbors(n_neighbors=6).fit(features)   # 6 = self + 5
    _, idx = nn6.kneighbors(features)
    preds = np.array([np.bincount(y[idx[i, 1:]]).argmax() for i in range(len(y))])
    return (preds == y).mean()

acc_orig = loo_5nn_acc(X)
acc_new = loo_5nn_acc(Xt)

# --- Plot: original space (left) vs learned 1D metric (right) -----------------
fig, ax = plt.subplots(1, 2, figsize=(11, 4))
colors = ["tab:blue", "tab:orange"]

# Left: original 2D, equal aspect so the stretched noise axis is visible as
# the dominant direction (which is exactly why Euclidean k-NN struggles).
for c in range(2):
    ax[0].scatter(X[:, 0][y == c], X[:, 1][y == c], color=colors[c], s=18,
                  alpha=0.6, label=f"class {c}")
ax[0].scatter(query[:, 0], query[:, 1], color="black", marker="*", s=260,
              label="query")
ax[0].scatter(X[idx_orig[0], 0], X[idx_orig[0], 1], facecolors="none",
              edgecolors="red", s=130, linewidths=2, label="5 NN (Euclidean)")
ax[0].set_aspect("equal")
ax[0].set_title("Before: original space (plain Euclidean)")
ax[0].set_xlabel("feature 1 (informative, squeezed)")
ax[0].set_ylabel("feature 2 (noise, stretched)")
ax[0].legend(fontsize=8)

# Right: the single learned LDA axis. Points are spread along it by class.
rng_jit = np.random.default_rng(0)
jitter = rng_jit.uniform(-0.18, 0.18, size=len(y))   # vertical jitter for visibility
for c in range(2):
    sel = y == c
    ax[1].scatter(Xt[sel, 0], jitter[sel], color=colors[c], s=18, alpha=0.6,
                  label=f"class {c}")
ax[1].scatter(query_t[0, 0], 0.0, color="black", marker="*", s=260, label="query")
ax[1].scatter(Xt[idx_new[0], 0], jitter[idx_new[0]], facecolors="none",
              edgecolors="red", s=130, linewidths=2, label="5 NN (learned metric)")
ax[1].set_title("After: LDA-learned metric (1 axis)")
ax[1].set_xlabel("learned axis  (LDA component 1)")
ax[1].set_yticks([])
ax[1].legend(fontsize=8)

plt.tight_layout()
plt.show()

print(f"Leave-one-out 5-NN accuracy in ORIGINAL space     : {acc_orig:.3f}")
print(f"Leave-one-out 5-NN accuracy in LEARNED-metric space: {acc_new:.3f}")
print("Query's 5 neighbors (original) classes:", y[idx_orig[0]])
print("Query's 5 neighbors (learned)   classes:", y[idx_new[0]])

Leave-one-out 5-NN accuracy in ORIGINAL space     : 0.727
Leave-one-out 5-NN accuracy in LEARNED-metric space: 0.993
Query's 5 neighbors (original) classes: [0 0 1 0 1]
Query's 5 neighbors (learned)   classes: [1 1 1 1 1]

Reading the result

On the left (equal aspect), the data is really a tall, thin cloud: feature 2 is stretched so wide that, in plain Euclidean terms, points that are close are mostly close along the noisy vertical direction — so the query's five nearest neighbors (red rings) are a mix of both classes, and leave-one-out 5-NN accuracy is low.

On the right, LDA has learned the single direction that best separates the classes and measured distance along it. The two classes now sit cleanly apart, the query's neighbors are all one class, and 5-NN accuracy jumps to near-perfect. The learned metric effectively down-weighted the noise feature and up-weighted the informative one — exactly what LMNN-style metric learning aims for.

This is the whole pitch of metric learning: the distance itself is a model, and it can be trained.

10.2 Learning to Rank

Think about a search engine. When you type a query, it returns a list of documents. We don't really care about the absolute score of each document — we care about their order: the most relevant one should be first, the next-best second, and so on. Learning to rank is the supervised problem of learning a function that produces a good ordering, not good individual numbers.

This is subtly different from classification/regression:

• Classification asks "what category is this document?"
• Regression asks "what number is this document?"
• Ranking asks "in what order should these documents appear?"

There are three classic ways to frame it:

Approach	What it optimizes	Flavour
Pointwise	Each document's score independently (treated as regression)	Ignores that documents compete for positions
Pairwise	For each pair, which document should rank higher	Better, but still treats pairs in isolation
Listwise	A metric over the whole ranked list directly	Best in practice (e.g. LambdaMART)

The cleverness of LambdaMART (a gradient-boosted-tree ranker) is that it tweaks the gradient using the ranking metric itself, so the model optimizes the thing we actually care about — something ordinary supervised models rarely do. Usually we optimize a cost (like cross-entropy) and only afterwards check a metric; LambdaMART blurs that line.

Measuring a ranking: NDCG

To know whether a ranking is good, we need a metric. A popular one is NDCG (Normalized Discounted Cumulative Gain). The idea is intuitive, in four steps:

1. Relevance: each document has a relevance grade (say 0 = useless, 3 = perfect). Higher is better.
2. Cumulative Gain (CG): sum the relevances of the documents you returned.
3. Discount: a relevant document at position 10 helps the user far less than the same document at position 1, so we discount gains lower down the list — typically dividing by log2(rank + 1). This rewards putting good stuff on top.
4. Normalize: divide by the ideal DCG (the DCG of the perfect ordering) so the score lands in [0, 1], where 1.0 means "as good as it gets."

Looking at only the first k positions gives NDCG@k — we judge just the top of the list, because users rarely scroll past it.

import numpy as np

def dcg_at_k(relevances, k):
    """Discounted Cumulative Gain for the top k positions.
    relevances = list of relevance grades IN THE ORDER SHOWN to the user."""
    r = np.asarray(relevances, dtype=float)[:k]
    if r.size == 0:
        return 0.0
    # discount: position 1 -> /log2(2)=1, position 2 -> /log2(3), position 3 -> /log2(4)...
    discounts = 1.0 / np.log2(np.arange(2, r.size + 2))
    return float(np.sum(r * discounts))

def ndcg_at_k(relevances, k):
    """NDCG@k = DCG@k / IDCG@k, where IDCG uses the ideally sorted list."""
    dcg = dcg_at_k(relevances, k)
    ideal = sorted(relevances, reverse=True)   # best possible ordering
    idcg = dcg_at_k(ideal, k)
    return dcg / idcg if idcg > 0 else 0.0

# --- A tiny search result: 5 documents with relevance grades 0..3 ------------
# The TRUE best order would be [3, 3, 2, 1, 0]. Our model returns them
# in THIS order instead:
ranking = [1, 3, 0, 2, 3]
k = 3                         # judge only the top 3

print("Model ranking (top 5) :", ranking)
print(f"NDCG@{k} for model ranking : {ndcg_at_k(ranking, k):.3f}")

# --- Swap the first two positions to put a '3' on top ------------------------
better = [3, 1, 0, 2, 3]
print("Swapped ranking       :", better)
print(f"NDCG@{k} for swapped     : {ndcg_at_k(better, k):.3f}")

# --- The ideal ordering (sanity check: should be 1.0) ------------------------
ideal = [3, 3, 2, 1, 0]
print("Ideal ordering        :", ideal)
print(f"NDCG@{k} for ideal       : {ndcg_at_k(ideal, k):.3f}")

Model ranking (top 5) : [1, 3, 0, 2, 3]
NDCG@3 for model ranking : 0.491
Swapped ranking       : [3, 1, 0, 2, 3]
NDCG@3 for swapped     : 0.616
Ideal ordering        : [3, 3, 2, 1, 0]
NDCG@3 for ideal       : 1.000

Reading the result

• The model's ordering [1, 3, 0, 2, 3] puts a low-relevance 1 on top, so its NDCG@3 is well below 1.
• Swapping the first two slots to [3, 1, ...] lifts a high-relevance document to position 1, and NDCG@3 jumps up — that is the discount at work: gains at the top count much more than gains lower down.
• The ideal ordering [3, 3, 2, 1, 0] scores exactly 1.0.

A ranker's job is to push NDCG@k toward 1.0. Notice NDCG only cares about the order and the position — it completely ignores the raw scores the model emits, which is exactly the "ranking != regression" point from above.

10.3 Learning to Recommend

A recommender system suggests new content (a movie on Netflix, a book on Amazon, a song on Spotify) that a user is likely to enjoy, based on their consumption history. There are two traditional pillars:

• Content-based filtering learns what a user likes from the description of the content they consume. If you keep reading science-and-tech articles, it suggests more science-and-tech articles. You can think of it as building a small "will this user click?" classifier per user, using content features (words, topic, price, recency) as inputs.

• Collaborative filtering recommends based on what similar users consume or rate. If you and another user both loved the same ten movies, movies that user loved (but you haven't seen) are probably good picks for you. Crucially, it ignores the content itself and leans on the pattern of overlapping tastes.

Everyday analogy: Content-based is "you liked Italian food, here's more Italian food." Collaborative is "your foodie twin loved this place, so you probably will too."

Each has a weakness: content-based can trap users in a filter bubble (endless more-of-the-same, possibly items they already know about), while collaborative filtering struggles with cold starts and extremely sparse preference matrices (each user rates only a tiny fraction of items). Real systems are usually hybrid — they blend both signals.

The data: a giant, mostly-empty matrix

Collaborative filtering stores preferences in a user x item matrix: rows are users, columns are items, and each cell is a rating (or a 1 for "consumed"). In practice this matrix is huge and almost entirely empty — millions of users, hundreds of thousands of items, but each user touches only a handful. That sparsity is the central headache, and it is exactly what the next two algorithms (factorization machines and denoising autoencoders) are designed to handle. Let's visualize it first.

import numpy as np
import matplotlib.pyplot as plt

# --- Simulate a sparse user x item rating matrix (like a real recommender) ----
rng = np.random.default_rng(42)
n_users, n_items = 40, 60
density = 0.08                                  # each user rates ~8% of items
mask = rng.random((n_users, n_items)) < density
ratings = rng.integers(1, 6, size=(n_users, n_items)).astype(float)
R_sparse = np.where(mask, ratings, np.nan)      # missing -> NaN (drawn white)

plt.figure(figsize=(7, 5))
plt.imshow(R_sparse, aspect="auto", cmap="viridis", interpolation="nearest")
plt.colorbar(label="rating (1-5)")
plt.title("Sparse user x item matrix\n"
          f"({int(mask.sum())} known ratings out of {n_users*n_items} "
          f"= {100*mask.mean():.0f}% filled)")
plt.xlabel("item (movie)")
plt.ylabel("user")
plt.show()

10.3.1 Factorization Machines

Factorization machines (FM) were designed specifically for these sparse, high-dimensional datasets. A plain linear model is

$$f(\mathbf{x}) = b + \sum_i w_i\, x_i.$$

The problem: with one-hot user/item features, most $x_i$ are zero, so the model rarely sees most pairwise interactions $x_i x_j$, and it can't learn that "user A together with movie B" matters. Adding a separate weight $w_{ij}$ for every pair would explode the parameter count (about $D(D-1)$ new parameters).

FM's trick is to factorize each interaction weight as a dot product of two small learned vectors:

$$f(\mathbf{x}) = b + \sum_i w_i\, x_i + \sum_{i=1}^{D}\sum_{j=i+1}^{D} (\mathbf{v}_i \cdot \mathbf{v}_j)\, x_i x_j.$$

Each feature $i$ gets a $k$-dimensional factor vector $\mathbf{v}_i$ with $k \ll D$. Now the interaction weight between features $i$ and $j$ is $\mathbf{v}_i \cdot \mathbf{v}_j$, and the total extra parameters are only $Dk \ll D(D-1)$. Because factor vectors are shared across pairs, even a rarely-seen combination can borrow strength from related features — which is why FM generalizes far better than full pairwise weights on sparse data.

Everyday analogy: Instead of memorizing a separate opinion for every possible (user, movie) pair, FM gives each user and each movie a short "taste profile" vector, and estimates a pairing by how well the two profiles align.

import numpy as np

# --- A tiny user x movie rating matrix. NaN = "not rated" (the sparsity!) -----
R = np.array([
    [5, 4, np.nan, 1, np.nan],
    [np.nan, 5, 4, np.nan, 2],
    [1, np.nan, np.nan, 5, 4],
    [4, 3, np.nan, np.nan, 5],
])
print("Raw rating matrix (NaN = missing):\n", R)

# --- Fill the holes with the global mean, then TRUNCATED SVD ------------------
# This is the classic "matrix factorization" flavour of collaborative filtering:
# approximate the matrix as a low-rank product U @ V. The top factors capture
# the main "taste x item" structure and let us predict the missing cells.
# This is the same *factorization* spirit that FM generalizes to all features.
mask = ~np.isnan(R)
R_filled = np.where(mask, R, np.nanmean(R))     # temporary fill for the holes

k = 2                                           # keep only 2 factors (like FM's k)
U, s, Vt = np.linalg.svd(R_filled, full_matrices=False)
R_hat = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]   # low-rank reconstruction

print("\nLow-rank (k=2) prediction for EVERY user-item pair:\n",
      np.round(R_hat, 2))

# --- Highlight predicted ratings only for the originally-missing cells -------
pred_missing = R_hat.copy()
pred_missing[mask] = np.nan                     # blank out cells we already knew
print("\nPredicted ratings ONLY for missing cells (recommendation candidates):\n",
      np.round(pred_missing, 2))

Raw rating matrix (NaN = missing):
 [[ 5.  4. nan  1. nan]
 [nan  5.  4. nan  2.]
 [ 1. nan nan  5.  4.]
 [ 4.  3. nan nan  5.]]

Low-rank (k=2) prediction for EVERY user-item pair:
 [[5.21 3.95 3.54 1.25 3.25]
 [3.58 3.96 3.74 3.23 3.69]
 [1.2  3.48 3.53 5.22 3.78]
 [3.63 4.13 3.92 3.5  3.88]]

Predicted ratings ONLY for missing cells (recommendation candidates):
 [[ nan  nan 3.54  nan 3.25]
 [3.58  nan  nan 3.23  nan]
 [ nan 3.48 3.53  nan  nan]
 [ nan  nan 3.92 3.5   nan]]

Reading the result

The low-rank reconstruction R_hat predicts a rating for every user-movie pair, including the ones that were missing (NaN). The last matrix highlights just those predicted holes — these are exactly the cells a recommender would use to suggest "you haven't seen this, but you'd probably rate it four stars."

This is the factorization spirit that factorization machines generalize: represent users and items by small latent vectors, and estimate any pairing by how those vectors combine. FM extends the same idea to arbitrary sparse features (not just user/item IDs) by factorizing all pairwise interactions $x_i x_j$ through shared factor vectors.

10.3.2 Denoising Autoencoders (for recommendation)

You met denoising autoencoders (DAE) in Chapter 7: a neural net that rebuilds its input from a compressed bottleneck, after the input has been deliberately corrupted with noise. For recommendation, the trick is to reinterpret that corruption:

• The user's full set of liked items is the "clean" signal.
• Unseen-but-would-likely-enjoy items are treated as if a corruption process removed them.
• Train the autoencoder to reconstruct the clean, complete preference vector from the corrupted (partly-emptied) one.

At prediction time you feed in the user's known ratings, let the DAE reconstruct the full vector, and recommend the items whose reconstructed scores are highest — i.e. the ones the model "fills back in." Because the network must compress through a bottleneck, it learns latent tastes rather than memorizing ratings, which is what makes it work on sparse data.

A related collaborative-filtering model is a small two-input neural net: one one-hot input for the user, one for the item, and a single output predicting the rating $r$ (a sigmoid for $r \in [0,1]$, or a ReLU for $r \in [1,5]$). We leave these as concepts — the takeaway is that both factorization and autoencoders attack the same sparse-matrix problem from different angles.

10.4 Self-Supervised Learning: Word Embeddings

Where do word embeddings — the feature vectors that represent words, where similar words get similar vectors — actually come from? They are learned from data, and the clever part is that the data is unlabeled text. The labels are created automatically from the text itself, which is why this is called self-supervised learning.

The flagship algorithm is word2vec, and its most popular variant is skip-gram. The intuition is pure common sense:

You can often guess a missing word from its neighbors. In "I almost finished reading the ___ on machine learning," you'd guess book, article, or paper. Words that appear in similar contexts tend to have similar meanings.

Skip-gram turns this into a training task: take a center word and train a small neural net to predict the surrounding context words within a window (e.g. window size 5 = two words on each side). Each (center word -> context word) pair becomes a self-generated labeled example — no human annotation needed. After training, the weights of the hidden embedding layer are the word vectors: feed in a word's one-hot encoding, read out its embedding.

This is the essence of self-supervised learning: the structure of the data itself supplies the labels. (Large word2vec models use tricks like hierarchical softmax and negative sampling to stay tractable across huge vocabularies — good topics for further reading.)

import numpy as np

# --- A tiny corpus and a small vocabulary ------------------------------------
sentence = "the cat sat on the mat the dog sat on the log".split()
vocab = sorted(set(sentence))
word_to_idx = {w: i for i, w in enumerate(vocab)}
print("Vocabulary:", vocab)

# --- Extract skip-gram (center -> context) pairs with window size 2 ----------
# For each center word, pair it with the 2 words on its left and its right.
# These (center, context) pairs are the SELF-SUPERVISED training examples.
window = 2
pairs = []
for i, center in enumerate(sentence):
    for j in range(max(0, i - window), min(len(sentence), i + window + 1)):
        if j != i:
            pairs.append((word_to_idx[center], word_to_idx[sentence[j]],
                          center, sentence[j]))

print(f"\nGenerated {len(pairs)} self-supervised (center -> context) pairs")
print("(center_idx, context_idx, center_word, context_word):")
for p in pairs:
    print(p)

Vocabulary: ['cat', 'dog', 'log', 'mat', 'on', 'sat', 'the']

Generated 42 self-supervised (center -> context) pairs
(center_idx, context_idx, center_word, context_word):
(6, 0, 'the', 'cat')
(6, 5, 'the', 'sat')
(0, 6, 'cat', 'the')
(0, 5, 'cat', 'sat')
(0, 4, 'cat', 'on')
(5, 6, 'sat', 'the')
(5, 0, 'sat', 'cat')
(5, 4, 'sat', 'on')
(5, 6, 'sat', 'the')
(4, 0, 'on', 'cat')
(4, 5, 'on', 'sat')
(4, 6, 'on', 'the')
(4, 3, 'on', 'mat')
(6, 5, 'the', 'sat')
(6, 4, 'the', 'on')
(6, 3, 'the', 'mat')
(6, 6, 'the', 'the')
(3, 4, 'mat', 'on')
(3, 6, 'mat', 'the')
(3, 6, 'mat', 'the')
(3, 1, 'mat', 'dog')
(6, 6, 'the', 'the')
(6, 3, 'the', 'mat')
(6, 1, 'the', 'dog')
(6, 5, 'the', 'sat')
(1, 3, 'dog', 'mat')
(1, 6, 'dog', 'the')
(1, 5, 'dog', 'sat')
(1, 4, 'dog', 'on')
(5, 6, 'sat', 'the')
(5, 1, 'sat', 'dog')
(5, 4, 'sat', 'on')
(5, 6, 'sat', 'the')
(4, 1, 'on', 'dog')
(4, 5, 'on', 'sat')
(4, 6, 'on', 'the')
(4, 2, 'on', 'log')
(6, 5, 'the', 'sat')
(6, 4, 'the', 'on')
(6, 2, 'the', 'log')
(2, 4, 'log', 'on')
(2, 6, 'log', 'the')

Reading the result

From a single short sentence, window size 2 already produced a couple dozen (center -> context) training pairs — for free, with no labels provided by any human. Scale this up to billions of words of web text and you get the hundreds of millions of skip-grams that train real word2vec models. Words like "cat" and "dog" end up with similar embeddings because they share similar contexts ("sat", "the", "on"), which is exactly how the model discovers meaning from raw text.

That is self-supervised learning in one sentence: turn the data's own structure into supervised examples.

Key Takeaways

• A metric is a learnable object: by parametrizing Euclidean distance as $d_A(\mathbf{x},\mathbf{x'})=\sqrt{(\mathbf{x}-\mathbf{x'})^\top A (\mathbf{x}-\mathbf{x'})}$ with A positive semidefinite, we can train the distance (e.g. LMNN) and boost k-NN and clustering.
• Learning to rank optimizes the order of items, not absolute scores. The three framings are pointwise, pairwise, and listwise (LambdaMART is listwise and optimizes a ranking metric directly).
• NDCG@k measures ranking quality by discounting gains lower in the list and normalizing by the ideal ordering; 1.0 means a perfect ranking.
• Recommender systems use content-based filtering (item descriptions) and collaborative filtering (taste overlap); real systems are hybrid.
• The core challenge in recommendation is a huge, sparse user x item matrix.
• Factorization machines handle sparse features by factorizing pairwise interaction weights as $\mathbf{v}_i \cdot \mathbf{v}_j$, adding only $Dk$ parameters instead of $D(D-1)$.
• Denoising autoencoders recommend by reconstructing a user's (corrupted) preference vector and filling in the "missing" liked items.
• Self-supervised learning (e.g. word2vec skip-gram) manufactures its own labels from unlabeled data — a center word predicts its context — to learn word embeddings.

What's Next

In Chapter 11 — Conclusion, we'll step back and reflect on the whole journey: what you now know, what we deliberately left out, and where to go next as a practicing machine learning practitioner.

---

---

Chapter 11 — Conclusion

You made it! If you've followed along from Chapter 1 to here, you have just walked the full arc of modern machine learning — and that is genuinely something to be proud of. This short closing chapter is not an exam; it's a friendly send-off. We'll look back at the road we traveled, peek at a few important topics the book only waved at, and point you toward where to go next.

In this chapter you will learn:

• How the pieces you studied fit together into one big picture
• Seven important topics that were beyond our scope — and what each one is about in one breath
• Concrete suggestions for what to study, build, and read next
• A compact glossary recapping the key terms from the whole book

The Arc of the Book

Let's retrace the journey, because seeing the whole shape helps everything stick together:

• Foundations. We started with the big idea — machine learning is finding a formula that maps inputs to outputs and generalizes to new inputs — and learned the vocabulary and light math (vectors, random variables, probability, Bayes' rule) that the rest of the book uses.
• Fundamental algorithms. The workhorses: linear and logistic regression, decision trees, k-nearest neighbors, and the SVM — each drawing its own kind of decision boundary.
• Anatomy of a learner. We looked inside every algorithm and found the same four parts: a model, a loss function, an optimizer (like gradient descent), and a regularizer to keep things from overfitting.
• Basic practice. The day-to-day craft: feature engineering, one-hot encoding, scaling, cross-validation, hyperparameter tuning, and the bias–variance tradeoff.
• Neural networks and deep learning. From one neuron to stacked layers; CNNs for images, RNNs for sequences, and backpropagation training them all.
• Problems and solutions. Real-world messiness — imbalanced classes, text, images, ranking, sequence labeling — and practical fixes for each.
• Advanced and unsupervised learning. Clustering, PCA, dimensionality reduction, and finding structure when you have no labels.
• Other forms of learning. Semi-supervised, one-shot, and the broader zoo of learning setups beyond plain supervised learning.

You started with "what is a feature vector?" and ended up able to reason about CNNs, ensembles, and regularization. That is the arc — and it's a lot of ground.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (7, 4)
plt.rcParams["axes.grid"] = True

# Seven topics the book names but didn't fully teach.
# Think of this as a visual "menu" of directions to explore next.
topics = [
    "11.1  Topic Modeling (LDA)",
    "11.2  Gaussian Processes",
    "11.3  Generalized Linear Models",
    "11.4  Probabilistic Graphical Models",
    "11.5  Markov Chain Monte Carlo",
    "11.6  Genetic Algorithms",
    "11.7  Reinforcement Learning",
]

# Uniform bars -- the point is the list of names, not the lengths.
vals = np.ones(len(topics))
colors = plt.cm.viridis(np.linspace(0.15, 0.85, len(topics)))

fig, ax = plt.subplots()
ax.barh(range(len(topics)), vals, color=colors)
ax.set_yticks(range(len(topics)))
ax.set_yticklabels(topics)
ax.set_xlabel("ready for you to explore")
ax.set_title("Seven topics waiting for you")
ax.set_xlim(0, 1.25)
ax.invert_yaxis()  # first topic on top
# tidy up the frame
for spine in ["top", "right"]:
    ax.spines[spine].set_visible(False)
plt.tight_layout()
plt.show()

Reading the chart

Each bar is one of the seven topics the book names but didn't fully teach. They are not harder or easier than what you've already done — they are simply neighbors of the ideas you now know. Think of this chart as a menu: pick whichever catches your curiosity first.

11.1 Topic Modeling

Imagine you have thousands of customer reviews, news articles, or research papers — and no labels at all. Topic modeling is the unsupervised task of discovering what themes run through that pile of text.

The best-known algorithm is Latent Dirichlet Allocation (LDA). The idea is surprisingly intuitive:

1. You decide how many topics the collection has (say, 5).
2. LDA looks at which words tend to appear together and assigns each word to one or more topics.
3. To describe a document, you just count how many words of each topic it contains — so a document heavy in "pitch", "strike", and "inning" words is mostly the baseball topic.

It's clustering, but for words and documents instead of feature points. If you work with text, LDA (or its modern cousins) is one of the first tools you'll reach for.

11.2 Gaussian Processes

A Gaussian Process (GP) is a supervised learning method that competes with kernel regression, with one big bonus: it doesn't just predict a value — it also tells you how confident it is at each point, by giving you a confidence interval around the prediction.

Think of it as regression that draws not just a line but a band of uncertainty. Where it has seen lots of training data, the band is narrow (confident); where data is sparse, the band is wide (unsure). GPs are elegant and powerful, but the math behind them is genuinely heavy (it rests on multivariate Gaussians and kernel functions), which is why the book waved at them rather than teaching them. If you love regression and care about uncertainty, learning GPs is, as the book says, "time well spent."

11.3 Generalized Linear Models

A Generalized Linear Model (GLM) is a broad family that extends ordinary linear regression to many kinds of output. Plain linear regression assumes the target is a noisy straight-line function of the features. But what if the target is a yes/no, or a count, or a strictly positive quantity?

GLMs let you swap in a different link function to match the shape of your target. You've already met one: logistic regression is a GLM — it uses a logit link so the output lands between 0 and 1, perfect for probabilities. Other GLMs handle counts (Poisson regression) or skewed positive values. If you want simple, explainable models for regression-like tasks, the GLM family is well worth a deeper read.

11.4 Probabilistic Graphical Models

A Probabilistic Graphical Model (PGM) represents how random variables depend on each other as a graph — nodes are variables, edges are dependencies. For example, the node "sidewalk wetness" depends on the node "weather condition."

This is powerful because it lets you see and reason about how features influence each other, and — if the edges are directed — even make claims about causality, not just correlation. Two famous examples:

• Conditional Random Fields (CRF) model sequences (like labeling each word in a sentence) and were big in text and image processing before neural networks took over.
• Hidden Markov Models (HMM) model time series and were the go-to for speech recognition — again, eventually surpassed by neural networks.

PGMs also go by the names Bayesian networks or belief networks. They're beautiful but demanding: building one by hand needs strong probability skills and deep domain knowledge, which is why they're less common in everyday practice than simpler models.

11.5 Markov Chain Monte Carlo

Markov Chain Monte Carlo (MCMC) is a family of algorithms for one specific but important job: sampling from a probability distribution that is too complicated to sample from directly.

Sampling from a plain normal or uniform distribution is easy — those are well understood. But once you've learned a gnarly dependency graph (a PGM) from data, the distribution can have almost any shape, and drawing samples from it becomes hard. MCMC solves this with a clever trick: it builds a Markov chain (a random walk where each step depends only on where you are now) that, after enough steps, visits locations in proportion to the distribution you want. You then just record where the walk goes.

If you go deep into Bayesian methods or graphical models, MCMC is the engine under the hood.

11.6 Genetic Algorithms

Genetic Algorithms (GA) are an optimization technique inspired by evolution, used when your objective function is non-differentiable — that is, when you can't compute a gradient, so plain gradient descent is off the table.

The recipe mirrors biology:

1. Start with a random generation of candidate solutions (each is a set of parameter values — a point in parameter space).
2. Score every candidate against your objective.
3. Build the next generation using three moves: selection (keep the best), crossover (combine good candidates, like mixing parents' genes), and mutation (randomly tweak a candidate to explore new territory).
4. Repeat for many generations — the population drifts toward better solutions.

GAs can optimize almost any measurable objective (even hyperparameter search), but they are usually much slower than gradient-based methods. Reach for them when gradients don't exist; stick with gradient descent when they do.

11.7 Reinforcement Learning

Reinforcement Learning (RL) tackles a different flavor of problem entirely: sequential decision making. An agent acts in an environment it doesn't fully understand. Each action earns a reward and moves the agent to a new state. The goal is to maximize long-term reward — not just the next snack, but the whole future.

Classic algorithms like Q-learning (and its neural-network-powered cousins) are behind systems that learn to play video games, navigate robots, manage power grids, optimize supply chains, and even trade financial markets. RL is a big, rich field of its own — if the idea of an agent learning by trial and error excites you, this is your next rabbit hole.

Where to Go from Here

You now have a working map of machine learning. Here are concrete next steps, roughly ordered from easy to ambitious:

• Build projects. Pick a small dataset (Kaggle, UCI, or your own), and take it end to end: clean, explore, train a couple of models, evaluate with cross-validation, and write up what you found. One real project teaches more than ten tutorials.
• Enter Kaggle competitions. Start with the "Getting Started" Playground competitions. You'll see how others engineer features and stack models — the practical craft from the practice chapter in action.
• Go deeper on the math. Brush up on linear algebra, probability, and multivariate calculus. A stronger math foundation makes everything from SVMs to neural nets feel less like magic.
• Go deeper on deep learning. Work through a dedicated deep learning course or book. Build a CNN on a real image dataset and an RNN/transformer on a real text dataset — by hand, not just by copying a tutorial.
• Explore reinforcement learning. Try a simple Q-learning agent on a toy environment. Watching an agent learn to balance a pole is a great first taste.
• Read more detailed books. This companion is a map; now pick one or two thicker books on the areas that grabbed you most — pattern recognition, deep learning, or probabilistic ML — and go deep.
• Stay current. Follow the field: read papers, blogs, and course notes. Machine learning moves fast, and the wiki-style updates the original book mentions capture exactly this idea.

Quick Glossary Recap

A one-line refresher of the most important terms from the whole book. If a term feels fuzzy, that's your signal to flip back to its chapter.

Term	Plain definition
Feature vector	The list of numbers describing one example (an input x).
Label	The thing we predict for an example (the output y).
Supervised learning	Learning from labeled examples {(xᵢ, yᵢ)}.
Unsupervised learning	Finding structure in unlabeled examples {xᵢ}.
Decision boundary	The surface that separates predicted classes.
Margin	The width of the "no-man's land" around a boundary; wider usually generalizes better.
Gradient descent	Updating parameters step-by-step in the direction that reduces the loss.
Loss function	A score for how wrong the model's predictions are.
Overfitting	Memorizing training data so well that the model fails on new data.
Regularization	A penalty added to the loss to keep the model simple and reduce overfitting.
Cross-validation	Splitting data into folds to estimate how a model will do on unseen data.
Hyperparameter	A setting you choose before training (vs. parameters learned during training).
Ensemble	Combining several models for a stronger overall prediction.
Bagging	Training models on random subsets of data and averaging (e.g., random forest).
Boosting	Training models in sequence, each fixing the previous one's errors.
PCA	Rotating data onto axes that capture the most variance, for dimensionality reduction.
Clustering	Grouping unlabeled examples by similarity (e.g., k-means).
SVM	A classifier that finds the widest-margin boundary between classes.
Logistic regression	A linear classifier that outputs probabilities (a GLM with a logit link).
Neural network	Stacked layers of simple neurons trained by backpropagation.
CNN	A neural net with convolutional layers, ideal for images.
RNN	A neural net that processes sequences by carrying state across steps.

Key Takeaways

• You've walked the full arc of modern ML: foundations → fundamental algorithms → the anatomy of a learner → practice → deep learning → real-world problems → unsupervised learning → other forms of learning.
• Seven important topics sat just outside our scope: topic modeling (LDA), Gaussian processes, generalized linear models, probabilistic graphical models, MCMC, genetic algorithms, and reinforcement learning — each is a natural next direction.
• GPs give you confidence intervals, not just predictions; GLMs extend linear regression to many output shapes; PGMs model dependencies and even causality.
• MCMC is the engine for sampling from complex distributions; genetic algorithms optimize when gradients don't exist; RL is about agents learning by trial and error for long-term reward.
• The best next step is to build something real — a project teaches more than any chapter.
• Keep a glossary handy: when a term feels fuzzy, go back and revisit it. Repetition is how it sticks.

What's Next

You've reached the end of the companion. Revisit any chapter anytime, and keep experimenting!