Chapter 7 — Problems and Solutions

Most of this book so far has focused on the tools — linear models, SVMs, decision trees, neural nets. This chapter flips the view and looks at common problems you'll meet in practice, and the solutions (families of models) that fit each one. Think of it as a field guide: "I have data shaped like this and a goal like that — what do I reach for?"

In this chapter you will learn:

• How to handle more than two classes at once (multiclass classification)
• How to spot anomalies when you only have "normal" examples (one-class classification)
• How to predict several labels per example instead of just one (multi-label classification)
• Why combining many weak models beats one strong model (ensemble learning: bagging, boosting, stacking)
• How to learn from mostly unlabeled data (semi-supervised learning)
• The ideas behind one-shot and zero-shot learning, plus quick notes on a few other problem types

7.1 Kernel Regression

Sometimes a straight line is a bad fit for your data. You could bend it with polynomial features, but picking the right polynomial gets hard once your input has more than a couple of features.

Kernel regression takes a different, non-parametric route: there are no weights to learn. Instead, to predict the value at a new point x, you look at all training points and take a weighted average of their labels. Points close to x get a lot of weight; far-away points get almost none. The weight comes from a kernel function (a bell-shaped Gaussian is the usual choice), and a single knob — the kernel's width b — controls how local the average is. A tiny b chases every wiggly point (overfit); a huge b flattens everything into a near-line (underfit). You tune b on a validation set, just like other hyperparameters.

Everyday analogy: estimating tomorrow's temperature by averaging recent days, but weighting the most recent days much more heavily than days from a month ago.

7.2 Multiclass Classification

In multiclass classification the label is one of C classes (not just two): y ∈ {1, 2, …, C}. For example, sorting handwritten digits into 0–9.

Some algorithms handle many classes natively — decision trees, k-NN, and logistic regression all extend naturally. But many workhorses (like the basic SVM) are fundamentally binary. When you have a binary-only algorithm but a multiclass problem, the standard fix is one-vs-rest (OvR): build C separate binary classifiers, each one asking "is this class c or not?". To predict, run all C classifiers and pick the class whose model is most confident.

When an algorithm can be multiclass natively, the cleanest extension of logistic regression is to swap the sigmoid for the softmax function. Softmax turns a vector of raw scores into a probability distribution: it exponentiates each score and normalizes so the numbers are positive and sum to 1. The predicted class is simply the one with the largest probability.

Everyday analogy: one-vs-rest is like asking C yes/no questions ("is it a cat?", "is it a dog?", …) and going with the loudest "yes". Softmax is like one ranked ballot that spreads 100% of your confidence across all the options at once.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# --- Softmax from scratch, just to see what it does ---
def softmax(scores):
    # Subtract the max for numerical stability (keeps exp() from blowing up)
    s = scores - np.max(scores)
    exp_s = np.exp(s)
    return exp_s / np.sum(exp_s)

# Raw scores a model might produce for 3 classes (cat / dog / bird)
scores = np.array([2.0, 1.0, 0.1])
probs  = softmax(scores)

print("Raw scores      :", scores)
print("Softmax probs   :", np.round(probs, 4))
print("Sum of probs    :", round(probs.sum(), 6))   # always 1.0
print("Predicted class :", np.argmax(probs))

Raw scores      : [2.  1.  0.1]
Softmax probs   : [0.659  0.2424 0.0986]
Sum of probs    : 1.0
Predicted class : 0

Reading the output

The three raw scores {2.0, 1.0, 0.1} became probabilities {0.659, 0.242, 0.099}. Two things always hold:

• every probability is positive (because exp is always positive), and
• they sum to exactly 1 — softmax's whole job is to produce a valid probability distribution.

The biggest score (2.0) became the biggest probability (≈0.66), so we'd predict class 0 ("cat"). Softmax is soft: it doesn't just pick a winner, it tells you how confident it is about every option.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# 3 classes, 2 informative features so we can plot in 2D
X, y = make_classification(n_samples=300, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           n_classes=3, random_state=42)

# Modern LogisticRegression uses softmax (multinomial) by default for >2 classes
clf = LogisticRegression(max_iter=500, random_state=42)
clf.fit(X, y)

# Build a fine grid and ask the model to label every point -> decision regions
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))
grid = np.c_[xx.ravel(), yy.ravel()]
Z = clf.predict(grid).reshape(xx.shape)

plt.contourf(xx, yy, Z, alpha=0.3, cmap="tab10")
plt.scatter(X[:, 0], X[:, 1], c=y, cmap="tab10", edgecolors="k", s=30)
plt.title("Multiclass logistic regression (softmax) — 3 classes")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.show()

# Predicted probabilities for one example should also sum to 1
proba = clf.predict_proba(X[:1])
print("Probabilities for first example:", np.round(proba[0], 4))
print("Sum:", round(proba[0].sum(), 6))

Probabilities for first example: [1.20e-03 9.98e-01 7.00e-04]
Sum: 1.0

Reading the plot

Each colored region is the area the model "owns" for one class; the colored dots are the actual training examples. The boundaries between regions are linear because logistic regression draws straight-line borders.

Notice the predicted probabilities for the first example again sum to 1 — the same softmax behavior we built by hand, now happening inside the model.

One-vs-rest in disguise: if you'd used a plain SVM instead, scikit-learn would have quietly trained three binary SVMs (one per class) under the hood and let them vote — that's the OvR strategy from the intro, handled automatically via decision_function_shape='ovr'.

7.3 One-Class Classification (Anomaly & Novelty Detection)

Sometimes you have piles of "normal" examples and almost no examples of the "bad" ones. Think of network traffic: you have endless normal traffic, but attacks are rare and constantly changing. You can't train a normal classifier to separate "attack" from "normal" if you barely have any attack examples.

One-class classification flips the problem: instead of learning the boundary between two classes, you learn a tight description of what "normal" looks like, and anything that falls outside it gets flagged as an outlier. Common approaches fit a probability density to the normal data (a one-class Gaussian), or draw a boundary around it (one-class SVM), or isolate points by randomly splitting the feature space (Isolation Forest). You then set a threshold: inside the description = normal, outside = anomaly.

Everyday analogy: you've never seen a counterfeit $100 bill, but you've handled thousands of real ones — so a bill that feels off (wrong paper, odd color) stands out because it doesn't match your mental model of "normal bill".

from sklearn.ensemble import IsolationForest

# A tight cluster of "normal" points, plus a few scattered outliers
rng = np.random.RandomState(42)
X_normal   = rng.randn(200, 2) * 0.8                       # normal data near origin
X_outliers = rng.uniform(low=-6, high=6, size=(15, 2))     # 15 random outliers
X_all = np.vstack([X_normal, X_outliers])

# IsolationForest: "isolate" points by random splits; outliers are easiest to isolate
iso = IsolationForest(contamination=15 / 215, random_state=42, n_jobs=1)
pred = iso.fit_predict(X_all)   # +1 = normal, -1 = outlier

normal   = X_all[pred == 1]
outliers = X_all[pred == -1]

plt.scatter(normal[:, 0], normal[:, 1], c="tab:blue", s=20, label="normal")
plt.scatter(outliers[:, 0], outliers[:, 1], c="tab:red", s=60,
            marker="x", label="flagged outliers")
plt.title("One-class detection with Isolation Forest")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

print(f"Flagged {len(outliers)} points as outliers (we planted 15).")

Flagged 15 points as outliers (we planted 15).

Reading the plot

The blue cluster is the model's idea of "normal". The red x's are the points the forest decided were too easy to isolate — they sit far from the dense region, exactly where we secretly placed the outliers. contamination is the key hyperparameter: it's your guess at what fraction of the data is anomalous, and you tune it on a validation set (or set it from domain knowledge).

7.4 Multi-Label Classification

Don't confuse this with multiclass. In multiclass each example has one label out of many. In multi-label each example can have several labels at once. A photo might be tagged people, concert, and nature simultaneously.

The common trick is to turn one multi-label problem into L independent binary problems — one per label ("does this image contain people? yes/no", "…concert? yes/no", …). Any classifier that gives a score per class (trees, logistic regression, neural nets, k-NN) can be applied this way: predict every label, then keep the ones whose score clears a threshold you choose on a validation set. Neural nets do this naturally with one sigmoid output per label and a binary cross-entropy loss.

Everyday analogy: multiclass is a single-choice quiz (pick one answer). Multi-label is "tick all that apply".

scikit-learn stores multi-label targets as an indicator matrix — a 0/1 grid with one column per label. The MultiLabelBinarizer converts human-readable label lists into this matrix and back.

from sklearn.datasets import make_multilabel_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MultiLabelBinarizer

# First, see how MultiLabelBinarizer turns label LISTS into a 0/1 grid.
# Each row is one example; each column is one possible label; 1 = label present.
raw_labels = [["people", "concert"], ["nature"], ["people", "nature", "concert"]]
mlb = MultiLabelBinarizer()
Y_demo = mlb.fit_transform(raw_labels)
print("Indicator matrix:\n", Y_demo)
print("Label columns:", list(mlb.classes_))
print("Back to lists:", mlb.inverse_transform(Y_demo))
print()

# Now build a real (tiny) multi-label dataset directly as an indicator matrix.
X, Y = make_multilabel_classification(n_samples=120, n_features=6, n_classes=3,
                                      n_labels=2, random_state=42)
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.25,
                                                    random_state=42)

# A RandomForest can train on the 0/1 matrix directly: it effectively learns
# one binary model per label column.
clf = RandomForestClassifier(n_estimators=50, random_state=42, n_jobs=1)
clf.fit(X_train, Y_train)

print("True labels (first 3 test rows):\n", Y_test[:3])
print("Predicted labels (first 3 test rows):\n", clf.predict(X_test[:3]))
print("Subset accuracy on test set:", round(clf.score(X_test, Y_test), 3))

Indicator matrix:
 [[1 0 1]
 [0 1 0]
 [1 1 1]]
Label columns: ['concert', 'nature', 'people']
Back to lists: [('concert', 'people'), ('nature',), ('concert', 'nature', 'people')]

True labels (first 3 test rows):
 [[0 1 0]
 [0 1 1]
 [1 0 0]]
Predicted labels (first 3 test rows):
 [[0 1 0]
 [0 1 1]
 [1 0 0]]
Subset accuracy on test set: 0.667

Reading the output

The indicator matrix is just a tidy 0/1 table: row 0 [1 1 0] means "people yes, concert yes, nature no". inverse_transform flips it back to readable label lists.

The RandomForest trains on this matrix directly and predicts a full 0/1 row per example — so one photo can come out with several 1's. The score shown is subset accuracy: it only counts a prediction correct when every label is right, which is harsh, so don't be alarmed if it looks modest. In practice people report per-label accuracy or F1 instead.

7.5 Ensemble Learning

So far we've tried to build one accurate model. Ensemble learning takes the opposite bet: train a crowd of cheap, individually-weak models and combine their votes into one strong "meta-model". The crowd is right more often than any single member — if the members disagree with each other in different ways.

Why does this work? Every model makes errors, but if the errors are uncorrelated, they cancel out when you average or vote. Two main strategies dominate:

• Bagging builds many models in parallel on random resamples of the data, then averages. It tames variance (overfitting). Random Forest is the star example.
• Boosting builds models sequentially: each new model focuses on fixing the previous one's mistakes. It tames bias (underfitting), but can overfit if pushed too far. Gradient Boosting is the star example.

A third, stacking, trains several different models and then a meta-learner that learns whose advice to trust.

Everyday analogy: bagging asks 100 people the same trivia question and takes the majority. Boosting has each person study only the questions the previous people got wrong. Stacking is a manager who learns which employee to trust on which topic.

7.5.1 Bagging & Random Forest

Bagging ("bootstrap aggregating") goes like this: draw B random samples with replacement from your training set (each sample the same size as the original, so some rows repeat and some are missing), train a shallow decision tree on each sample, then average their predictions (or majority-vote for classification).

Random Forest adds one twist: at every split, each tree only looks at a random subset of the features. Why? If one feature is a super-predictor, every tree would latch onto it and become nearly identical — and identical trees can't cancel each other's errors. Forcing each tree to see different features keeps the forest diverse, and diversity is what makes ensembling pay off.

The big dials to tune are the number of trees B and the feature-subset size.

7.5.2 Boosting & Gradient Boosting

Boosting builds its crowd in sequence. For regression you start with a constant prediction (the average label), then repeatedly:

1. Compute each example's residual = true label − current prediction (how far off the model is).
2. Train a small tree to predict those residuals — i.e., to fix the current model's errors.
3. Add the new tree to the ensemble, scaled by a small learning rate.

Repeat for M trees. Each tree patches the mistakes of the ones before it. It's called "gradient" boosting because the residuals act as a stand-in for the gradient of the error — they point in the direction that reduces the loss, just like gradient descent did for linear regression in Chapter 4.

Key dials: number of trees M, learning rate, and tree depth. Boosting reduces bias and often beats random forest on accuracy, but its sequential nature makes it slower to train, and with too many deep trees it can overfit (unlike bagging, which rarely does).

from sklearn.datasets import make_moons
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Two interleaving half-circles -- a classic non-linear problem
X, y = make_moons(n_samples=200, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=42)

# Bagging family: Random Forest (many trees in parallel, averaged)
rf = RandomForestClassifier(n_estimators=50, random_state=42, n_jobs=1)
rf.fit(X_train, y_train)

# Boosting family: Gradient Boosting (trees in sequence, each fixes the last)
gb = GradientBoostingClassifier(n_estimators=50, max_depth=3,
                                learning_rate=0.1, random_state=42)
gb.fit(X_train, y_train)

# Shared grid for decision-boundary plots
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 250),
                     np.linspace(y_min, y_max, 250))
grid = np.c_[xx.ravel(), yy.ravel()]

fig, axes = plt.subplots(1, 2, figsize=(11, 4), sharey=True)
for ax, model, name in [(axes[0], rf, "Random Forest (bagging)"),
                        (axes[1], gb, "Gradient Boosting (boosting)")]:
    Z = model.predict(grid).reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.3, cmap="coolwarm")
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap="coolwarm",
               edgecolors="k", s=30)
    acc = accuracy_score(y_test, model.predict(X_test))
    ax.set_title(f"{name}\ntest accuracy = {acc:.2f}")
    ax.set_xlabel("feature 1")
axes[0].set_ylabel("feature 2")
plt.tight_layout()
plt.show()

print(f"Random Forest test accuracy     : {accuracy_score(y_test, rf.predict(X_test)):.3f}")
print(f"Gradient Boosting test accuracy : {accuracy_score(y_test, gb.predict(X_test)):.3f}")

Random Forest test accuracy     : 0.900
Gradient Boosting test accuracy : 0.833

Reading the hero plot

The two half-moons can't be separated by a straight line, but both ensembles carve out a curved, flexible boundary that hugs the moons — something a single shallow tree couldn't do. On this small dataset the two accuracies are usually close; the story is in how they got there:

• Random Forest averaged 50 independently-grown trees, smoothing away each tree's overfitting (lower variance).
• Gradient Boosting grew 50 trees in a chain, each one cleaning up the previous one's errors (lower bias).

Both beat what a lone tree would do. That's the whole appeal of ensembling: many weak, diverse models combine into one strong one.

7.5.3 Stacking

Bagging and boosting combine copies of the same weak learner. Stacking combines different kinds of models — say a tree, a k-NN, and a logistic regression — and then trains a meta-learner on their predictions. The meta-learner learns which base model to trust for a given input, rather than just averaging everyone equally.

The honest way to train it: split your data, train the base models on one part, have them predict on the other part, and use those predictions (plus the true labels) to train the meta-learner. Otherwise the meta-learner just trusts whichever base model memorized the training data. scikit-learn's StackingClassifier handles this cross-fitting for you.

from sklearn.ensemble import StackingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Same moons data as the hero demo (regenerated so this cell stands alone)
X, y = make_moons(n_samples=200, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=42)

# Two DIFFERENT base learners + a logistic-regression meta-learner on top
base = [("tree", DecisionTreeClassifier(max_depth=3, random_state=42)),
        ("knn",  KNeighborsClassifier(n_neighbors=5))]
stack = StackingClassifier(estimators=base,
                           final_estimator=LogisticRegression(max_iter=300,
                                                               random_state=42),
                           cv=3, n_jobs=1)
stack.fit(X_train, y_train)

# Compare: a lone shallow tree vs k-NN vs the stacked ensemble
tree_only = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)
knn_only  = KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train)

print(f"Lone shallow tree  : {accuracy_score(y_test, tree_only.predict(X_test)):.3f}")
print(f"k-NN (k=5)         : {accuracy_score(y_test, knn_only.predict(X_test)):.3f}")
print(f"Stacked ensemble   : {accuracy_score(y_test, stack.predict(X_test)):.3f}")

Lone shallow tree  : 0.850
k-NN (k=5)         : 0.917
Stacked ensemble   : 0.883

Reading the output

The lone shallow tree does worst (one small tree can't capture both moons). The stacked ensemble usually matches or beats its best base model, because the meta-learner can lean on the tree where it's strong and on k-NN where the tree is weak. Stacking pays off most when your base models make different kinds of mistakes.

Other Problem Types (Brief Notes)

A few more supervised flavors you should know exist — no demos here, just the idea.

7.6 Learning to Label Sequences

A sequence is structured data where order matters: words in a sentence, daily stock prices, notes in a melody. In sequence labeling each position in the input gets its own label — e.g., tagging each word with its part of speech ("big"→adjective, "car"→noun). Recurrent neural nets (RNNs) handle this naturally by reading one step at a time. A classic non-neural alternative is the Conditional Random Field (CRF), which you can think of as logistic regression generalized to whole sequences — it scores how well a sequence of labels fits a sequence of inputs.

7.7 Sequence-to-Sequence Learning

Seq2seq generalizes sequence labeling: now the input sequence and output sequence can have different lengths. The poster child is machine translation (English sentence in, French sentence out). The standard architecture has two parts — an encoder that reads the input into a compact vector called an embedding, and a decoder that generates the output sequence from that embedding, feeding each output back in as the next input. Adding an attention mechanism lets the decoder focus on the most relevant input positions, which dramatically helps with longer sequences.

7.8 Active Learning

Labels can be expensive — a doctor must read each scan, an analyst each transaction. Active learning starts with a few labeled examples and a pile of unlabeled ones, then chooses which unlabeled example to ask the expert about next — picking the one the current model is most uncertain about (or that sits in a dense, confusing region). Each new label is added, the model is retrained, and the cycle repeats until the budget runs out or accuracy plateaus. It's supervised learning that shops for labels wisely.

7.9 Semi-Supervised Learning

What if you have a handful of labeled examples and a mountain of unlabeled ones? Throwing the unlabeled data away seems wasteful, but you can't train a supervised model on blanks. Semi-supervised learning (SSL) tries to squeeze useful signal out of the unlabeled mass too.

One simple idea is self-learning: train on the labeled few, predict the unlabeled many, and "promote" the high-confidence predictions into the training set, then repeat. It can help, but it can also drift off course if the early model is confidently wrong. A gentler, graph-based idea is label propagation: treat every example (labeled or not) as a node, connect nearby examples, and let the known labels bleed across the graph to their neighbors. Unlabeled points that sit inside a labeled region inherit that region's label. sklearn gives us LabelPropagation for exactly this.

from sklearn.semi_supervised import LabelPropagation
from sklearn.datasets import make_blobs

# 3 blobs; we'll reveal only a handful of labels and hide the rest
X, y_true = make_blobs(n_samples=150, centers=3, cluster_std=1.2,
                       random_state=42)

# Copy the labels, then hide ~90% of them (mark as -1 = "unlabeled")
rng = np.random.RandomState(1)
y_hidden = y_true.copy()
unlabeled_idx = rng.choice(len(y_true), size=135, replace=False)
y_hidden[unlabeled_idx] = -1

# LabelPropagation spreads the known labels across the graph of nearby points
lp = LabelPropagation(kernel="knn", n_neighbors=7)
lp.fit(X, y_hidden)            # entries equal to -1 are treated as unlabeled
y_learned = lp.transduction_   # the labels the model assigned to every point

fig, axes = plt.subplots(1, 2, figsize=(11, 4), sharey=True)
# Left: what we actually gave the model (only 15 labeled dots)
axes[0].scatter(X[:, 0], X[:, 1], c="lightgray", s=20)
labeled = y_hidden != -1
axes[0].scatter(X[labeled, 0], X[labeled, 1], c=y_hidden[labeled],
                cmap="viridis", edgecolors="k", s=80)
axes[0].set_title("Given to model: 15 labels, 135 unknown (gray)")
axes[0].set_xlabel("feature 1")
axes[0].set_ylabel("feature 2")

# Right: labels propagated to everyone
axes[1].scatter(X[:, 0], X[:, 1], c=y_learned, cmap="viridis",
                edgecolors="k", s=30)
axes[1].set_title("After label propagation: everyone gets a label")
axes[1].set_xlabel("feature 1")
plt.tight_layout()
plt.show()

acc = np.mean(y_learned == y_true)
print(f"Recovered label accuracy vs. truth: {acc:.3f} (from only 15 seeds)")

Recovered label accuracy vs. truth: 1.000 (from only 15 seeds)

Reading the plot

On the left, the model saw mostly gray (unknown) dots plus just 15 colored seeds. On the right, label propagation has spread those seeds across the graph: each unlabeled point adopted the label most common among its 7 nearest neighbors, so the three blobs get painted in even though we never labeled 90% of them. The recovered accuracy tells you how often those propagated labels match the hidden ground truth — usually quite good when the clusters are clean and well-separated like here.

7.10 One-Shot Learning

The name is slightly misleading: it doesn't mean you train on one example. It means the deployed model only needs one reference example of a new identity to recognize it — the classic case is face unlock: you enroll with a single selfie, and your phone can later tell "you" from "not you".

The usual approach is a siamese network: the same neural net processes two images and produces an embedding (a compact vector) for each. The network is trained with a triplet loss using triplets (anchor, positive, negative) — two photos of the same person and one of a different person — pushing same-person embeddings close together and different-person embeddings far apart. At inference time you compare a new photo's embedding to the stored one using a distance threshold.

Everyday analogy: you don't need to see a hundred photos of a new coworker to recognize them tomorrow — one good look is enough, because you've already learned generally what "faces" and "differences between faces" look like.

7.11 Zero-Shot Learning

Zero-shot learning (ZSL) asks the impossible-sounding question: can a model predict a label it has never seen in training? The trick is to represent not just the inputs but also the labels as vectors (think word embeddings, where words with similar meanings sit close together in space).

Instead of predicting a class index, the model predicts an embedding vector. To name the prediction, you find the word in your dictionary whose embedding is closest (by cosine similarity) to the predicted one. Because the model learns features that transfer — "stripiness", "mammalness", "orangeness" — it can land on a label like tiger even if no tiger appeared in training, as long as related concepts (zebra, clownfish) did.

Everyday analogy: you've never tasted a "pluot", but you know plums and apricots, so when you taste one you can guess its name from the two familiar flavors it combines.

Key Takeaways

• Multiclass = one label out of many; solve it natively with softmax (logistic regression) or with one-vs-rest when your algorithm is binary-only.
• Softmax turns raw scores into a probability distribution that is positive and sums to 1.
• One-class classification learns what "normal" looks like and flags the rest as outliers — handy when anomalous examples are rare (Isolation Forest, one-class SVM).
• Multi-label = several labels per example; usually modeled as one binary problem per label, stored as a 0/1 indicator matrix.
• Ensembles beat single models by combining many diverse weak learners: bagging (Random Forest) lowers variance; boosting (Gradient Boosting) lowers bias; stacking lets a meta-learner choose which base model to trust.
• Semi-supervised learning extracts signal from unlabeled data — label propagation spreads known labels across a graph of nearby points.
• One-shot and zero-shot learning extend supervised ideas to settings with almost no labeled data, using embeddings and similarity.

What's Next

In Chapter 8 — Advanced Practice, we'll move beyond individual problem types and look at the craft of putting machine learning to work for real: feature engineering, model evaluation, hyperparameter tuning, and avoiding the practical traps that sink naive projects.

Exercises

These exercises cover the problem types and solutions from Chapter 7: kernel regression, multiclass, one-class/anomaly detection, multi-label, ensemble learning, sequence tasks, and semi-supervised/one-shot/zero-shot learning. Try each before reading the hint.

1. (Conceptual) Kernel regression is non-parametric. What does it predict for a new point x, and what does the bandwidth b control? Hint: it takes a kernel-weighted average of training labels; a tiny b chases every wiggle (overfit), a huge b flattens to near-line (underfit).
2. (Conceptual) Distinguish multiclass from multi-label classification with one example each. Hint: multiclass = one label out of many (a digit 0–9); multi-label = several labels at once (a photo tagged "people, concert, nature").
3. (Conceptual) Explain what softmax does and why its outputs sum to 1, and how it differs from sigmoid. Hint: softmax exponentiates each of C scores and normalizes so the C values are positive and sum to 1 (a distribution); sigmoid outputs a single (0,1) value.
4. (Conceptual) What is one-vs-rest (OvR), and when do you need it? Hint: you have a binary-only algorithm (e.g. SVM) but a multiclass problem; train C binary "is this class c?" classifiers and pick the most confident.
5. (Conceptual) Describe the use case for one-class classification and name two approaches. Hint: you have piles of "normal" examples and almost no "bad" ones (network anomaly detection); fit a density, a one-class SVM, or an Isolation Forest.
6. (Conceptual) Contrast bagging and boosting: what does each reduce, parallel vs sequential, and one example of each. Hint: bagging trains in parallel on resamples and lowers variance (Random Forest); boosting trains sequentially, each fixing prior errors, and lowers bias (Gradient Boosting).
7. (Conceptual) Why does a Random Forest randomly subset the features at each split? Hint: to keep the trees diverse — without it they'd all latch onto the same strong feature and couldn't cancel each other's errors.
8. (Conceptual) Explain label propagation in one sentence, and state the assumption it relies on. Hint: known labels "bleed" across a graph connecting nearby points; it assumes clusters are clean and well-connected so neighbors usually share a label.

Hands-On Coding Problems

9. (Coding) Implement softmax(scores) from scratch on [2.0, 1.0, 0.1]; print the probabilities and confirm they sum to 1. Hint: np.exp(scores) / np.sum(np.exp(scores)).
10. (Coding) Multiclass: fit LogisticRegression(max_iter=500) on a 3-class, 2-feature make_classification dataset; plot the decision regions and print predict_proba for one test point (should sum to 1). Hint: make_classification(n_classes=3, n_features=2, n_informative=2, n_redundant=0, n_clusters_per_class=1).
11. (Coding) One-class anomaly detection: build normal data with make_blobs, add a few far-away outlier points, fit IsolationForest(contamination=0.1, random_state=0), and print which points it flags as anomalies (predict returns 1 for inliers, −1 for outliers). Hint: .predict on the combined normal+outlier set.
12. (Coding) Multi-label: take a tiny list of label lists ([["people","concert"],["concert","nature"],["people"]]), encode it with MultiLabelBinarizer, fit a RandomForestClassifier, and predict the labels for one new example. Hint: MultiLabelBinarizer().fit_transform(labels).
13. (Coding) Ensembles on make_moons: fit a single DecisionTreeClassifier, a RandomForestClassifier(n_estimators=50), and a GradientBoostingClassifier(n_estimators=50); split train/test and print all three test accuracies. Hint: the lone tree should do worst; both ensembles should beat it.
14. (Coding) Stacking: use StackingClassifier with decision-tree, k-NN, and logistic-regression base estimators and a logistic-regression meta-learner on make_moons; print its test accuracy next to the lone tree's. Hint: StackingClassifier(estimators=[("tree",...),("knn",...),("log",...)], final_estimator=LogisticRegression()).
15. (Coding) Semi-supervised: create 3 blobs with make_blobs, reveal only ~15 labels (set the rest to −1 for "unknown"), fit LabelPropagation, and print the recovered accuracy versus the hidden ground truth. Hint: LabelPropagation(kernel="knn", n_neighbors=7).fit(X, y_partly_unknown); −1 means unlabeled to sklearn.

# Exercise 9: softmax from scratch
import numpy as np

def softmax(scores):
    # TODO: exponentiate and normalize so the values sum to 1
    return np.zeros_like(scores)  # placeholder

scores = np.array([2.0, 1.0, 0.1])
probs = softmax(scores)
print("probs =", probs)
print("sum  =", probs.sum())

# Exercise 10: multiclass logistic regression with softmax
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_classes=3, n_features=2,
                           n_informative=2, n_redundant=0,
                           n_clusters_per_class=1, random_state=0)

# TODO: fit LogisticRegression(max_iter=500) and plot decision regions on a meshgrid
clf = None  # placeholder

# TODO: print predict_proba for X[0] and confirm it sums to 1

# Exercise 11: one-class anomaly detection with Isolation Forest
import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.datasets import make_blobs

X_normal, _ = make_blobs(n_samples=200, centers=1, cluster_std=0.6, random_state=0)
rng = np.random.default_rng(0)
X_outliers = rng.uniform(low=-8, high=8, size=(20, 2))
X_all = np.vstack([X_normal, X_outliers])

# TODO: fit IsolationForest(contamination=0.1, random_state=0) on X_normal (or X_all),
#       then predict on X_all and count how many are flagged as outliers (-1)
flagged = 0  # placeholder

print("number of points flagged as outliers:", flagged)

# Exercise 12: multi-label classification
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.ensemble import RandomForestClassifier

labels = [["people", "concert"], ["concert", "nature"], ["people"], ["nature", "people", "concert"]]
mlb = MultiLabelBinarizer()

# TODO: fit_transform the labels into an indicator matrix Y
Y = None  # placeholder

# TODO: make a tiny feature matrix X (one row per example) and fit RandomForestClassifier()
clf = None  # placeholder

# TODO: predict the labels for one new example and use mlb.inverse_transform to read them back
print("indicator matrix:\n", Y)

# Exercise 13: lone tree vs Random Forest vs Gradient Boosting
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=300, noise=0.25, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

models = {
    "tree": DecisionTreeClassifier(random_state=0),
    "random forest": RandomForestClassifier(n_estimators=50, random_state=0),
    "gradient boosting": GradientBoostingClassifier(n_estimators=50, random_state=0),
}

for name, model in models.items():
    # TODO: fit on Xtr and print .score on Xte
    acc = 0.0  # placeholder
    print(f"{name:18s} test accuracy = {acc:.3f}")

# Exercise 14: stacking with a meta-learner
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import StackingClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=300, noise=0.25, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

base = [
    ("tree", DecisionTreeClassifier(random_state=0)),
    ("knn", KNeighborsClassifier(n_neighbors=5)),
    ("log", LogisticRegression(max_iter=500)),
]

# TODO: build StackingClassifier(estimators=base, final_estimator=LogisticRegression(max_iter=500))
stack = None  # placeholder

# TODO: fit and print its test accuracy, plus the lone tree's test accuracy for comparison

# Exercise 15: semi-supervised label propagation
import numpy as np
from sklearn.datasets import make_blobs
from sklearn.semi_supervised import LabelPropagation
from sklearn.metrics import accuracy_score

X, y_true = make_blobs(n_samples=300, centers=3, cluster_std=1.0, random_state=42)
rng = np.random.default_rng(0)

# Reveal only ~15 labels; mark the rest as unknown (-1)
y_partial = np.full_like(y_true, -1)
known_idx = rng.choice(len(y_true), size=15, replace=False)
y_partial[known_idx] = y_true[known_idx]

# TODO: fit LabelPropagation(kernel="knn", n_neighbors=7) on X and y_partial
lp = None  # placeholder

# TODO: predict on X and print accuracy_score(y_true, predicted)
acc = 0.0  # placeholder
print("recovered accuracy:", acc)