Chapter 8 — Advanced Practice

Some techniques aren't harder than the basics — they're just extra tools you reach for in specific situations. This chapter collects the most useful "advanced practice" tricks: dealing with lopsided datasets, blending several models together, training neural networks without losing your mind, regularizing in cleverer ways, handling several inputs or outputs at once, borrowing knowledge from one model to solve a different problem, and thinking about how fast your code runs.

In this chapter you will learn:

• Why a 99% accuracy score can be terrible news, and how to fix imbalanced datasets with class weights, resampling, and threshold tuning.
• How to combine several models (averaging, voting, stacking) so they're smarter together than alone.
• Practical tips for training neural networks: learning-rate schedules, batch size, and early stopping.
• Advanced regularization ideas: dropout, batch normalization, and data augmentation.
• How to feed a model several kinds of input at once, or ask it for several outputs.
• What transfer learning is and why it's a superpower of neural networks.
• How to reason about algorithm speed with big-O notation.

8.1 Handling Imbalanced Datasets

Imagine you're building a fraud detector for an online store. Out of every 1,000 transactions, maybe 5 are fraud and 995 are honest. If your model simply shouted "honest!" every single time, it would be right 99.5% of the time. That sounds great — until you realize it catches exactly zero fraud.

This is the imbalanced dataset trap: when one class vastly outnumbers the others, a model can rack up a high accuracy (fraction of correct predictions) while being useless for the class you actually care about. The majority class drowns out the minority.

The fix in one line: make the minority class "count more" during training, or change how you measure success.

A few standard techniques:

• Class weights — tell the learning algorithm "a mistake on a minority example is N times more expensive than a mistake on a majority example." Most sklearn classifiers accept class_weight='balanced', which sets weights inversely proportional to class frequency.
• Resampling — oversample the minority class (duplicate its examples, or invent synthetic ones) so both classes are equally common, or undersample the majority class (randomly drop some of its examples).
• Threshold tuning — many classifiers output a probability; the default decision threshold is 0.5. With imbalanced data, a different threshold often gives far better recall on the minority class.

Everyday analogy: it's like a teacher grading a class where 30 students are fluent in the topic and 1 student is struggling. If the teacher only looks at "class average score," everything looks fine and the struggling student vanishes from the statistics. You need a metric (and a strategy) that sees the minority.

Let's see the problem and the fixes in code.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, balanced_accuracy_score,
                             f1_score, confusion_matrix, ConfusionMatrixDisplay)

# Consistent figure size for the whole notebook
plt.rcParams['figure.figsize'] = (6, 4)

# --- Build a lopsided dataset: 5% minority (class 1), 95% majority (class 0) ---
X, y = make_classification(n_samples=1000, n_features=8, n_informative=5,
                           n_redundant=2, n_classes=2, weights=[0.95, 0.05],
                           flip_y=0.02, random_state=7)

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=7,
                                      stratify=y)

print("Training class counts:", np.bincount(ytr))

# Train a plain logistic regression (no special handling of imbalance)
clf = LogisticRegression(max_iter=1000, random_state=7)
clf.fit(Xtr, ytr)
pred = clf.predict(Xte)

print("\n--- Plain model (ignores imbalance) ---")
print("Accuracy            : %.3f" % accuracy_score(yte, pred))
print("Balanced accuracy   : %.3f" % balanced_accuracy_score(yte, pred))
print("F1 (minority class) : %.3f" % f1_score(yte, pred))
print("Confusion matrix (rows=true, cols=pred):\n", confusion_matrix(yte, pred))

Training class counts: [659  41]

--- Plain model (ignores imbalance) ---
Accuracy            : 0.963
Balanced accuracy   : 0.676
F1 (minority class) : 0.522
Confusion matrix (rows=true, cols=pred):
 [[283   0]
 [ 11   6]]

Reading those numbers

The accuracy looks impressive (well above 90%), but the balanced accuracy and F1 tell a different, sadder story. The confusion matrix usually shows a big block of correct majority predictions and a worrying number of missed minority examples (false negatives).

• Balanced accuracy averages the recall of each class, so a model that ignores the minority class can't hide.
• F1 combines precision and recall for the minority class — exactly what you care about in fraud detection.

When the classes are lopsided, never trust accuracy alone.

# Now flip on class weighting: sklearn will up-weight the rare class
clf_bal = LogisticRegression(max_iter=1000, class_weight='balanced', random_state=7)
clf_bal.fit(Xtr, ytr)
pred_bal = clf_bal.predict(Xte)

print("--- Weighted model (class_weight='balanced') ---")
print("Accuracy            : %.3f" % accuracy_score(yte, pred_bal))
print("Balanced accuracy   : %.3f" % balanced_accuracy_score(yte, pred_bal))
print("F1 (minority class) : %.3f" % f1_score(yte, pred_bal))

# Side-by-side confusion matrices: before vs after weighting
fig, axes = plt.subplots(1, 2, figsize=(10, 4))
ConfusionMatrixDisplay.from_predictions(yte, pred, ax=axes[0], colorbar=False)
axes[0].set_title("Plain model\n(ignores imbalance)")
ConfusionMatrixDisplay.from_predictions(yte, pred_bal, ax=axes[1], colorbar=False)
axes[1].set_title("Weighted model\n(class_weight='balanced')")
plt.tight_layout(); plt.show()

--- Weighted model (class_weight='balanced') ---
Accuracy            : 0.797
Balanced accuracy   : 0.754
F1 (minority class) : 0.282

What changed

Weighting the classes trades a little accuracy on the majority class for a lot more recall on the minority class. Look at the confusion matrix: the weighted model catches more fraud (fewer false negatives), at the cost of a few more false alarms (false positives). Whether that trade is worth it depends on your problem — in fraud or disease screening, missing a real case is usually far more expensive than a false alarm.

Now let's try resampling and threshold tuning, two more knobs you can turn.

# --- 1) Resampling by hand (no extra libraries needed) ---
# Oversample the minority class: duplicate its training rows until ~balanced
mask_min = (ytr == 1)
Xtr_over = np.vstack([Xtr, Xtr[mask_min].repeat(18, axis=0)])  # ~match majority
ytr_over = np.concatenate([ytr, ytr[mask_min].repeat(18)])
print("After oversampling, train class counts:", np.bincount(ytr_over))

clf_over = LogisticRegression(max_iter=1000, random_state=7)
clf_over.fit(Xtr_over, ytr_over)
print("Oversample F1 : %.3f" % f1_score(yte, clf_over.predict(Xte)))

# Undersample the majority class: randomly drop majority rows until balanced
rng = np.random.RandomState(7)
maj_idx = np.where(ytr == 0)[0]
keep = rng.choice(maj_idx, size=int(mask_min.sum()), replace=False)
idx_under = np.concatenate([keep, np.where(ytr == 1)[0]])
clf_under = LogisticRegression(max_iter=1000, random_state=7)
clf_under.fit(Xtr[idx_under], ytr[idx_under])
print("Undersample F1: %.3f" % f1_score(yte, clf_under.predict(Xte)))

# --- 2) Threshold tuning on the PROBABILITY output ---
proba = clf.predict_proba(Xte)[:, 1]   # P(class=1)
thresholds = np.linspace(0.05, 0.95, 19)
f1s = [f1_score(yte, proba >= t) for t in thresholds]

best_t = thresholds[int(np.argmax(f1s))]
plt.plot(thresholds, f1s, 'o-')
plt.axvline(best_t, color='r', ls='--', label="best threshold=%.2f" % best_t)
plt.xlabel("Decision threshold"); plt.ylabel("F1 (minority class)")
plt.title("Threshold tuning: F1 vs decision threshold")
plt.legend(); plt.tight_layout(); plt.show()
print("Default 0.5 F1: %.3f  |  best F1: %.3f" % (f1_score(yte, proba>=0.5), max(f1s)))

After oversampling, train class counts: [659 779]
Oversample F1 : 0.252
Undersample F1: 0.276

Default 0.5 F1: 0.522  |  best F1: 0.643

8.2 Combining Models

A Random Forest already combines many trees of the same kind. But you can also mix different kinds of models — a logistic regression, a support vector machine, and a random forest, say — and the mixture can beat any single one. The magic word is uncorrelated: if the models make different mistakes, those mistakes cancel out when they vote.

Three classic ways to combine:

• Averaging — each model outputs a score; you take the mean. Great for regression and for classifiers that return probabilities.
• Majority vote — each model outputs a class label; you pick the class that the most models agree on. (Ties are broken randomly or reported as "unsure.")
• Stacking — train a meta-model whose input features are the predictions of the base models. The meta-model learns which base model to trust in which situation.

Everyday analogy: asking three friends with different backgrounds for restaurant recommendations and going with the place two of them agree on. If they all have the same taste, you gain nothing; if their tastes differ, the consensus is usually safer than any single pick.

from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.svm import SVC
from sklearn.datasets import make_moons

# A small, slightly tricky 2D dataset
X, y = make_moons(n_samples=300, noise=0.25, random_state=4)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=4)

# Three different "base" models
log = LogisticRegression(max_iter=1000, random_state=4)
rf  = RandomForestClassifier(n_estimators=50, max_depth=5, random_state=4, n_jobs=1)
svm = SVC(probability=True, random_state=4)   # probability=True so soft voting works

# Hard voting = majority of predicted labels
# Soft voting = average of predicted probabilities
vote_hard = VotingClassifier([('log', log), ('rf', rf), ('svm', svm)], voting='hard')
vote_soft = VotingClassifier([('log', log), ('rf', rf), ('svm', svm)], voting='soft')

for name, m in [('LogReg', log), ('RandForest', rf), ('SVM', svm),
                ('Vote(hard)', vote_hard), ('Vote(soft)', vote_soft)]:
    m.fit(Xtr, ytr)
    print("%-12s test accuracy: %.3f" % (name, m.score(Xte, yte)))

LogReg       test accuracy: 0.878
RandForest   test accuracy: 0.956
SVM          test accuracy: 0.956
Vote(hard)   test accuracy: 0.944
Vote(soft)   test accuracy: 0.933

Reading the result

Often the ensemble matches or beats the best individual model, and — just as importantly — it's less jittery: it rarely collapses on a bad split the way a single model can. Soft voting (averaging probabilities) usually edges out hard voting because it uses more information than a flat label.

from sklearn.ensemble import StackingClassifier

# Stacking: base models feed a "meta" model that learns who to trust
stack = StackingClassifier(
    estimators=[('log', log), ('rf', rf), ('svm', svm)],
    final_estimator=LogisticRegression(max_iter=1000),  # the meta-model
    cv=5,           # use cross-validation to build the meta-features (avoids leakage)
    n_jobs=1,
)
stack.fit(Xtr, ytr)
print("Stacking     test accuracy: %.3f" % stack.score(Xte, yte))

Stacking     test accuracy: 0.933

Why stacking can help

The meta-model doesn't just average — it learns weights. If the SVM tends to be right where the logistic model is wrong, the meta-model can learn to lean on the SVM in that region. The cv=5 setting builds the meta-features with cross-validation so the meta-model never sees predictions made on data a base model was already trained on (that would be leakage — cheating).

Just remember the warning: stacking two near-identical models (say, two SVMs with similar settings) gains little. Combine models of different natures.

8.3 Training Neural Networks

Training a neural network has a few extra dials compared to a simple model. The most important practical ones:

• Learning rate — the step size for gradient descent. Too big and training bounces around; too small and it crawls. A learning-rate schedule starts large (to move fast) and decays over time (to settle in). sklearn's MLPClassifier supports learning_rate='invscaling', which shrinks the step each epoch.
• Batch size — how many examples you process before updating the weights. Small batches = noisier but cheaper updates; large batches = smoother but slower per epoch.
• Early stopping — instead of training for a fixed number of epochs, you watch a validation set after each epoch and stop the moment validation performance stops improving (or starts getting worse). This prevents overfitting and saves time.

Everyday analogy: tuning a guitar. You turn the peg boldly at first to get close, then make tiny adjustments to fine-tune (that's a learning-rate schedule). You stop when it sounds right, not after some fixed number of turns (that's early stopping) — otherwise you'll overshoot and snap the string.

from sklearn.neural_network import MLPClassifier

X, y = make_classification(n_samples=800, n_features=10, n_informative=6,
                           n_redundant=2, flip_y=0.1, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=3)

# Model A: train a fixed 300 epochs, no early stopping
mlp_fixed = MLPClassifier(hidden_layer_sizes=(16, 8), max_iter=300,
                          random_state=3, n_iter_no_change=10000)  # effectively no early stop
mlp_fixed.fit(Xtr, ytr)

# Model B: early stopping -- halt when validation score stops improving
mlp_early = MLPClassifier(hidden_layer_sizes=(16, 8), max_iter=300,
                          early_stopping=True, validation_fraction=0.2,
                          n_iter_no_change=10, random_state=3)
mlp_early.fit(Xtr, ytr)

print("Fixed  (no early stop): %d epochs, train acc %.3f, test acc %.3f" %
      (mlp_fixed.n_iter_, mlp_fixed.score(Xtr,ytr), mlp_fixed.score(Xte,yte)))
print("Early stop           : %d epochs, train acc %.3f, test acc %.3f" %
      (mlp_early.n_iter_, mlp_early.score(Xtr,ytr), mlp_early.score(Xte,yte)))

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (300) reached and the optimization hasn't converged yet.
  warnings.warn(

Fixed  (no early stop): 300 epochs, train acc 0.923, test acc 0.858
Early stop           : 48 epochs, train acc 0.761, test acc 0.787

Reading the result

The early-stopping model usually trains in fewer epochs and often holds a slightly better test accuracy, because it quits before it starts memorizing the training set's noise. The fixed model, left to run, can overfit: its training accuracy keeps climbing while its test accuracy flattens or drops.

Early stopping is one of the cheapest, most reliable forms of regularization — try it first.

8.4 Advanced Regularization

Beyond L1 and L2, neural networks have their own regularizers:

• Dropout — during each training step, randomly turn off a fraction of the neurons (set their output to zero). The network can't rely on any single neuron, so it spreads its knowledge around — like a sports team that trains with random players benched, so no one becomes a single point of failure. The dropout rate (0 to 1) is tuned on validation data.
• Batch normalization ("batch standardization") — standardize each layer's outputs (zero mean, unit variance) before passing them to the next layer. It's not officially a regularizer, but it makes training faster, more stable, and often slightly more generalizing. In libraries you insert a BatchNorm layer between two layers.
• Data augmentation — instead of only training on your original examples, create synthetic ones by transforming them (rotate, flip, zoom, darken an image; add jitter to numbers). The label stays the same. More diverse training data means less overfitting. This works for any model, not just neural networks.

Everyday analogy for dropout: studying for an exam by occasionally hiding random pages of your notes — it forces you to learn the material more robustly instead of memorizing one page's layout.

Let's see data augmentation in action on a tiny 2D dataset (no images needed).

from sklearn.datasets import make_moons
from sklearn.neighbors import KNeighborsClassifier

# A small training set (intentionally small, so it overfits easily)
X, y = make_moons(n_samples=60, noise=0.30, random_state=1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=1)

# Baseline: KNN on the original (tiny) training set
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(Xtr, ytr)
base_train = knn.score(Xtr, ytr); base_test = knn.score(Xte, yte)

# Data augmentation: add Gaussian jitter (noise) to copies of training points.
# Same labels -- we're inventing "nearby but slightly moved" moons.
rng = np.random.RandomState(1)
X_aug = [Xtr]; y_aug = [ytr]
for _ in range(20):                       # make 20 jittered copies
    X_aug.append(Xtr + rng.normal(0, 0.10, size=Xtr.shape))
    y_aug.append(ytr)
Xtr_aug = np.vstack(X_aug); ytr_aug = np.concatenate(y_aug)

knn_aug = KNeighborsClassifier(n_neighbors=3)
knn_aug.fit(Xtr_aug, ytr_aug)
aug_train = knn_aug.score(Xtr, ytr); aug_test = knn_aug.score(Xte, yte)

print("Original data : train %.3f, test %.3f" % (base_train, base_test))
print("Augmented data: train %.3f, test %.3f" % (aug_train, aug_test))

# Visualize the original vs augmented training points
plt.figure(figsize=(6,4))
plt.scatter(Xtr[:,0], Xtr[:,1], c=ytr, cmap='coolwarm', edgecolor='k', s=60, label='original')
aug_X = Xtr_aug[len(Xtr):]; aug_y = np.concatenate(y_aug)[len(ytr):]
plt.scatter(aug_X[:,0], aug_X[:,1], c=aug_y, cmap='coolwarm', alpha=0.15, s=15, label='augmented (jittered)')
plt.title("Data augmentation: jittered copies of the training points")
plt.xlabel("x1"); plt.ylabel("x2"); plt.legend(); plt.tight_layout(); plt.show()

Original data : train 0.944, test 0.917
Augmented data: train 1.000, test 0.917

What augmentation did

By adding jittered copies, we effectively told the model "the moon shape is fuzzy — points near these ones belong to the same class." The augmented model usually generalizes a bit better to the test set even though it's the same KNN with the same settings. That's the whole idea of augmentation: cheap, label-preserving diversity that fights overfitting.

Dropout and batch normalization aren't exposed in sklearn's MLPClassifier in a fine-grained way, but the concept is identical — and you'll meet them the moment you touch a deep-learning library like Keras.

8.5 Handling Multiple Inputs

Many real problems are multimodal: the input is several different kinds of thing at once. "Does this text describe this image?" combines an image and a sentence. How do you feed both into one model?

With shallow models, two options:

1. Train separate models on each input, then combine their predictions (averaging / voting / stacking — see the previous section).
2. Vectorize each input separately, then concatenate the feature vectors into one long vector. If the image gives features [i1, i2, i3] and the text gives [t1, t2, t3, t4], you feed the model [i1, i2, i3, t1, t2, t3, t4].

With neural networks you get more flexibility: build a subnetwork for each input (a CNN for the image, an RNN for the text), let each produce an embedding (a compact vector summary), then concatenate the embeddings and add a classification layer on top. Libraries like Keras make this "two towers, then merge, then classify" pattern easy.

Everyday analogy: judging a restaurant from both the menu (text) and the photos (images). You could score each separately and average, or read them together for a combined verdict — the combined view is usually richer.

Let's illustrate the simplest version — concatenating two feature sources.

from sklearn.tree import DecisionTreeClassifier

# Toy "multimodal" setup: the SAME samples described by two different feature blocks.
# "Modality A": 3 numeric features ; "Modality B": 4 numeric features.
# The target depends on information split across BOTH modalities.
rng = np.random.RandomState(5)
n = 300
A = rng.normal(size=(n, 3))          # modality A (e.g. "image features")
B = rng.normal(size=(n, 4))          # modality B (e.g. "text features")
# Label = 1 if a combo across A and B exceeds a threshold
y = (A[:,0] + B[:,2] > 0.5).astype(int)

# Same random_state + same row count => identical split indices for A and B
Xtr_A, Xte_A, ytr, yte = train_test_split(A, y, test_size=0.3, random_state=5)
Xtr_B, Xte_B, _, _ = train_test_split(B, y, test_size=0.3, random_state=5)

m_A = DecisionTreeClassifier(max_depth=3, random_state=5).fit(Xtr_A, ytr)
m_B = DecisionTreeClassifier(max_depth=3, random_state=5).fit(Xtr_B, ytr)

# Concatenated input: both modalities side by side
X_both = np.hstack([A, B])
Xtr_both, Xte_both, _, _ = train_test_split(X_both, y, test_size=0.3, random_state=5)
m_both = DecisionTreeClassifier(max_depth=3, random_state=5).fit(Xtr_both, ytr)

print("Only modality A : test acc %.3f" % m_A.score(Xte_A, yte))
print("Only modality B : test acc %.3f" % m_B.score(Xte_B, yte))
print("Both (concat)   : test acc %.3f" % m_both.score(Xte_both, yte))

Only modality A : test acc 0.744
Only modality B : test acc 0.767
Both (concat)   : test acc 0.900

Reading the result

Neither single modality has the full story (the label depends on both A[:,0] and B[:,2]), so each alone struggles. Concatenating them gives the model everything it needs in one vector — and accuracy jumps. That's the shallow-learning recipe for multimodal input in a nutshell.

For genuinely different types of data (pixels vs. words), the neural-network "two towers plus merge" approach scales better — but the underlying idea is the same: turn each input into a vector, then let the model see them together.

8.6 Handling Multiple Outputs

Sometimes one input should produce several outputs. Detect an object in a photo and return both its bounding-box coordinates and its category ("cat" / "dog" / "hamster").

Some multi-output problems can be flattened into multi-label classification (especially when the outputs are tags of the same kind). But when the outputs are different types — a vector of real numbers (coordinates) and a class label — flattening doesn't work.

The neural-network solution: one encoder subnetwork reads the input and produces an embedding. Then two heads branch off the embedding:

• Head 1 predicts coordinates (a regression head, often with ReLU output, trained with mean-squared-error cost C1).
• Head 2 predicts the class label (a softmax head, trained with cross-entropy cost C2).

You can't minimize both costs at once perfectly — improving one can hurt the other. The standard trick is a combined cost: C = alpha*C1 + (1-alpha)*C2, where alpha is a hyperparameter in (0, 1) you tune on validation data. It says "this much do I care about coordinates, the rest about the label."

Everyday analogy: a teacher grading both math correctness and writing neatness on the same exam. They combine the two sub-scores into one grade, weighting them by how much each matters.

Let's illustrate multi-output prediction with sklearn (one input, two targets).

from sklearn.multioutput import MultiOutputRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

# One input x -> TWO numeric outputs (a 2D regression target)
rng = np.random.RandomState(9)
X = rng.uniform(-3, 3, size=(120, 1))
# Output 1 ~ sin(x), Output 2 ~ 0.5*x^2 (very different shapes)
y_multi = np.column_stack([np.sin(X[:,0]) + rng.normal(0, 0.1, 120),
                           0.5 * X[:,0]**2 + rng.normal(0, 0.3, 120)])

Xtr, Xte, ytr, yte = train_test_split(X, y_multi, test_size=0.3, random_state=9)

# MultiOutputRegressor fits one independent model per output target
mor = MultiOutputRegressor(LinearRegression())
mor.fit(Xtr, ytr)
pred = mor.predict(Xte)

# r2_score with multioutput='raw_values' gives one R^2 per output target
per_target = r2_score(yte, pred, multioutput='raw_values')
print("Per-target R^2 on test set:", ["%.3f" % v for v in per_target])
print("Average R^2             : %.3f" % mor.score(Xte, yte))

# Plot both true targets and predictions
order = np.argsort(Xte[:,0])
plt.figure(figsize=(7,4))
plt.scatter(Xte[order,0], yte[order,0], s=20, label='true out 1 (sin)')
plt.plot(Xte[order,0], pred[order,0], 'r-', lw=1.5, label='pred out 1')
plt.scatter(Xte[order,0], yte[order,1], s=20, marker='s', label='true out 2 (0.5x^2)')
plt.plot(Xte[order,0], pred[order,1], 'g-', lw=1.5, label='pred out 2')
plt.title("Multiple outputs from one input")
plt.xlabel("x"); plt.ylabel("target values"); plt.legend(); plt.tight_layout(); plt.show()

Per-target R^2 on test set: ['0.717', '-0.026']
Average R^2             : 0.346

8.7 Transfer Learning

This is where neural networks shine brightest. Transfer learning means: take a model trained on one dataset, and adapt it to a different but related problem.

The classic recipe (for neural networks):

1. Train a deep model on a big "source" dataset (e.g. wild animals).
2. Gather a much smaller labeled dataset for your real "target" problem (e.g. domestic animals).
3. Remove the last layer(s) of the source model (the task-specific head — usually everything after the embedding layer).
4. Add fresh layers suited to your new problem.
5. Freeze the old layers (don't update their weights).
6. Train only the new layers on your small dataset with gradient descent.

The old layers have already learned generic, reusable features (edges, textures, shapes); you're just bolting a new "decision head" on top. This lets you get great results from surprisingly little labeled data.

Transfer learning isn't limited to reusing whole networks. A simpler, shallow version of the same idea: fit a transformer (like PCA or a scaler) on one dataset, then reuse its learned .transform() on a different dataset. Let's see that lightweight version.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# "Source" problem: a BIG labeled dataset (wild-animal analogue)
X_src, y_src = make_classification(n_samples=2000, n_features=20, n_informative=10,
                                   n_redundant=5, random_state=11)
# "Target" problem: a SMALL labeled dataset, different distribution (domestic-animal analogue)
X_tgt, y_tgt = make_classification(n_samples=150, n_features=20, n_informative=10,
                                   n_redundant=5, flip_y=0.05, random_state=99)

Xtr_t, Xte_t, ytr_t, yte_t = train_test_split(X_tgt, y_tgt, test_size=0.3,
                                              random_state=99, stratify=y_tgt)

# --- Baseline: train from scratch on the small target data only ---
from sklearn.pipeline import make_pipeline
scratch = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scratch.fit(Xtr_t, ytr_t)
print("From scratch (small data only) : test acc %.3f" % scratch.score(Xte_t, yte_t))

# --- Transfer: fit PCA on the BIG source data, REUSE it on the target data ---
# We "freeze" the PCA (fit on source) and only train the classifier on target.
pca = PCA(n_components=8, random_state=11).fit(StandardScaler().fit_transform(X_src))

scaler_t = StandardScaler().fit(Xtr_t)
Xtr_t_pca = pca.transform(scaler_t.transform(Xtr_t))
Xte_t_pca = pca.transform(scaler_t.transform(Xte_t))
transfer_clf = LogisticRegression(max_iter=1000).fit(Xtr_t_pca, ytr_t)
print("Transfer (PCA from source data) : test acc %.3f" % transfer_clf.score(Xte_t_pca, yte_t))

From scratch (small data only) : test acc 0.689
Transfer (PCA from source data) : test acc 0.622

Reading the result

The PCA was trained on the large source dataset, so it found good low-dimensional directions (features). We then reused those directions to compress the small target dataset before training a fresh classifier — a miniature of "freeze the feature extractor, retrain the head." Depending on how related the two problems are, this can match or beat training from scratch on limited data.

In real transfer learning with deep nets you'd reuse a whole pretrained network (downloadable online) instead of just a PCA, but the philosophy is identical: don't throw away learned features — repurpose them.

8.8 Algorithmic Efficiency

Not every algorithm that works is practical. As your dataset grows, some algorithms slow down gracefully and some blow up. Big-O notation describes how running time (or memory) grows as the input size N grows, ignoring constant factors:

• O(N) — time grows roughly linearly with N. Double the data, double the time.
• O(N squared) — time grows with the square of N. Double the data, four times the time.
• O(log N) — time grows with the logarithm of N. Double the data, add just one extra step. Very fast.
• O(N log N) — between linear and quadratic; the sweet spot for sorting-like problems.

An algorithm is usually called efficient if its complexity is polynomial in N (so O(N), O(N squared), O(N cubed) all count). But in the big-data era, even O(N squared) can be too slow, and people hunt for O(N log N) or O(log N) solutions.

Everyday analogy: looking up a word in a dictionary. Flipping every page one-by-one is O(N). Flipping to the middle and throwing away the wrong half each time (binary search) is O(log N) — a 1,000-page dictionary takes about 10 flips instead of up to 1,000.

Let's feel the difference with a tiny timing experiment.

import time

# Two ways to find the two most distant 1-D numbers in a list S of size N.
def find_max_distance_slow(S):           # O(N^2): compare every pair
    best = 0; result = None
    for x1 in S:
        for x2 in S:
            d = abs(x1 - x2)
            if d >= best:
                best = d; result = (x1, x2)
    return result

def find_max_distance_fast(S):           # O(N): just track min and max
    mn = float('inf'); mx = float('-inf')
    for x in S:
        if x < mn: mn = x
        if x > mx: mx = x
    return (mx, mn)

sizes = [500, 1000, 2000, 4000]
slow_t, fast_t = [], []
rng = np.random.RandomState(0)
for N in sizes:
    S = rng.normal(size=N)
    t0 = time.perf_counter(); find_max_distance_slow(S); slow_t.append(time.perf_counter()-t0)
    t0 = time.perf_counter(); find_max_distance_fast(S); fast_t.append(time.perf_counter()-t0)

plt.figure(figsize=(6,4))
plt.plot(sizes, slow_t, 'o-', label='O(N^2) slow')
plt.plot(sizes, fast_t, 's-', label='O(N) fast')
plt.xlabel("input size N"); plt.ylabel("time (seconds)")
plt.title("Big-O in action: same answer, very different speed")
plt.legend(); plt.tight_layout(); plt.show()
print("Slow times:", ["%.4fs" % t for t in slow_t])
print("Fast times:", ["%.4fs" % t for t in fast_t])

Slow times: ['1.6167s', '3.6168s', '8.9517s', '45.3968s']
Fast times: ['0.0011s', '0.0013s', '0.0010s', '0.0173s']

Practical efficiency tips

• Avoid Python loops for math — use numpy vector operations (numpy.dot(w, x)) instead of element-by-element loops.
• Pick the right data structure — use set for membership tests (fast) instead of list (slow); use dict for key-to-value lookups.
• Prefer battle-tested libraries (numpy, scipy, scikit-learn) — their core routines are written in C for speed.
• Use generators to stream huge collections one element at a time instead of loading them all into memory.
• Profile first with cProfile to find the actual bottleneck.
• When the algorithm itself can't be improved: parallelize with multiprocessing, or JIT-compile with Numba / PyPy.

Typical complexity of common ML algorithms (sklearn-style)

Algorithm	Training	Prediction (per sample)
k-NN	O(1) (just store data)	O(N * d) (scan all training points)
Logistic Regression	O(N * d * iters)	O(d)
Decision Tree	O(N * d * log N)	O(depth)
Random Forest (T trees)	O(T * N * d * log N)	O(T * depth)
SVM (kernel)	O(N^2 to N^3)	O(#support vectors * d)
k-Means	O(N * k * d * iters)	O(k * d)

Here N = number of training examples, d = number of features, k = number of clusters, T = number of trees, and "iters" = iterations of the optimizer. The takeaway: k-NN trains instantly but predicts slowly; SVMs train expensively; linear models stay cheap everywhere.

Key Takeaways

• Imbalanced data breaks accuracy. Use balanced accuracy / F1 to see the minority class, and fix it with class_weight='balanced', resampling, or threshold tuning.
• Combining uncorrelated models (averaging, voting, stacking) often beats any single model — diversity is the secret ingredient.
• Neural-network training dials: pick a learning-rate schedule, mind your batch size, and use early stopping as your first line of defense against overfitting.
• Advanced regularization = dropout (randomly silence neurons), batch normalization (standardize between layers), and data augmentation (invent label-preserving training examples).
• Multiple inputs: vectorize each modality and concatenate, or (for neural nets) build a subnetwork per input and merge their embeddings.
• Multiple outputs: branch several heads off one embedding; combine their costs as alpha*C1 + (1-alpha)*C2 and tune alpha.
• Transfer learning: reuse a pretrained model's features and retrain only a new head — a huge saver of labeled data, and a unique strength of deep nets.
• Big-O thinking: prefer O(N) or O(N log N) over O(N squared); vectorize, choose smart data structures, and profile before you optimize.

What's Next

We've spent two chapters on practical supervised learning. Next, in Chapter 9 — Unsupervised Learning, we drop the labels entirely and ask a different question: what structure hides in the data on its own? We'll meet clustering, dimensionality reduction, and anomaly detection.

Exercises

These exercises cover advanced practice from Chapter 8: imbalanced data, model combination, neural-network training, advanced regularization, multi-input/multi-output models, transfer learning, and algorithmic efficiency. Try each before reading the hint.

1. (Conceptual) Why can a 99% accuracy score be terrible news on a fraud dataset, and which metric would you report instead? Hint: the majority class drowns out the rare fraud class; use balanced accuracy or the F1 (precision/recall) of the minority class.
2. (Conceptual) List three standard techniques for handling imbalanced data. Hint: class_weight='balanced', resampling (oversample minority / undersample majority), and threshold tuning.
3. (Conceptual) Why does combining several models tend to beat any single one, and what key property must the base models share? Hint: uncorrelated errors cancel when you average/vote; the models must make different mistakes (diversity).
4. (Conceptual) Contrast averaging, majority vote, and stacking as model-combination strategies. Hint: averaging = mean of scores; majority vote = most common label; stacking = a meta-learner that learns weights on the base models' predictions.
5. (Conceptual) Explain early stopping and why it is considered a cheap regularizer. Hint: watch a validation set after each epoch and stop the moment it stops improving; it quits before the net starts memorizing training noise.
6. (Conceptual) What is dropout, and how does it regularize a neural network? Hint: each training step randomly silences a fraction of neurons, so the network can't rely on any single neuron and spreads its knowledge around.
7. (Conceptual) Describe the combined cost used for multi-output networks and what the hyperparameter α controls. Hint: C = α·C1 + (1−α)·C2; α sets how much you care about output 1 (e.g. coordinates) versus output 2 (e.g. class label).
8. (Conceptual) Outline the transfer-learning recipe and explain why it saves labeled data. Hint: take a pretrained model, remove its task-specific head, freeze the old layers, and train only a fresh head on your small dataset; the old layers already learned reusable generic features.

Hands-On Coding Problems

9. (Coding) Imbalanced data: make a dataset with make_classification(weights=[0.95, 0.05]); fit LogisticRegression plain and again with class_weight='balanced'; print accuracy, balanced_accuracy_score, and f1_score for both. Hint: from sklearn.metrics import balanced_accuracy_score, f1_score.
10. (Coding) Threshold tuning: fit a logistic model on imbalanced data, get predict_proba, sweep thresholds from 0.1 to 0.9, and plot precision vs recall; print the threshold that maximizes F1. Hint: for each threshold, compute pred = (proba >= t), then precision_score/recall_score/f1_score.
11. (Coding) Voting ensemble: build a soft VotingClassifier from logistic regression, random forest, and k-NN on make_moons; print each base model's test accuracy and the ensemble's. Hint: VotingClassifier(estimators=[...], voting='soft').
12. (Coding) Early stopping: train two MLPClassifiers (one with early_stopping=True, one without) on make_moons; print the number of iterations each ran and the test accuracy. Hint: n_iter_ holds the epoch count; n_iter_no_change=10 controls patience.
13. (Coding) Data augmentation: take make_moons, then add jittered copies (X + np.random.normal(0, 0.05, X.shape), same labels) to the training set; train a KNeighborsClassifier on the original vs the augmented training set and compare test accuracies. Hint: keep the test set untouched; only augment the training split.
14. (Coding) Multi-output regression: create make_regression(n_targets=2) and fit LinearRegression directly on the 2-column target Y; print the coefficients for each output and a prediction for one test example. Hint: sklearn's LinearRegression supports multi-output targets natively.
15. (Coding) Transfer learning (lite): fit PCA(n_components=2) on a large source make_classification dataset, then use it to .transform a small target dataset; train LogisticRegression on the PCA features vs the raw features and compare test accuracies. Hint: reuse the source-fitted PCA's transform on the target data — don't refit it.

# Exercise 9: imbalanced data -- plain vs class_weight='balanced'
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, balanced_accuracy_score, f1_score

X, y = make_classification(n_samples=1000, weights=[0.95, 0.05], random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

for cw in [None, "balanced"]:
    # TODO: fit LogisticRegression(class_weight=cw, max_iter=500) and evaluate
    acc = 0.0; bal = 0.0; f1 = 0.0  # placeholders
    print(f"class_weight={cw}: acc={acc:.3f}, balanced_acc={bal:.3f}, f1={f1:.3f}")

# Exercise 10: threshold tuning for F1
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score, f1_score

X, y = make_classification(n_samples=1000, weights=[0.95, 0.05], random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
clf = LogisticRegression(max_iter=500).fit(Xtr, ytr)
proba = clf.predict_proba(Xte)[:, 1]

thresholds = np.linspace(0.1, 0.9, 17)
precisions, recalls, f1s = [], [], []
for t in thresholds:
    # TODO: pred = (proba >= t).astype(int); compute precision, recall, f1
    precisions.append(0.0); recalls.append(0.0); f1s.append(0.0)  # placeholders

best_t = 0.5  # TODO: thresholds[int(np.argmax(f1s))]
print("best threshold by F1:", best_t)
# TODO: plot precision and recall vs threshold
plt.show()

# Exercise 11: soft voting ensemble
from sklearn.ensemble import VotingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=300, noise=0.25, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

base = [
    ("log", LogisticRegression(max_iter=500)),
    ("rf", RandomForestClassifier(n_estimators=50, random_state=0)),
    ("knn", KNeighborsClassifier(n_neighbors=5)),
]
# TODO: build VotingClassifier(estimators=base, voting='soft'), fit, and print
#       each base model's test accuracy plus the ensemble's
print("ensemble test accuracy:", None)  # placeholder

# Exercise 12: early stopping with an MLP
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=400, noise=0.25, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

for es in [False, True]:
    # TODO: fit MLPClassifier(hidden_layer_sizes=(20,20), max_iter=500,
    #       early_stopping=es, n_iter_no_change=10, random_state=0)
    n_iters = 0  # placeholder: clf.n_iter_
    test_acc = 0.0  # placeholder: clf.score(Xte, yte)
    print(f"early_stopping={es}: epochs={n_iters}, test_acc={test_acc:.3f}")

# Exercise 13: data augmentation with jittered copies
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X, y = make_moons(n_samples=300, noise=0.20, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

# Baseline: KNN on the original training set
base_acc = KNeighborsClassifier(n_neighbors=5).fit(Xtr, ytr).score(Xte, yte)

# TODO: build augmented training set: Xtr plus jittered copies (same labels)
X_aug = Xtr  # placeholder: np.vstack([Xtr, Xtr + np.random.normal(0, 0.05, Xtr.shape)])
y_aug = ytr  # placeholder: np.concatenate([ytr, ytr])

aug_acc = 0.0  # placeholder: KNeighborsClassifier(n_neighbors=5).fit(X_aug, y_aug).score(Xte, yte)
print("baseline test acc:", base_acc)
print("augmented test acc:", aug_acc)

# Exercise 14: multi-output regression
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

X, Y = make_regression(n_samples=200, n_features=4, n_targets=2, noise=5.0, random_state=0)
Xtr, Xte, Ytr, Yte = train_test_split(X, Y, test_size=0.3, random_state=0)

# TODO: fit LinearRegression() on Xtr/Ytr (multi-output), print coef_ shape and predict Xte[0]
reg = None  # placeholder

print("coef_ shape:", None)
print("prediction for Xte[0]:", None)

# Exercise 15: transfer learning (lite) with PCA
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression

# Large "source" dataset and small "target" dataset
X_src, _ = make_classification(n_samples=2000, n_features=10, n_informative=6, random_state=1)
X_tgt, y_tgt = make_classification(n_samples=150, n_features=10, n_informative=6, random_state=2)
Xtr, Xte, ytr, yte = train_test_split(X_tgt, y_tgt, test_size=0.3, random_state=2)

# TODO: fit PCA(n_components=2) on X_src, then .transform Xtr/Xte (reuse -- do NOT refit)
pca = None  # placeholder
Xtr_pca = Xtr  # placeholder: pca.transform(Xtr)
Xte_pca = Xte  # placeholder: pca.transform(Xte)

# TODO: compare LogisticRegression on raw features vs PCA features
raw_acc = 0.0   # placeholder
pca_acc = 0.0   # placeholder
print("raw test acc :", raw_acc)
print("pca test acc :", pca_acc)