Chapter 17 — Classification

Classification assigns inputs to discrete categories. This chapter builds and evaluates several classifiers — logistic regression, k-NN, and decision trees — and introduces the key metrics and visualizations for classification performance.

Learning Objectives

Prerequisites / Imports

In [1]:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer, make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.pipeline import Pipeline
from sklearn.metrics import (accuracy_score, classification_report, ConfusionMatrixDisplay)

1 The Breast Cancer Dataset

Binary classification: malignant vs benign from 30 cell-nucleus features.

In [1]:
cancer = load_breast_cancer(as_frame=True)
X, y = cancer.data, cancer.target
print('shape:', X.shape, 'classes:', cancer.target_names.tolist())
print('class counts:', np.bincount(y))
X.head()

shape: (569, 30) classes: ['malignant', 'benign']
class counts: [212 357]
mean radius mean texture mean perimeter mean area mean smoothness mean compactness mean concavity mean concave points mean symmetry mean fractal dimension radius error texture error perimeter error area error smoothness error compactness error concavity error concave points error symmetry error fractal dimension error worst radius worst texture worst perimeter worst area worst smoothness worst compactness worst concavity worst concave points worst symmetry worst fractal dimension
0 17.99 10.38 122.80 1001.0 0.11840 0.27760 0.3001 0.14710 0.2419 0.07871 1.0950 0.9053 8.589 153.40 0.006399 0.04904 0.05373 0.01587 0.03003 0.006193 25.38 17.33 184.60 2019.0 0.1622 0.6656 0.7119 0.2654 0.4601 0.11890
1 20.57 17.77 132.90 1326.0 0.08474 0.07864 0.0869 0.07017 0.1812 0.05667 0.5435 0.7339 3.398 74.08 0.005225 0.01308 0.01860 0.01340 0.01389 0.003532 24.99 23.41 158.80 1956.0 0.1238 0.1866 0.2416 0.1860 0.2750 0.08902
2 19.69 21.25 130.00 1203.0 0.10960 0.15990 0.1974 0.12790 0.2069 0.05999 0.7456 0.7869 4.585 94.03 0.006150 0.04006 0.03832 0.02058 0.02250 0.004571 23.57 25.53 152.50 1709.0 0.1444 0.4245 0.4504 0.2430 0.3613 0.08758
3 11.42 20.38 77.58 386.1 0.14250 0.28390 0.2414 0.10520 0.2597 0.09744 0.4956 1.1560 3.445 27.23 0.009110 0.07458 0.05661 0.01867 0.05963 0.009208 14.91 26.50 98.87 567.7 0.2098 0.8663 0.6869 0.2575 0.6638 0.17300
4 20.29 14.34 135.10 1297.0 0.10030 0.13280 0.1980 0.10430 0.1809 0.05883 0.7572 0.7813 5.438 94.44 0.011490 0.02461 0.05688 0.01885 0.01756 0.005115 22.54 16.67 152.20 1575.0 0.1374 0.2050 0.4000 0.1625 0.2364 0.07678

2 Train/Test Split and Baseline

Always establish a train/test split and a simple baseline before tuning.

In [1]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
print('majority-class baseline accuracy:', round(max(np.bincount(y_test))/len(y_test), 3))

majority-class baseline accuracy: 0.626

3 Logistic Regression

A linear, probabilistic classifier. Scaling improves convergence and performance.

In [1]:
pipe = Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression(max_iter=5000))])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
print('accuracy:', round(accuracy_score(y_test, y_pred), 3))
print(classification_report(y_test, y_pred, target_names=cancer.target_names))

accuracy: 0.988
              precision    recall  f1-score   support

   malignant       0.98      0.98      0.98        64
      benign       0.99      0.99      0.99       107

    accuracy                           0.99       171
   macro avg       0.99      0.99      0.99       171
weighted avg       0.99      0.99      0.99       171

4 Confusion Matrix

Shows true/false positives and negatives for each class.

In [1]:
ConfusionMatrixDisplay.from_predictions(y_test, y_pred, display_labels=cancer.target_names, cmap='Blues')
plt.title('Confusion matrix (logistic regression)')
plt.show()

5 Comparing Classifiers

Benchmark several models with the same split.

In [1]:
models = {
    'LogisticRegression': Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression(max_iter=5000))]),
    'KNN': Pipeline([('scaler', StandardScaler()), ('clf', KNeighborsClassifier())]),
    'DecisionTree': DecisionTreeClassifier(random_state=42),
}
for name, m in models.items():
    m.fit(X_train, y_train)
    print(f'{name:18s} accuracy: {m.score(X_test, y_test):.3f}')

LogisticRegression accuracy: 0.988
KNN                accuracy: 0.959
DecisionTree       accuracy: 0.918

6 Decision Boundary

Train on two synthetic features so we can visualize the boundary each model learns.

In [1]:
Xb, yb = make_classification(n_samples=300, n_features=2, n_redundant=0, n_clusters_per_class=1, random_state=7)
Xtr, Xte, ytr, yte = train_test_split(Xb, yb, test_size=0.3, random_state=7)
fig, axes = plt.subplots(1, 3, figsize=(14,4))
for ax, (name, model) in zip(axes, [('k-NN', KNeighborsClassifier(7)), ('Logistic', LogisticRegression()), ('Tree', DecisionTreeClassifier(random_state=7))]):
    model.fit(Xtr, ytr)
    x_min, x_max = Xb[:,0].min()-1, Xb[:,0].max()+1
    y_min, y_max = Xb[:,1].min()-1, Xb[:,1].max()+1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.05), np.arange(y_min, y_max, 0.05))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.3, cmap='coolwarm')
    ax.scatter(Xtr[:,0], Xtr[:,1], c=ytr, edgecolor='k', cmap='coolwarm')
    ax.set_title(f'{name} (acc={model.score(Xte,yte):.2f})')
plt.tight_layout(); plt.show()

7 Precision, Recall, and the Tradeoff

Precision: of predicted positives, how many are correct. Recall: of actual positives, how many we caught. In medicine, recall (sensitivity) often matters most.

Case Study: Tumor Diagnosis with Cost-Sensitive Recall

In cancer screening, missing a malignant case (false negative) is costly. We weight the malignant class more heavily to raise recall.

In [1]:
pipe = Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression(max_iter=5000, class_weight='balanced'))])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
print('accuracy:', round(accuracy_score(y_test, y_pred), 3))
print(classification_report(y_test, y_pred, target_names=cancer.target_names))
ConfusionMatrixDisplay.from_predictions(y_test, y_pred, display_labels=cancer.target_names, cmap='Greens')
plt.title('Balanced logistic regression'); plt.show()

accuracy: 0.971
              precision    recall  f1-score   support

   malignant       0.94      0.98      0.96        64
      benign       0.99      0.96      0.98       107

    accuracy                           0.97       171
   macro avg       0.97      0.97      0.97       171
weighted avg       0.97      0.97      0.97       171

Exercises

  1. Load the breast cancer data and report the class distribution.
  2. Train a logistic regression pipeline and print accuracy.
  3. Display a confusion matrix for the predictions.
  4. Compare accuracy of logistic regression, k-NN, and a decision tree.
  5. Plot decision boundaries for three models on 2-D synthetic data.
  6. Explain why scaling matters for k-NN but not for a decision tree.
  7. Compute precision and recall for the malignant class.
  8. Explain when you would prioritize recall over precision.
  9. Use classification_report and identify the class with lower recall.
  10. Train a classifier on the wine dataset and report accuracy.

Python Data Science: From Foundations to Applications — Chapter 17