Chapter 15 — Introduction to Machine Learning with Scikit-Learn

Machine learning builds models that learn patterns from data. Scikit-Learn provides a uniform, well-documented API for classical ML. This chapter introduces the landscape and the estimator interface, then trains and evaluates a first model end-to-end.

Learning Objectives

Prerequisites / Imports

In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score, classification_report

1 What Is Machine Learning?

ML learns a mapping from inputs $X$ to outputs $y$ (supervised) or structure in $X$ (unsupervised) from data, then generalizes to new data.

2 The Estimator API

Every model is an object with the same interface: fit(X, y) to train, predict(X) to infer, and score(X, y) to evaluate. Preprocessors add transform(X).

3 The Iris Dataset

A classic benchmark: 150 iris flowers, 4 measurements, 3 species.

In [1]:
iris = load_iris(as_frame=True)
X = iris.data
y = iris.target
print('feature names:', list(iris.feature_names))
print('target names:', list(iris.target_names))
print('shape:', X.shape)
X.head()

feature names: ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
target names: ['setosa', 'versicolor', 'virginica']
shape: (150, 4)
sepal length (cm) sepal width (cm) petal length (cm) petal width (cm)
0 5.1 3.5 1.4 0.2
1 4.9 3.0 1.4 0.2
2 4.7 3.2 1.3 0.2
3 4.6 3.1 1.5 0.2
4 5.0 3.6 1.4 0.2

4 Train/Test Split

Hold out a portion of data to evaluate generalization. Always split before scaling or training to avoid data leakage.

In [1]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
print('train:', X_train.shape, 'test:', X_test.shape)

train: (105, 4) test: (45, 4)

5 Feature Scaling

Distance-based models (k-NN, SVM, k-means) need features on the same scale. StandardScaler standardizes to zero mean, unit variance.

In [1]:
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)   # use train statistics!
print('scaled train mean (should be ~0):', X_train_scaled.mean(axis=0).round(3))
print('scaled train std  (should be ~1):', X_train_scaled.std(axis=0).round(3))

scaled train mean (should be ~0): [ 0. -0. -0. -0.]
scaled train std  (should be ~1): [1. 1. 1. 1.]

6 A First Model: k-Nearest Neighbors

k-NN classifies a point by the majority label among its $k$ nearest neighbors.

In [1]:
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)
y_pred = knn.predict(X_test_scaled)
print('accuracy:', round(accuracy_score(y_test, y_pred), 3))
print(classification_report(y_test, y_pred, target_names=iris.target_names))

accuracy: 0.911
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        15
  versicolor       0.79      1.00      0.88        15
   virginica       1.00      0.73      0.85        15

    accuracy                           0.91        45
   macro avg       0.93      0.91      0.91        45
weighted avg       0.93      0.91      0.91        45

7 Pipelines Prevent Leakage

A Pipeline bundles scaling + modeling so the test set is never used to fit the scaler.

In [1]:
pipe = Pipeline([('scaler', StandardScaler()), ('knn', KNeighborsClassifier(n_neighbors=5))])
pipe.fit(X_train, y_train)
print('pipeline test accuracy:', round(pipe.score(X_test, y_test), 3))

pipeline test accuracy: 0.911

8 Choosing k

Compare accuracy across values of $k$ to select a good one.

In [1]:
ks = range(1, 21)
accs = []
for k in ks:
    p = Pipeline([('scaler', StandardScaler()), ('knn', KNeighborsClassifier(n_neighbors=k))])
    p.fit(X_train, y_train)
    accs.append(p.score(X_test, y_test))

plt.figure(figsize=(7,4))
plt.plot(ks, accs, marker='o')
plt.title('k-NN accuracy vs k'); plt.xlabel('k'); plt.ylabel('test accuracy')
plt.show()
print('best k:', ks[int(np.argmax(accs))], 'accuracy:', round(max(accs), 3))

best k: 9 accuracy: 0.956

Case Study: Predicting Iris Species

Put it all together: load, split, pipeline, train, evaluate, and inspect a few predictions.

In [1]:
pipe = Pipeline([('scaler', StandardScaler()), ('knn', KNeighborsClassifier(n_neighbors=7))])
pipe.fit(X_train, y_train)
sample = X_test.iloc[:5]
print('predictions:', pipe.predict(sample))
print('true labels:', y_test.iloc[:5].values)
print('overall accuracy:', round(pipe.score(X_test, y_test), 3))
result = pd.DataFrame({'predicted': iris.target_names[pipe.predict(sample)], 'true': iris.target_names[y_test.iloc[:5].values]})
result

predictions: [2 1 1 1 2]
true labels: [2 1 2 1 2]
overall accuracy: 0.933
predicted true
0 virginica virginica
1 versicolor versicolor
2 versicolor virginica
3 versicolor versicolor
4 virginica virginica

Exercises

  1. List three differences between supervised and unsupervised learning.
  2. Split the iris data 80/20 with stratification and print the shapes.
  3. Explain why fit_transform is used on the training set but transform on the test set.
  4. Train a k-NN with $k=1$ and $k=15$ and compare accuracies.
  5. Build a Pipeline with StandardScaler and KNeighborsClassifier.
  6. Plot test accuracy vs $k$ for $k=1\dots25$.
  7. Print a classification report and identify which class has the lowest F1-score.
  8. Why is data leakage a problem when scaling before splitting?
  9. Replace k-NN with LogisticRegression in the pipeline and compare.
  10. Describe the estimator API in two sentences.

Python Data Science: From Foundations to Applications — Chapter 15