Chapter 18 — Unsupervised Learning and Clustering

Without labels, we seek structure. Clustering groups similar items; dimensionality reduction projects high-dimensional data to a readable space. This chapter covers k-means, silhouette evaluation, and PCA.

Learning Objectives

Apply k-means clustering.
Choose $k$ with the elbow method and silhouette score.
Scale data before clustering.
Use PCA for dimensionality reduction and visualization.
Interpret cluster centers and explained variance.
Compare clustering vs classification.

Prerequisites / Imports

In [1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs, load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, confusion_matrix
from sklearn.decomposition import PCA

1 Synthetic Blobs

We generate 3 separable clusters to make the mechanics clear.

In [1]:

X, true_labels = make_blobs(n_samples=300, centers=3, cluster_std=0.9, random_state=42)
X = StandardScaler().fit_transform(X)
plt.figure(figsize=(6,5))
plt.scatter(X[:,0], X[:,1], c=true_labels, cmap='viridis', edgecolor='k')
plt.title('Synthetic blobs (true labels)'); plt.show()

2 k-Means Clustering

k-means finds $k$ centers that minimize within-cluster variance.

In [1]:

km = KMeans(n_clusters=3, n_init=10, random_state=42)
labels = km.fit_predict(X)
print('inertia:', round(km.inertia_, 2))
print('silhouette:', round(silhouette_score(X, labels), 3))
plt.figure(figsize=(6,5))
plt.scatter(X[:,0], X[:,1], c=labels, cmap='viridis', edgecolor='k')
plt.scatter(km.cluster_centers_[:,0], km.cluster_centers_[:,1], c='red', s=200, marker='X', label='centers')
plt.legend(); plt.title('k-means clusters'); plt.show()

inertia: 14.92
silhouette: 0.863

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.

3 Choosing k: Elbow and Silhouette

Plot inertia (lower is better) and silhouette (higher is better, $\le 1$) across $k$.

In [1]:

ks = range(2, 8)
inertias, sils = [], []
for k in ks:
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X)
    inertias.append(km.inertia_)
    sils.append(silhouette_score(X, km.labels_))
fig, ax = plt.subplots(1, 2, figsize=(11,4))
ax[0].plot(ks, inertias, marker='o'); ax[0].set_title('Elbow (inertia)'); ax[0].set_xlabel('k')
ax[1].plot(ks, sils, marker='o', color='green'); ax[1].set_title('Silhouette score'); ax[1].set_xlabel('k')
plt.tight_layout(); plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.

4 Clustering the Iris Data (No Labels)

Pretend we don't know species. Scale, cluster with $k=3$, and compare to true labels.

In [1]:

iris = load_iris(as_frame=True)
X = StandardScaler().fit_transform(iris.data)
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
print('silhouette:', round(silhouette_score(X, km.labels_), 3))
print(pd.DataFrame(confusion_matrix(iris.target, km.labels_)))

silhouette: 0.46
    0   1   2
0   0  50   0
1  39   0  11
2  14   0  36

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=1.

5 PCA for Visualization

Project the 4-D iris data to 2-D and color by species. PCA also reports explained variance.

In [1]:

pca = PCA(n_components=2)
X2 = pca.fit_transform(X)
print('explained variance ratio:', np.round(pca.explained_variance_ratio_, 3))
print('cumulative:', np.round(pca.explained_variance_ratio_.cumsum(), 3))
plt.figure(figsize=(7,5))
plt.scatter(X2[:,0], X2[:,1], c=iris.target, cmap='viridis', edgecolor='k')
plt.xlabel('PC1'); plt.ylabel('PC2'); plt.title('PCA of iris (2 components)'); plt.show()

explained variance ratio: [0.73  0.229]
cumulative: [0.73  0.958]

6 How Many Components?

Plot cumulative explained variance to choose how many PCs to keep.

In [1]:

pca_full = PCA().fit(X)
plt.figure(figsize=(7,4))
plt.plot(np.arange(1, len(pca_full.explained_variance_ratio_)+1),
         pca_full.explained_variance_ratio_.cumsum(), marker='o')
plt.axhline(0.95, color='r', linestyle='--', label='95% variance')
plt.title('Cumulative explained variance (iris)'); plt.xlabel('components'); plt.ylabel('cumulative variance')
plt.legend(); plt.show()

Case Study: Segmenting Customers

Synthetic customer spending data is clustered to identify segments, and PCA visualizes them.

In [1]:

rng = np.random.default_rng(0)
customers = np.vstack([rng.normal(50, 10, (60,2)), rng.normal(120, 15, (60,2)), rng.normal(200, 20, (60,2))])
customers = StandardScaler().fit_transform(customers)
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(customers)
print('silhouette:', round(silhouette_score(customers, km.labels_), 3))
plt.figure(figsize=(7,5))
plt.scatter(customers[:,0], customers[:,1], c=km.labels_, cmap='viridis', edgecolor='k')
plt.scatter(km.cluster_centers_[:,0], km.cluster_centers_[:,1], c='red', s=200, marker='X')
plt.title('Customer segments'); plt.xlabel('scaled income'); plt.ylabel('scaled spending'); plt.show()

silhouette: 0.732

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=1.

Exercises

Generate 4 blobs and cluster them with k-means.
Plot inertia vs $k$ for $k=1\dots8$ and locate the elbow.
Compute silhouette scores for $k=2\dots7$.
Explain why scaling matters before k-means.
Apply PCA to the iris data and keep 2 components.
Plot cumulative explained variance and find how many components reach 95%.
Cluster the wine dataset and compare clusters to true classes.
Explain the difference between clustering and classification.
What does a silhouette score near 0 indicate?
Describe one business use case for customer segmentation.

Python Data Science: From Foundations to Applications — Chapter 18