Chapter 16 — Regression Analysis

Regression predicts a continuous target from features. This chapter covers linear regression, evaluation metrics, polynomial features, and regularization (Ridge/Lasso) to combat overfitting.

Learning Objectives

Prerequisites / Imports

In [1]:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

1 The Diabetes Dataset

10 physiological features; target is a disease progression measure.

In [1]:
data = load_diabetes(as_frame=True)
X, y = data.data, data.target
print('shape:', X.shape)
X.head()

shape: (442, 10)
age sex bmi bp s1 s2 s3 s4 s5 s6
0 0.038076 0.050680 0.061696 0.021872 -0.044223 -0.034821 -0.043401 -0.002592 0.019907 -0.017646
1 -0.001882 -0.044642 -0.051474 -0.026328 -0.008449 -0.019163 0.074412 -0.039493 -0.068332 -0.092204
2 0.085299 0.050680 0.044451 -0.005670 -0.045599 -0.034194 -0.032356 -0.002592 0.002861 -0.025930
3 -0.089063 -0.044642 -0.011595 -0.036656 0.012191 0.024991 -0.036038 0.034309 0.022688 -0.009362
4 0.005383 -0.044642 -0.036385 0.021872 0.003935 0.015596 0.008142 -0.002592 -0.031988 -0.046641

2 Simple Linear Regression

Fit $y = w_0 + w_1 x_1 + \dots + w_p x_p$ by ordinary least squares.

In [1]:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
lr = LinearRegression()
lr.fit(X_train, y_train)
print('intercept:', round(lr.intercept_, 2))
print('coefficients:', np.round(lr.coef_, 2))

intercept: 151.35
coefficients: [  37.9  -241.96  542.43  347.7  -931.49  518.06  163.42  275.32  736.2
   48.67]

3 Regression Metrics

MSE, RMSE, MAE (in target units), and $R^2$ (fraction of variance explained).

In [1]:
y_pred = lr.predict(X_test)
print('MSE :', round(mean_squared_error(y_test, y_pred), 2))
print('RMSE:', np.sqrt(mean_squared_error(y_test, y_pred)).round(2))
print('MAE :', round(mean_absolute_error(y_test, y_pred), 2))
print('R2  :', round(r2_score(y_test, y_pred), 3))

MSE : 2900.19
RMSE: 53.85
MAE : 42.79
R2  : 0.453

4 Predicted vs Actual

A good regression has points clustered around the diagonal.

In [1]:
plt.figure(figsize=(6,6))
plt.scatter(y_test, y_pred, alpha=0.6)
lims = [y_test.min(), y_test.max()]
plt.plot(lims, lims, 'r--')
plt.xlabel('actual'); plt.ylabel('predicted'); plt.title('Predicted vs actual')
plt.show()

5 Polynomial Regression

Add polynomial features to capture curvature. Beware: higher degrees overfit.

In [1]:
rng = np.random.default_rng(0)
x = np.sort(rng.uniform(0, 6, 40))
y_true = np.sin(x)
y = y_true + rng.normal(0, 0.2, size=len(x))
X = x.reshape(-1, 1)

for deg in [1, 3, 9]:
    pipe = Pipeline([('poly', PolynomialFeatures(degree=deg)), ('lr', LinearRegression())])
    pipe.fit(X, y)
    xs = np.linspace(0, 6, 100).reshape(-1, 1)
    plt.plot(xs, pipe.predict(xs), label=f'degree {deg}')
plt.scatter(x, y, color='black', alpha=0.6, label='data')
plt.plot(x, y_true, 'g--', label='true')
plt.legend(); plt.title('Polynomial regression: under/overfitting'); plt.show()

6 Regularization: Ridge and Lasso

Ridge (L2) shrinks coefficients; Lasso (L1) can zero them out (feature selection). Scaling is essential before regularization.

In [1]:
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2, random_state=42)
ridge = Pipeline([('scaler', StandardScaler()), ('model', Ridge(alpha=1.0))]).fit(X_train, y_train)
lasso = Pipeline([('scaler', StandardScaler()), ('model', Lasso(alpha=0.1))]).fit(X_train, y_train)
print('Ridge R2 :', round(ridge.score(X_test, y_test), 3))
print('Lasso R2 :', round(lasso.score(X_test, y_test), 3))
print('Lasso nonzero coefs:', int(np.sum(lasso.named_steps['model'].coef_ != 0)))
print('Lasso coefs:', np.round(lasso.named_steps['model'].coef_, 2))

Ridge R2 : 0.454
Lasso R2 : 0.456
Lasso nonzero coefs: 10
Lasso coefs: [  1.73 -11.32  25.82  16.64 -29.36  13.28   0.55  10.24  29.63   2.39]

7 The Regularization Strength

Larger $\alpha$ shrinks coefficients more. Cross-validation picks a good value.

In [1]:
alphas = [0.001, 0.01, 0.1, 1, 10, 100]
ridge_scores = []
for a in alphas:
    p = Pipeline([('scaler', StandardScaler()), ('model', Ridge(alpha=a))])
    ridge_scores.append(cross_val_score(p, data.data, data.target, cv=5).mean())
plt.figure(figsize=(7,4))
plt.plot(alphas, ridge_scores, marker='o')
plt.xscale('log'); plt.title('Ridge CV R2 vs alpha'); plt.xlabel('alpha'); plt.ylabel('CV R2')
plt.show()

Case Study: Predicting Disease Progression

Compare linear, Ridge, and Lasso with 5-fold cross-validation and report the winner.

In [1]:
models = {
    'Linear': Pipeline([('scaler', StandardScaler()), ('model', LinearRegression())]),
    'Ridge': Pipeline([('scaler', StandardScaler()), ('model', Ridge(alpha=0.1))]),
    'Lasso': Pipeline([('scaler', StandardScaler()), ('model', Lasso(alpha=0.1))]),
}
for name, m in models.items():
    scores = cross_val_score(m, data.data, data.target, cv=5)
    print(f'{name:7s} CV R2: {scores.mean():.3f} +/- {scores.std():.3f}')

Linear  CV R2: 0.482 +/- 0.049
Ridge   CV R2: 0.482 +/- 0.049
Lasso   CV R2: 0.482 +/- 0.048

Exercises

  1. Fit a linear regression on the diabetes data and report $R^2$.
  2. Compute RMSE and MAE for the predictions.
  3. Make a predicted-vs-actual scatter with a diagonal reference line.
  4. Fit polynomial models of degree 1, 3, and 10 on synthetic data and observe overfitting.
  5. Explain why scaling matters before Ridge/Lasso.
  6. Compare Ridge and Lasso $R^2$ on the diabetes data.
  7. Use cross-validation to select the best Ridge alpha.
  8. Count how many coefficients Lasso sets to zero for a large alpha.
  9. Explain the bias-variance tradeoff in one paragraph.
  10. State when you would prefer Lasso over Ridge.

Python Data Science: From Foundations to Applications — Chapter 16