Chapter 6 — Neural Networks and Deep Learning

If you understood linear and logistic regression, you already understand the building block of a neural network. In this chapter we take that building block — the humble weighted-sum-plus-activation — stack it into layers, and see how the whole stack learns. Then we go "deep": we look at why adding more layers helps, and at the two special architectures that dominate modern deep learning, convolutional networks for images and recurrent networks for sequences.

In this chapter you will learn:

• What a neuron is (weighted sum + activation) and how neurons stack into layers
• How a feed-forward pass turns an input into a prediction
• The three most common activation functions: ReLU, sigmoid, and tanh
• How backpropagation trains a network using gradient descent
• How to choose layer sizes, and the idea of universal approximation
• What makes a network "deep" and why depth helps
• How convolutional neural networks (CNNs) process images
• How recurrent neural networks (RNNs) handle sequences

6.1 Neural Networks

Here is a nice surprise: you already know what a neural network is. Logistic regression — the model you saw in earlier chapters — is, in fact, a single neuron. Its generalization to multiple classes (softmax regression) is a standard unit inside a neural network.

A neural network, just like a regression or SVM model, is a mathematical function: y = f_NN(x). The special thing about f_NN is that it is a nested function — one function wrapped inside another, layer by layer. For a 3-layer network that outputs a single number, it looks like this:

y = f_NN(x) = f3(f2(f1(x)))

Each layer function has the same shape:

fl(z) = gl(Wl · z + bl)

where:

• Wl is a matrix of weights (the "knobs" the network learns),
• bl is a vector of biases (one per unit),
• gl is a fixed, usually nonlinear activation function chosen by you before training starts.

Compare that to logistic regression: it is exactly sigmoid(w·x + b). A neural network just chains several of these together, with each layer's output becoming the next layer's input.

Everyday analogy: Think of a neural network as an assembly line. Each workstation (layer) takes in some parts, does a small calculation (weighted sum plus a nonlinear squashing), and passes the result to the next workstation. The final workstation outputs the finished product — your prediction.

The Neuron: Weighted Sum + Activation

The smallest unit inside a neural network is a neuron (also called a unit). A neuron does two simple things:

1. Weighted sum: it takes its input vector z, multiplies each entry by a weight, adds them up, and adds a bias. In math: a = w·z + b. This is exactly what linear regression does.
2. Activation: it applies a nonlinear function g to that sum: output = g(a).

Why the nonlinear step? Without it, stacking many layers would be pointless — a linear function of a linear function is still a linear function, so a deep stack of purely-linear layers collapses into one boring linear model. The nonlinearity is what gives a neural network the power to model curved, complicated relationships.

In a multilayer perceptron (MLP) — the most common "vanilla" architecture — every unit in one layer sends its output to every unit in the next layer. That is called a fully-connected (or dense) layer.

Everyday analogy: A neuron is like a tiny voting committee member. Each input is a piece of evidence, the weight is how much the member trusts that evidence, the bias is the member's personal leaning, and the activation is how strongly they react to the total — maybe they stay silent unless the evidence is strong enough (ReLU), or they always give a measured "confidence" between 0 and 1 (sigmoid).

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# --- A tiny neural network by hand, in pure numpy ---
# Architecture: 2 inputs -> 3 hidden neurons (ReLU) -> 1 output (sigmoid)
# The whole network is just: y = sigmoid( W2 @ relu(W1 @ x + b1) + b2 )

def relu(z):
    return np.maximum(0, z)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

# Two input examples, each a 2-dimensional feature vector
X = np.array([[0.5, -1.0],
              [2.0,  0.5]])

# Layer 1 (hidden): W1 has shape (3 neurons, 2 inputs); b1 has 3 entries
W1 = np.array([[ 0.2,  0.8],
               [-0.5,  1.0],
               [ 0.9, -0.3]])
b1 = np.array([0.0, 0.1, -0.2])

# Layer 2 (output): W2 has shape (1 output, 3 hidden); b2 has 1 entry
W2 = np.array([[1.0, -0.5, 0.3]])
b2 = np.array([0.0])

# --- Feed-forward pass, step by step ---

# 1) Hidden layer: linear part (weighted sum), then ReLU activation
z1 = X @ W1.T + b1          # pre-activation: weighted sum for each hidden neuron
h1 = relu(z1)               # activation: squash negatives to zero
print("Hidden layer pre-activations (z1):")
print(z1)
print("Hidden layer activations (h1, after ReLU):")
print(h1)

# 2) Output layer: linear part, then sigmoid to get a probability
z2 = h1 @ W2.T + b2         # weighted sum for the output neuron
y_hat = sigmoid(z2)         # squash to (0, 1) -> a probability
print("\nOutput pre-activations (z2):")
print(z2)
print("Final predictions (y_hat, after sigmoid):")
print(y_hat)
print("\nInterpretation: y_hat > 0.5 -> predict class 1, else class 0.")

Hidden layer pre-activations (z1):
[[-0.7  -1.15  0.55]
 [ 0.8  -0.4   1.45]]
Hidden layer activations (h1, after ReLU):
[[0.   0.   0.55]
 [0.8  0.   1.45]]

Output pre-activations (z2):
[[0.165]
 [1.235]]
Final predictions (y_hat, after sigmoid):
[[0.54115667]
 [0.77469249]]

Interpretation: y_hat > 0.5 -> predict class 1, else class 0.

Reading the output

That little script did a complete feed-forward pass: data flowed from the input, through the hidden layer, to the output, with no learning involved — we just picked the weights by hand. Notice:

• The hidden layer's z1 is just a weighted sum; relu then zeros out the negative entries. Those zeros are the neuron "staying silent."
• The output neuron combines the three hidden signals and sigmoid squashes the result into a number between 0 and 1, which we can read as a probability.

In a real network the weights W1, b1, W2, b2 are not hand-picked — they are learned by gradient descent, which we get to shortly.

Activation Functions

The activation function g is the nonlinear "spark" inside each neuron. Three choices cover most of what you will see:

Function	Formula	Output range	Typical use
Sigmoid	1 / (1 + e^-z)	(0, 1)	output layer for binary classification (a probability)
Tanh	(e^z - e^-z) / (e^z + e^-z)	(-1, 1)	hidden layers (older networks); centered around 0
ReLU	max(0, z)	[0, infinity)	hidden layers (modern default); fast and simple

• Sigmoid squashes anything into a 0-to-1 probability. The downside: its gradient is tiny for very large or very negative inputs, which slows learning in deep networks.
• Tanh is shaped like a softer S, but ranges from -1 to 1. Being centered at zero often helps it learn faster than sigmoid.
• ReLU is the simplest: if the input is negative, output 0; otherwise, pass the input through. It is cheap to compute and, importantly, its gradient does not vanish for large positive inputs — which is a big reason deep networks became practical.

Let's plot all three so you can see their shapes.

# Plot the three most common activation functions side by side
z = np.linspace(-6, 6, 300)

fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))

axes[0].plot(z, sigmoid(z), color="tab:blue")
axes[0].set_title("Sigmoid")
axes[0].set_ylim(-1.2, 1.2); axes[0].axhline(0, color="gray", linewidth=0.8)

axes[1].plot(z, np.tanh(z), color="tab:orange")
axes[1].set_title("Tanh")
axes[1].set_ylim(-1.2, 1.2); axes[1].axhline(0, color="gray", linewidth=0.8)

axes[2].plot(z, relu(z), color="tab:green")
axes[2].set_title("ReLU")
axes[2].set_ylim(-1.2, 2.5); axes[2].axhline(0, color="gray", linewidth=0.8)

for ax in axes:
    ax.set_xlabel("input z")
    ax.set_ylabel("output g(z)")

plt.suptitle("Common activation functions", y=1.02)
plt.tight_layout()
plt.show()

Reading the plot

• Sigmoid flattens out at 0 on the far left and 1 on the far right — that flatness is exactly the vanishing gradient problem: where the curve is flat, the slope (gradient) is near zero, so learning barely happens.
• Tanh is the same S-shape but stretched to go from -1 to 1. Its steepest part is around zero, so it learns well when inputs are small.
• ReLU is a hockey stick: dead flat at zero for negative inputs, then a straight 45-degree ramp for positive inputs. That ramp has a constant gradient of 1, which is why ReLU keeps deep networks trainable.

Layers and the Feed-Forward Pass

In a multilayer perceptron, neurons are organized into layers:

• The input layer is just your feature vector x — it does no computation, it only holds the data.
• The hidden layers do the real work: each is a set of neurons that all receive the previous layer's outputs.
• The output layer produces the final prediction. For regression it uses a linear (identity) activation; for binary classification it uses sigmoid; for multiclass it uses softmax.

Because every neuron in layer l connects to every neuron in layer l+1, these are fully-connected layers. Passing data from input to output is the feed-forward pass — exactly what our numpy code did above, just generalized to any number of layers.

A handy way to picture it: the number of neurons in the output layer matches your task — one for regression or binary classification, k for k-class classification. The hidden layers can be any size you choose.

Backpropagation: How the Network Learns

So far our weights were fixed. In reality they are learned from data. The workhorse algorithm is backpropagation, and it is not a new kind of magic — it is gradient descent (which you already know) applied to the network's weights.

Here is the idea in plain words:

1. Forward pass: push a training example through the network and get a prediction.
2. Measure the error: compare the prediction to the true label y using a cost function (mean squared error for regression, cross-entropy for classification).
3. Backward pass: figure out how much each weight contributed to the error. This is where the chain rule from calculus comes in: because the network is a nesting of functions, you can work out the derivative of the cost with respect to any weight by multiplying the derivatives of the functions it sits inside. Backpropagation is just an efficient way to do that chain-rule bookkeeping from the output layer back to the input.
4. Update: nudge each weight a tiny step in the direction that reduces the error (gradient descent).

The name "backpropagation" comes from step 3: the error signal flows backward through the network, opposite to the forward data flow.

Everyday analogy: You bake a cake (forward pass), taste it, and it is too salty. You work backward: "the saltiness came from the salt I added at the seasoning step, which depended on the recipe I chose at the planning step." You adjust each earlier decision a little so the next cake is better. Repeat many times and the cake improves.

6.1.2 Choosing the Architecture

A practical question: how many layers, and how many neurons per layer?

• The output layer size is fixed by your problem: 1 for regression / binary classification, k for k-class classification.
• The hidden layers are your choice. A common starting point is one or two hidden layers with a moderate number of neurons (say 16 to 128). You tune this like any hyperparameter.
• Bigger is not always better: adding a 1000-neuron layer adds roughly one million parameters, which makes training slower and raises the risk of overfitting.

There is a famous theoretical result called the universal approximation theorem: a feed-forward network with just one hidden layer (and enough neurons, and a suitable nonlinearity) can approximate essentially any continuous function. In practice, though, a deep network with several smaller layers usually needs far fewer total parameters than one giant hidden layer to model complicated patterns — which is one reason depth is so popular.

from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons

# Build a classic non-linearly-separable dataset: two interleaving moons
X, y = make_moons(n_samples=200, noise=0.25, random_state=42)

# Train a small MLP: 2 hidden layers of 16 neurons each, ReLU activation
clf = MLPClassifier(hidden_layer_sizes=(16, 16), activation="relu",
                    solver="adam", max_iter=400, random_state=42)
clf.fit(X, y)

# Plot the data and the learned decision boundary
plt.scatter(X[:, 0][y == 0], X[:, 1][y == 0], color="tab:blue", label="class 0")
plt.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:orange", label="class 1")

# Build a grid and predict the class of every grid point -> decision boundary
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))
grid = np.c_[xx.ravel(), yy.ravel()]
zz = clf.predict(grid).reshape(xx.shape)
plt.contourf(xx, yy, zz, alpha=0.15, cmap="coolwarm")

plt.title("MLP on make_moons - accuracy {:.2f}".format(clf.score(X, y)))
plt.xlabel("feature 1"); plt.ylabel("feature 2")
plt.legend()
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\neural_network\_multilayer_perceptron.py:691: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (400) reached and the optimization hasn't converged yet.
  warnings.warn(

Reading the plot

The two-moons dataset is not linearly separable — no straight line can split the classes. Yet the MLP carves out a smooth, curved decision boundary that wraps right around each moon. That curved boundary is the nonlinearity at work: stacked ReLU layers let the model bend the decision surface to fit the data's shape.

This is the key advantage of neural networks over plain linear models: with hidden layers and nonlinear activations, they can model arbitrarily complicated boundaries. Try this with a linear classifier and you would see a straight line cutting awkwardly through the moons.

6.2 Deep Learning

What makes a network "deep"? Deep learning refers to training neural networks with more than one or two hidden layers. Historically, stacking many layers was hard: as you added layers, gradient descent struggled with two problems:

• Vanishing gradient: when you backpropagate through many layers, you multiply many small partial derivatives together (courtesy of the chain rule). Multiply a dozen numbers smaller than 1 and you get something close to 0 — so the earliest layers barely update and never learn. This plagued deep networks for decades.
• Exploding gradient: the opposite — gradients can grow huge, making updates jump wildly and training unstable. This was easier to tame with tricks like gradient clipping and L1/L2 regularization.

Modern deep learning overcame these hurdles: ReLU suffers far less from vanishing gradients (its gradient is a constant 1 for positive inputs), and special architectures like LSTM networks and skip connections (used in residual networks) let gradients flow through hundreds or even thousands of layers. Today "deep learning" is used broadly to mean training neural networks with the modern toolkit, regardless of exact depth — and in practice, many business problems are solved just fine with 2-3 hidden layers.

Why does depth help? Each layer can learn a hierarchy of features: the first layer detects simple edges, the next combines edges into textures or parts, the next combines parts into objects, and so on. Depth lets the model build complicated concepts out of simpler ones, which is both more parameter-efficient and more natural for structured data like images and text.

Everyday analogy: You don't learn to recognize a face all at once. You first learn to see edges, then shapes, then features like eyes and noses, then the whole face. A deep network does the same — each layer builds on the previous one's work.

6.2.1 Convolutional Neural Networks (CNN)

When the input is an image, a fully-connected MLP runs into trouble: even a small 100x100 image has 10,000 pixels, so the first layer alone would need millions of weights. Convolutional neural networks (CNNs) solve this with a simple but powerful idea inspired by how we look at images.

The key insight: in an image, nearby pixels usually belong to the same thing — sky, fur, bricks, an edge. So instead of connecting every pixel to every neuron, a CNN uses a small filter (a p x p matrix of weights) that slides across the image, one patch at a time. At each position, the filter computes a convolution: it takes the dot product of the filter with the patch under it, sums the result, adds a bias, and applies an activation (usually ReLU). The output is a feature map — a grid showing where that pattern appears.

• Each filter acts like a little pattern detector: one might fire on vertical edges, another on horizontal edges, another on corners.
• A convolution layer has many filters, so it produces many feature maps at once.
• Pooling then shrinks each feature map (e.g., by taking the max of every 2x2 block), which cuts the number of parameters and makes the network robust to small shifts.
• Stacking several conv+pool layers builds the hierarchy: edges -> textures -> parts -> objects.

Because the same filter is reused at every position, a CNN has far fewer parameters than an equivalent MLP — and it naturally respects the 2D structure of images. The filter weights are still learned by gradient descent with backpropagation.

Let's see a convolution in action on a tiny synthetic image.

# --- A single convolution, by hand, in numpy ---
# This shows exactly what one Conv2D filter does to one small image.

# A tiny 8x8 grayscale "image": a white square on a black background
img = np.zeros((8, 8))
img[2:6, 2:6] = 1.0

# A 3x3 filter that detects VERTICAL edges
F_vert = np.array([[ 1, 0, -1],
                   [ 1, 0, -1],
                   [ 1, 0, -1]])

# A 3x3 filter that detects HORIZONTAL edges
F_horiz = np.array([[ 1,  1,  1],
                    [ 0,  0,  0],
                    [-1, -1, -1]])

def conv2d(image, filt):
    # Slide `filt` across `image`, computing one convolution per position.
    h, w = image.shape
    fh, fw = filt.shape
    out_h, out_w = h - fh + 1, w - fw + 1
    out = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            patch = image[i:i+fh, j:j+fw]    # the p x p window under the filter
            out[i, j] = np.sum(patch * filt)  # dot product -> one number
    return out

fm_vert  = conv2d(img, F_vert)
fm_horiz = conv2d(img, F_horiz)

fig, axes = plt.subplots(1, 4, figsize=(14, 3.5))
axes[0].imshow(img, cmap="gray_r");        axes[0].set_title("input image")
axes[1].imshow(F_vert, cmap="gray_r");     axes[1].set_title("vertical-edge filter")
axes[2].imshow(fm_vert, cmap="gray_r");    axes[2].set_title("vertical feature map")
axes[3].imshow(fm_horiz, cmap="gray_r");   axes[3].set_title("horizontal feature map")
for ax in axes:
    ax.set_xticks([]); ax.set_yticks([])
plt.suptitle("One convolution layer: a filter slides across an image", y=1.02)
plt.tight_layout()
plt.show()

print("The vertical-edge feature map lights up on the LEFT and RIGHT edges")
print("of the square, and is ~0 in the flat interior. That is the filter")
print("'recognizing' vertical boundaries.")

The vertical-edge feature map lights up on the LEFT and RIGHT edges
of the square, and is ~0 in the flat interior. That is the filter
'recognizing' vertical boundaries.

Reading the feature maps

• The vertical-edge filter produces large positive values on the left edge of the square and large negative values on the right edge — exactly where vertical boundaries are. In the flat interior the filter sees no contrast, so the output is near zero.
• The horizontal-edge filter does the same for the top and bottom edges.

That is the whole magic of a convolution layer: each filter is a reusable pattern detector, and the feature map tells you where in the image that pattern appears. A real CNN stacks dozens of these filters across many layers, with pooling in between, so later filters can detect increasingly complex patterns (edges -> shapes -> objects) while keeping the parameter count manageable.

6.2.2 Recurrent Neural Networks (RNN)

Some data is sequential — the order of the elements matters. A sentence is a sequence of words; a stock price is a sequence of daily values; an audio clip is a sequence of sound samples. A feed-forward network has no notion of order: shuffle the inputs and it gives the same answer. Recurrent neural networks (RNNs) are built for sequences.

The core idea is the hidden state — a kind of running memory. At each time step t, an RNN unit receives two things:

1. the current input x(t), and
2. its own hidden state h(t-1) from the previous time step.

It combines them into a new hidden state:

h(t) = g(W · x(t) + U · h(t-1) + b)

where W weights the new input, U weights the memory, and g is usually tanh. The same weights W, U, b are reused at every time step, which is what lets the RNN handle sequences of any length.

A convenient way to understand an RNN is to unroll it across time: instead of one unit with a loop, imagine one copy of the unit for each time step, with the hidden state flowing from step to step. Unrolling turns the recurrent network into a deep feed-forward network where "depth" equals the sequence length — which is exactly why RNNs also suffer from vanishing gradients on long sequences.

Everyday analogy: Reading a sentence word by word, you carry along a mental summary of what you have read so far (the hidden state). Each new word updates that summary. By the end, your summary captures the meaning of the whole sentence. That is what an RNN does.

# --- A tiny RNN forward pass, by hand, in numpy ---
# One recurrent unit processes a sequence of 5 input vectors,
# carrying a hidden state from step to step.

np.random.seed(0)
seq_len, in_dim, hid_dim = 5, 2, 4   # 5 timesteps, 2-dim input, 4-dim hidden

# The recurrent weights (shared across ALL time steps)
W_x = np.random.randn(hid_dim, in_dim) * 0.5   # input -> hidden
W_h = np.random.randn(hid_dim, hid_dim) * 0.5  # hidden -> hidden (the "memory" weights)
b   = np.zeros(hid_dim)

# A made-up input sequence: 5 vectors of length 2
x_seq = np.random.randn(seq_len, in_dim)

h = np.zeros(hid_dim)   # initial hidden state (start with "no memory")

print("Processing the sequence one time step at a time:\n")
for t in range(seq_len):
    # The recurrent equation: new state = tanh( W_x @ input + W_h @ old_state + b )
    h = np.tanh(W_x @ x_seq[t] + W_h @ h + b)
    print("step {}:  input {}  ->  hidden state {}".format(
        t + 1, np.round(x_seq[t], 2), np.round(h, 2)))

print("\nNotice how each hidden state depends on the CURRENT input AND the")
print("previous hidden state -- that is the 'memory' of the sequence at work.")

Processing the sequence one time step at a time:

step 1:  input [ 2.27 -1.45]  ->  hidden state [ 0.94 -0.48  0.99  0.83]
step 2:  input [ 0.05 -0.19]  ->  hidden state [ 0.49  0.46  0.59 -0.83]
step 3:  input [1.53 1.47]  ->  hidden state [0.82 0.99 0.9  0.61]
step 4:  input [0.15 0.38]  ->  hidden state [ 0.71  0.83  0.34 -0.47]
step 5:  input [-0.89 -1.98]  ->  hidden state [-0.88 -0.98  0.68 -0.53]

Notice how each hidden state depends on the CURRENT input AND the
previous hidden state -- that is the 'memory' of the sequence at work.

Reading the RNN output

Watch the hidden state evolve: at each step it is a blend of the new input and everything that came before (compressed into the previous state). By the final step, h holds a summary of the entire sequence. You could feed that final hidden state into a simple classifier to label the whole sequence (e.g., "is this sentence positive or negative?"), or take the hidden state at every step to label each word.

Gated RNNs: LSTM and GRU

Plain RNNs have a memory problem: on long sequences, early inputs get "forgotten" because the state keeps getting overwritten. The fix is gated units — Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. These add gates (sigmoid-controlled switches between 0 and 1) that learn what to store, what to forget, and what to output. A gate near 0 blocks information; a gate near 1 lets it through. Because storing information acts like the identity function (whose derivative is a constant 1), gradients no longer vanish — so gated RNNs can learn dependencies across many time steps. Modern sequence models (including the attention-based Transformers you may hear about) build on these ideas.

Key Takeaways

• A neuron is a weighted sum plus a nonlinear activation; logistic regression is a single neuron.
• A neural network is a nested function, y = f3(f2(f1(x))), where each layer computes gl(Wl · z + bl).
• ReLU, sigmoid, and tanh are the three workhorse activations; ReLU's constant gradient is a key reason deep networks train well.
• Backpropagation is gradient descent on the network's weights, using the chain rule to send the error signal backward through the layers.
• A network with enough hidden neurons can approximate essentially any function (universal approximation), but depth does so more efficiently by building a hierarchy of features.
• Deep learning = neural networks with several hidden layers, made practical by ReLU and tricks like skip connections that fight vanishing/exploding gradients.
• CNNs use sliding filters to detect local patterns in images, sharing weights across positions to stay parameter-efficient.
• RNNs use a hidden state carried across time steps to handle sequences; gated variants (LSTM, GRU) fix the long-memory problem.

What's Next

In Chapter 7 — Problems and Solutions, we'll see how the models from this and earlier chapters get applied to real problems: classification, regression, and more, along with practical advice on choosing and tuning the right model for the task.

Exercises

These exercises cover neural networks and deep learning from Chapter 6: neurons, activation functions, the feed-forward pass, backpropagation, architecture choice, and the CNN and RNN architectures. Try each before reading the hint.

1. (Conceptual) A neuron is a weighted sum plus an activation. Why must the activation be nonlinear, and what would go wrong with a stack of purely-linear layers? Hint: a linear function of a linear function is still linear, so a deep stack of linear layers collapses into one boring linear model.
2. (Conceptual) The chapter says "logistic regression is a single neuron." Show the correspondence explicitly. Hint: logistic regression is sigmoid(w·x + b) — one neuron with a sigmoid activation.
3. (Conceptual) Compare ReLU, sigmoid, and tanh by output range, and explain why ReLU is the modern default for hidden layers. Hint: sigmoid and tanh have tiny gradients in their flat regions (the vanishing gradient problem); ReLU's gradient is a constant 1 for positive inputs, so deep networks keep learning.
4. (Conceptual) Describe backpropagation in four steps, and explain why the chain rule is needed. Hint: forward pass → measure error → backward pass (the chain rule attributes the error to each weight) → gradient-descent update; the network is a nesting of functions, so you differentiate through the nesting.
5. (Conceptual) State the universal approximation theorem, then explain why depth is still preferred over one giant hidden layer. Hint: one wide hidden layer can approximate any continuous function, but a deep net needs far fewer total parameters and naturally builds a hierarchy of features (edges → parts → objects).
6. (Conceptual) How does a CNN keep its parameter count far below an equivalent MLP on images, and what do a filter and a feature map represent? Hint: a small filter is shared across all positions (weight sharing) and pooling shrinks the maps; the feature map shows where in the image that filter's pattern appears.
7. (Conceptual) What problem do plain RNNs suffer on long sequences, and how do LSTM gates fix it? Hint: vanishing gradients make early inputs get forgotten; gates learn what to store/forget/output, and the stored state behaves like the identity function (gradient ≈ 1), so long-range signals survive.

Hands-On Coding Problems

8. (Coding) Manual feed-forward: given W1 (3x2), b1 (3), W2 (1x3), b2 (1) and input x=[2,1], compute the output of a 2→3→1 network with ReLU hidden and sigmoid output. Hint: z1 = relu(W1·x + b1); out = sigmoid(W2·z1 + b2).
9. (Coding) Plot ReLU, sigmoid, and tanh over z in [−6, 6] on the same axes using numpy. Hint: np.maximum(0, z), 1/(1+np.exp(-z)), np.tanh(z).
10. (Coding) Train an MLPClassifier(hidden_layer_sizes=(10,10), max_iter=500, random_state=5) on make_moons(noise=0.20, random_state=5) and plot its curved decision boundary on a meshgrid. Hint: np.meshgrid + np.c_ + predict + contourf.
11. (Coding) Architecture vs overfitting: fit MLPs with hidden_layer_sizes of (4,), (50,), and (200,200) on a noisy make_moons dataset, split into train/test, and print train vs test accuracy for each; comment on which overfits. Hint: the biggest net should have the highest train accuracy and the largest train–test gap.
12. (Coding) Convolution by hand: define a 3×3 vertical-edge filter and a small 6×6 image (a numpy array with a bright square), apply 2D convolution with simple nested loops, and print the resulting feature map. Hint: for each top-left position, take np.sum(patch * filter); the filter [[−1,0,1]]*3 detects vertical edges.
13. (Coding) Simulate a tiny RNN forward pass with tanh: h_t = tanh(W·x_t + U·h_{t-1} + b) over a 5-step input sequence, starting from h_0 = 0; print h at each step. Hint: loop over steps, reusing the same W, U, b, updating h each time.
14. (Coding) Activation comparison: train two MLPClassifiers on make_moons, one with activation='relu' and one with activation='logistic' (sigmoid), each max_iter=500; print both test accuracies and comment on which converges better (a stand-in for the vanishing-gradient effect). Hint: set random_state=5 for both; ReLU usually wins on this non-linear dataset.

# Exercise 8: manual feed-forward pass
import numpy as np

def relu(z): return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-z))

W1 = np.array([[1, -1], [0, 2], [-2, 1]])   # 3x2
b1 = np.array([0, 0.5, -1])                  # 3
W2 = np.array([[1, 0.5, -1]])                # 1x3
b2 = np.array([0.0])                          # 1
x = np.array([2, 1])

# TODO: forward pass
z1 = np.zeros(3)    # placeholder: relu(W1 @ x + b1)
out = 0.0           # placeholder: sigmoid(W2 @ z1 + b2)

print("hidden z1 =", z1)
print("output    =", out)

# Exercise 9: plot ReLU, sigmoid, tanh
import numpy as np
import matplotlib.pyplot as plt

z = np.linspace(-6, 6, 200)
# TODO: compute relu, sigmoid, tanh and plot all three on the same axes
relu = np.zeros_like(z)        # placeholder
sigmoid = np.zeros_like(z)     # placeholder
tanh = np.zeros_like(z)        # placeholder

# TODO: plt.plot(z, relu, ...), etc., with a legend
plt.show()

# Exercise 10: MLP on make_moons with curved boundary
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=200, noise=0.20, random_state=5)

# TODO: fit MLPClassifier(hidden_layer_sizes=(10,10), max_iter=500, random_state=5)
clf = None  # placeholder

# TODO: build a meshgrid and plot clf's decision regions with contourf, plus the data points
plt.show()

# Exercise 11: architecture vs overfitting
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=300, noise=0.30, random_state=5)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=5)

for sizes in [(4,), (50,), (200, 200)]:
    # TODO: fit MLPClassifier(hidden_layer_sizes=sizes, max_iter=500, random_state=5)
    #       print train and test accuracy
    train_acc = 0.0  # placeholder
    test_acc = 0.0   # placeholder
    print(f"{sizes}: train={train_acc:.3f}, test={test_acc:.3f}")

# TODO: comment on which architecture overfits

# Exercise 12: 2D convolution by hand with a vertical-edge filter
import numpy as np

# 6x6 image: a bright square (3) in the middle, dark (0) outside
img = np.zeros((6, 6))
img[1:5, 1:5] = 3

# vertical-edge filter (3x3)
filt = np.array([[-1, 0, 1],
                 [-1, 0, 1],
                 [-1, 0, 1]])

H, W = img.shape
fh, fw = filt.shape
out_h, out_w = H - fh + 1, W - fw + 1
feature_map = np.zeros((out_h, out_w))

# TODO: fill feature_map[i, j] = np.sum(img[i:i+fh, j:j+fw] * filt) for each valid position
print("image:\n", img)
print("feature map:\n", feature_map)

# Exercise 13: tiny RNN forward pass
import numpy as np

# 5 input vectors of size 2
X = np.array([[1, 0], [0, 1], [1, 1], [0, 0], [1, 0]])
W = np.array([[0.5, -0.5], [0.2, 0.3]])   # input weights (2x2)
U = np.array([[0.9, 0.1], [0.0, 0.8]])     # recurrent weights (2x2)
b = np.array([0.0, 0.0])

h = np.zeros(2)  # h_0 = 0
for t in range(len(X)):
    # TODO: h = np.tanh(W @ X[t] + U @ h + b)
    print(f"step {t}: h = {h}")  # placeholder prints zeros

# Exercise 14: ReLU vs sigmoid (logistic) activations
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=300, noise=0.20, random_state=5)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=5)

for act in ["relu", "logistic"]:
    # TODO: fit MLPClassifier(hidden_layer_sizes=(20,20), activation=act, max_iter=500, random_state=5)
    #       print test accuracy
    test_acc = 0.0  # placeholder
    print(f"activation={act}: test accuracy={test_acc:.3f}")

# TODO: comment on which activation does better here and why