Chapter 1 — Introduction

Welcome! This is the first chapter of our beginner-friendly journey through machine learning, adapted from Andriy Burkov's The Hundred-Page Machine Learning Book. In this chapter we build the big-picture mental model that the rest of the book fills in.

In this chapter you will learn:

• What machine learning really is (and why the name is a bit misleading)
• The four main types of learning
• How a supervised learning system turns data into a model
• Why a model trained on old data can make good predictions on brand-new data

1.1 What Is Machine Learning?

Here is the honest truth that the book opens with: machines don't really "learn." A typical "learning machine" finds a mathematical formula that, when applied to a collection of inputs (the training data), produces the desired outputs. That same formula also gives correct outputs for most other inputs — as long as those new inputs come from the same kind of distribution as the training data.

Why isn't that learning? Because if you slightly distort the input, the output can become completely wrong. If you learned to play a video game by staring straight at the screen, you would still play well if someone tilted the screen a little. A machine learning model trained only on a straight screen would usually fail on a tilted one — unless it was also trained to handle rotation.

So why the name? Mostly marketing. Arthur Samuel coined "machine learning" at IBM in 1959. Like "artificial intelligence," the term stuck even though it is used by analogy to animal learning, not literally.

A practical working definition:

Machine learning is the process of solving a practical problem by (1) gathering a dataset, and (2) algorithmically building a statistical model based on that dataset, then using that model to solve the problem.

The Machine Learning Recipe

Almost every machine learning project follows the same three-step recipe:

1. Gather data — collect examples of the phenomenon you care about.
2. Build a model — run a learning algorithm on the data to produce a mathematical formula/model.
3. Use the model — feed it new inputs to get predictions or decisions.

Let's make this concrete with a tiny, visual example in code.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs

plt.rcParams["figure.figsize"] = (6, 4)
plt.rcParams["axes.grid"] = True

# Step 1 -- Gather data: two clusters = 'spam' (+1) and 'not spam' (-1)
X, y_raw = make_blobs(n_samples=200, centers=2, cluster_std=1.0, random_state=42)
y = np.where(y_raw == 0, -1, 1)

plt.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:orange", label="+1 (spam)")
plt.scatter(X[:, 0][y == -1], X[:, 1][y == -1], color="tab:blue", label="-1 (not spam)")
plt.title("Step 1 — Gather data: two classes of examples")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

1.2 Types of Learning

Learning comes in four main flavors, depending on what kind of data you have:

Type	Data	Goal
Supervised	labeled examples {(xᵢ, yᵢ)}	predict the label y for a new x
Unsupervised	unlabeled examples {xᵢ}	find structure (clusters, reduced dimensions, outliers)
Semi-supervised	a few labeled + many unlabeled	same as supervised, but leverage the unlabeled data
Reinforcement	states + actions + rewards	learn a policy that maximizes long-term reward

Let's unpack each one.

1.2.1 Supervised Learning

In supervised learning the dataset is a collection of labeled examples {(xᵢ, yᵢ)} for i = 1…N.

• Each xᵢ is a feature vector — a list of numbers that describe one example. If an example is a person, feature 1 could be height in cm, feature 2 weight in kg, feature 3 gender, and so on.
• Each feature at position j always stores the same kind of information across all examples (feature 2 is always "weight in kg").
• The label yᵢ is what we want to predict. It is usually either a class (spam / not_spam) or a real number (a price, a temperature).

The goal of a supervised learning algorithm is to use the dataset to produce a model that takes a feature vector x as input and outputs enough information to deduce its label.

Everyday analogy: You are given thousands of flashcards. On the front is a photo (the feature vector); on the back is "cat" or "dog" (the label). After studying them, you can label a new photo you have never seen. That is supervised learning.

1.2.2 Unsupervised Learning

In unsupervised learning the dataset is a collection of unlabeled examples {xᵢ}. There are no labels to learn from — the algorithm must find structure on its own.

Typical tasks:

• Clustering — group similar examples together; the model returns a cluster id for each input.
• Dimensionality reduction — compress a feature vector into one with fewer features while keeping the important information.
• Outlier detection — output a number saying how "unusual" an example is compared to a typical one.

Everyday analogy: You are handed a big pile of mixed photos with no captions and asked to sort them into albums of similar-looking pictures. Nobody told you the categories — you discover them. That is unsupervised learning.

from sklearn.cluster import KMeans

# Unlabeled data: 3 natural groups (we PRETEND we don't know the labels)
X_un, _ = make_blobs(n_samples=300, centers=3, cluster_std=1.1, random_state=7)

# Let the algorithm find 3 clusters on its own -- no labels given!
kmeans = KMeans(n_clusters=3, n_init=10, random_state=7)
labels = kmeans.fit_predict(X_un)

plt.scatter(X_un[:, 0], X_un[:, 1], c=labels, cmap="viridis", s=20)
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1],
            marker="X", s=200, color="red", edgecolors="white", label="found centers")
plt.title("Unsupervised: KMeans discovers 3 clusters with no labels")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend()
plt.show()

C:\Users\DELL\anaconda3\Lib\site-packages\sklearn\cluster\_kmeans.py:1419: UserWarning: KMeans is known to have a memory leak on Windows with MKL, when there are less chunks than available threads. You can avoid it by setting the environment variable OMP_NUM_THREADS=2.
  warnings.warn(

1.2.3 Semi-Supervised Learning

Here the dataset has both labeled and unlabeled examples — usually far more unlabeled than labeled. The goal is the same as supervised learning; the hope is that the many unlabeled examples reveal more about the data's distribution and so help build a better model.

It can feel counter-intuitive that more unlabeled data helps. But unlabeled examples still carry information: a bigger sample reflects the true distribution better, and a good algorithm can exploit that.

1.2.4 Reinforcement Learning

In reinforcement learning an agent "lives" in an environment. It perceives the environment's state as a feature vector, chooses an action, and receives a reward (and moves to a new state). The goal is to learn a policy — a function from states to the best action — that maximizes the expected average reward.

This is for sequential decision making with long-term goals: game playing, robotics, logistics. This book (and therefore our notebooks) focuses on one-shot decisions where examples are independent, so we leave reinforcement learning aside.

1.3 How Supervised Learning Works

Let's walk through the whole supervised pipeline once, end to end, using spam detection as the example.

Step 1 — Gather data. Collect, say, 10,000 emails, each manually labeled "spam" or "not_spam".

Step 2 — Convert each email into a feature vector. A classic method is the bag of words: take a dictionary of 20,000 English words and make a 20,000-dimensional vector where position j is 1 if word j appears in the email, else 0. Now every email is a vector of 0s and 1s, and every label is text.

Step 3 — Turn labels into numbers. Many algorithms need numeric labels. The book illustrates with a Support Vector Machine (SVM), which wants the positive class ("spam") as +1 and the negative class ("not_spam") as −1.

Step 4 — Run the learning algorithm to get a model. SVM imagines every feature vector as a point in a high-dimensional space (20,000 dimensions here!) and draws a hyperplane that separates the +1 points from the −1 points. The boundary between classes is called the decision boundary.

The hyperplane has the equation w·x + b = 0, where w is a vector (same size as x) and b is a number. The prediction for a new x is:

ŷ = sign(w·x + b) → +1 means spam, −1 means not spam.

Step 5 — Find the best w and b. The machine optimizes: it wants every training example on the correct side of the boundary, and it wants the largest possible margin (the gap between the closest +1 and −1 points). A big margin means better generalization — doing well on new data, not just the training data.

To get a big margin, SVM minimizes the length (norm) of w, because the distance between the two margin lines equals 2 / ‖w‖ — so a smaller ‖w‖ means a bigger gap.

Let's visualize this with a linear SVM on our 2D dataset, and draw the decision boundary and the margin.

from sklearn.svm import SVC

# Same two-blob dataset as the first plot
X, y_raw = make_blobs(n_samples=200, centers=2, cluster_std=1.0, random_state=42)
y = np.where(y_raw == 0, -1, 1)

# Step 4: train a linear SVM. Large C -> essentially a 'hard' margin.
clf = SVC(kernel="linear", C=1000, random_state=42)
clf.fit(X, y)

# Plot the data
plt.scatter(X[:, 0][y == 1], X[:, 1][y == 1], color="tab:orange", label="+1 (spam)")
plt.scatter(X[:, 0][y == -1], X[:, 1][y == -1], color="tab:blue", label="-1 (not_spam)")

# Draw the decision boundary w.x + b = 0 and the margins w.x + b = +/-1
ax = plt.gca()
xx = np.linspace(*ax.get_xlim(), 200)
w = clf.coef_[0]
b = clf.intercept_[0]
yy          = -(w[0] * xx + b) / w[1]       # decision boundary
yy_margin_p = -(w[0] * xx + b - 1) / w[1]   # margin: w.x + b = +1
yy_margin_m = -(w[0] * xx + b + 1) / w[1]   # margin: w.x + b = -1
plt.plot(xx, yy, "k-",  label="decision boundary  w·x + b = 0")
plt.plot(xx, yy_margin_p, "k--", alpha=0.6, label="margin  w·x + b = ±1")
plt.plot(xx, yy_margin_m, "k--", alpha=0.6)

# Highlight the support vectors (points lying on the margin)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=160, facecolors="none", edgecolors="red", linewidths=1.8,
            label="support vectors")
plt.title("Step 4 — SVM finds the widest-margin separating hyperplane")
plt.xlabel("feature 1")
plt.ylabel("feature 2")
plt.legend(loc="upper left", fontsize=8)
plt.show()

Reading the plot

• The solid black line is the decision boundary: points on one side are predicted +1, on the other side −1.
• The two dashed lines are the margins — the closest the training points get to the boundary on each side.
• The circled red points are the support vectors: the examples that sit exactly on the margin. They "support" the boundary — move one, and the boundary moves.
• SVM chose the boundary that makes the gap (margin) between the classes as wide as possible.

The book notes that any classification algorithm creates some decision boundary — straight, curved, or complex. The shape of that boundary is what makes one algorithm different from another, and it also determines accuracy. Two other practical differentiators are training speed and prediction speed — sometimes a slightly less accurate model that trains or predicts much faster is the better choice.

1.4 Why the Model Works on New Data

Why should a model built from old examples predict new ones correctly?

Look at the SVM plot again. If the two classes are separable by a boundary, then each class lives in its own region of space. If your training examples were collected randomly and independently from the same process, then a new example is statistically likely to land near other examples of its own class — not far away in empty space. So a boundary that separated the training examples well will, with high probability, also separate new examples well.

Errors happen when a new example lands on the "wrong" side of the boundary — but because that is less likely, correct predictions outnumber errors. A larger training set makes it even less likely that a new example is very different from anything seen before. And SVM's maximum-margin trick explicitly pushes the boundary as far as possible from both classes, which is exactly what helps on new data.

The formal study of when and under what conditions a learner will "probably output an approximately correct" model is called PAC learning (Probably Approximately Correct). You don't need the math now — just the intuition above.

Mini demo: bag-of-words features

Let's make the "bag of words" idea tangible with a tiny vocabulary of 6 words and a few short emails.

# A tiny 'dictionary' of 6 words (alphabetically sorted, like the book's
# 20,000-word idea, just much smaller)
vocab = ["free", "hello", "money", "now", "prize", "team"]

emails = {
    "hello team":            "not_spam",
    "free prize money now":  "spam",
    "team meeting now":      "not_spam",
    "free money now":        "spam",
}

def bag_of_words(text, vocab):
    # Convert text into a 0/1 feature vector over the vocabulary
    words = set(text.split())
    return [1 if word in words else 0 for word in vocab]

# Show the feature vectors and numeric labels (+1 spam, -1 not_spam)
print(f"{'email':<24}{'vector':<28}label")
print("-" * 60)
for text, label in emails.items():
    vec = bag_of_words(text, vocab)
    y_num = 1 if label == "spam" else -1
    print(f"{text:<24}{str(vec):<28}{y_num}")

print()
print("Notice: 'meeting' is not in the vocabulary, so it is simply ignored.")
print("That is exactly how a bag-of-words model treats unknown words.")

email                   vector                      label
------------------------------------------------------------
hello team              [0, 1, 0, 0, 0, 1]          -1
free prize money now    [1, 0, 1, 1, 1, 0]          1
team meeting now        [0, 0, 0, 1, 0, 1]          -1
free money now          [1, 0, 1, 1, 0, 0]          1

Notice: 'meeting' is not in the vocabulary, so it is simply ignored.
That is exactly how a bag-of-words model treats unknown words.

Key Takeaways

• "Machine learning" is really finding a formula that maps inputs to outputs and generalizes to new inputs. It is learning by analogy, not literally.
• The recipe is always: gather data → build a model → use the model.
• Four learning types by data: supervised (labeled), unsupervised (unlabeled), semi-supervised (mostly unlabeled), reinforcement (states / actions / rewards).
• A feature vector describes an example; a label is what we predict.
• Supervised learning builds a decision boundary; the algorithm determines the boundary's shape.
• SVM finds the widest-margin boundary, which helps it generalize to new data.
• Models generalize because new data statistically resembles training data — bigger training sets and bigger margins help.

What's Next

In Chapter 2 — Notation and Definitions, we'll learn the vocabulary and math notation (vectors, random variables, Bayes' rule, and more) that the rest of the book relies on. Don't worry — we'll keep it friendly and grounded in examples.

Exercises

Check your understanding of Chapter 1. The first block is conceptual; the second block gets you coding the spam-detection and clustering demos you saw in this chapter. Brief hints are included — try the problem before reading the hint.

1. (Conceptual) The chapter insists "machines don't really learn." Using the tilted-screen argument, explain why a model that scores 100% on training data can still fail badly on slightly altered inputs. Hint: think about which distribution the training data was drawn from versus what the new input looks like.
2. (Conceptual) Restate the three-step "machine learning recipe" in your own words and map the spam-detection walkthrough onto each step. Hint: gather → build → use; decide which step "convert emails to vectors" belongs to.
3. (Conceptual) For each scenario, say which of the four learning types it is and why: (a) grouping customers by buying behavior with no labels; (b) predicting house prices from labeled past sales; (c) a robot learning to walk from rewards; (d) 100 labeled tumors + 10,000 unlabeled scans, predicting malignancy. Hint: check whether labels exist and whether the task is sequential.
4. (Conceptual) Define feature vector and label precisely. Using the 6-word vocabulary from the demo, what is the feature vector of "free money now" and what is its numeric label? Hint: position j is 1 if vocab word j appears; spam = +1.
5. (Conceptual) The margin width equals 2/‖w‖, so SVM minimizes ‖w‖ to get a big margin. Explain in plain words why a bigger margin improves generalization to new data. Hint: a wider gap means a new point is less likely to land on the wrong side of the boundary.
6. (Conceptual) What are support vectors, and why does the chapter say "move one, and the boundary moves"? If you removed a non-support-vector point from the training set, would the SVM boundary change? Hint: only the points lying on the margin determine w and b.
7. (Conceptual) The chapter says more unlabeled data can help in semi-supervised learning, which feels counter-intuitive. Give one concrete reason unlabeled examples still carry useful information. Hint: they reveal the shape and density of the input distribution.
8. (Conceptual) In plain words, what does PAC ("Probably Approximately Correct") learning guarantee, and which two assumptions about the training data make it work? Hint: random, independent sampling from the same distribution, plus a large enough sample.

Hands-On Coding Problems

9. (Coding) Using the 6-word vocabulary ["free","hello","money","now","prize","team"], implement bag_of_words(text, vocab) and print the 0/1 vectors for "free prize now" and "hello team meeting". Confirm "meeting" is ignored. Hint: set(text.split()) then check membership for each vocab word.
10. (Coding) Recreate the supervised spam demo: two blobs via make_blobs (200 samples, 2 centers, random_state=42), labels mapped to +1/−1, train a linear SVC(C=1000), and plot the points plus the decision boundary w·x + b = 0. Hint: clf.coef_[0] is w, clf.intercept_[0] is b; boundary is -(w[0]*xx + b)/w[1].
11. (Coding) Starting from Exercise 10, increase cluster_std to 3.0 so the classes overlap. Retrain with a large C and again with a small C (e.g. 0.1). In a comment, compare the margins and the number of support vectors (len(clf.support_vectors_)) in the two cases. Hint: overlap ⇒ more support vectors and a softer margin; small C allows more margin violations.
12. (Coding) Rebuild the unsupervised demo: make 3 blobs (300 samples, random_state=7), run KMeans(n_clusters=3, n_init=10, random_state=7), and scatter-plot the points colored by discovered cluster with the cluster centers marked as red "X". Hint: fit_predict gives labels; cluster_centers_ gives the markers.
13. (Coding) Combine the ideas: build a tiny labeled dataset from the bag-of-words vectors of the four emails in the chapter, train a linear SVM on them, and predict the label of "free prize money now". Does it output +1 (spam)? Hint: stack vectors with np.array([...]), use labels [1,-1,-1,1] matching the chapter's emails dict, then clf.predict.

# Exercise 9: bag-of-words for new emails
vocab = ["free", "hello", "money", "now", "prize", "team"]

def bag_of_words(text, vocab):
    # TODO: split text into words and return a 0/1 vector over vocab
    return [0] * len(vocab)  # placeholder -- replace with your implementation

for text in ["free prize now", "hello team meeting"]:
    print(text, "->", bag_of_words(text, vocab))

# Exercise 10: linear SVM on two blobs + decision boundary
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.svm import SVC

X, y_raw = make_blobs(n_samples=200, centers=2, cluster_std=1.0, random_state=42)
y = np.where(y_raw == 0, -1, 1)

# TODO: train a linear SVM with a large C (e.g. C=1000)
clf = None  # replace: clf = SVC(kernel="linear", C=1000); clf.fit(X, y)

# TODO: plot the points and the decision boundary w.x + b = 0
# Hint: w = clf.coef_[0], b = clf.intercept_[0]; boundary: -(w[0]*xx + b)/w[1]
plt.show()

# Exercise 11: what happens when the classes overlap?
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.svm import SVC

X, y_raw = make_blobs(n_samples=200, centers=2, cluster_std=3.0, random_state=42)
y = np.where(y_raw == 0, -1, 1)

# TODO: train two SVMs -- SVC(kernel="linear", C=1000) and SVC(kernel="linear", C=0.1).
# Plot both decision boundaries, then write a comment comparing:
#   - the margin width, and
#   - len(clf.support_vectors_) for each model.
# 
# Your comparison comment:
# 
plt.show()

# Exercise 12: unsupervised KMeans discovers 3 clusters
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.cluster import KMeans

X, _ = make_blobs(n_samples=300, centers=3, cluster_std=1.1, random_state=7)

# TODO: fit KMeans and predict labels
labels = np.zeros(len(X), dtype=int)      # placeholder
centers = np.zeros((3, 2))                # placeholder

# TODO: scatter-plot X colored by `labels` and mark `centers` with red "X"
plt.show()

# Exercise 13: train an SVM on bag-of-words vectors and predict a new email
import numpy as np
from sklearn.svm import SVC

vocab = ["free", "hello", "money", "now", "prize", "team"]
emails = {
    "hello team":            "not_spam",
    "free prize money now":  "spam",
    "team meeting now":      "not_spam",
    "free money now":        "spam",
}

def bag_of_words(text, vocab):
    # TODO: implement the 0/1 vector
    return [0] * len(vocab)  # placeholder

X = np.array([bag_of_words(t, vocab) for t in emails])
y = np.array([1 if lab == "spam" else -1 for lab in emails.values()])

# TODO: train a linear SVM on (X, y) and predict the label of "free prize money now"
print("prediction:", None)  # placeholder -- expected: +1 (spam)