Chapter 11 — Exploratory Data Analysis
Exploratory Data Analysis (EDA) is the disciplined curiosity you apply before modeling: summarize, visualize, and question the data until you understand its structure, quirks, and stories. This chapter conducts a complete EDA on the penguins dataset.
Learning Objectives
- Frame and structure an EDA workflow.
- Assess dataset shape, types, and missingness.
- Summarize numeric and categorical variables.
- Explore univariate distributions.
- Explore bivariate relationships.
- Examine correlations and group differences.
- Translate findings into actionable insights.
Prerequisites / Imports
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style='whitegrid')
1 The Dataset
We use the Palmer Penguins data — measurements of three penguin species.
In [1]:
penguins = sns.load_dataset('penguins')
print('shape:', penguins.shape)
penguins.head()
shape: (344, 7)
| species | island | bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | sex | |
|---|---|---|---|---|---|---|---|
| 0 | Adelie | Torgersen | 39.1 | 18.7 | 181.0 | 3750.0 | Male |
| 1 | Adelie | Torgersen | 39.5 | 17.4 | 186.0 | 3800.0 | Female |
| 2 | Adelie | Torgersen | 40.3 | 18.0 | 195.0 | 3250.0 | Female |
| 3 | Adelie | Torgersen | NaN | NaN | NaN | NaN | NaN |
| 4 | Adelie | Torgersen | 36.7 | 19.3 | 193.0 | 3450.0 | Female |
2 Structure and Missingness
Check dtypes and where values are missing.
In [1]:
penguins.info()
print('\nmissing per column:'); print(penguins.isna().sum())
<class 'pandas.core.frame.DataFrame'> RangeIndex: 344 entries, 0 to 343 Data columns (total 7 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 species 344 non-null object 1 island 344 non-null object 2 bill_length_mm 342 non-null float64 3 bill_depth_mm 342 non-null float64 4 flipper_length_mm 342 non-null float64 5 body_mass_g 342 non-null float64 6 sex 333 non-null object dtypes: float64(4), object(3) memory usage: 18.9+ KB missing per column: species 0 island 0 bill_length_mm 2 bill_depth_mm 2 flipper_length_mm 2 body_mass_g 2 sex 11 dtype: int64
3 Drop Missing Values for the EDA
For this walkthrough we drop rows with missing measurements.
In [1]:
df = penguins.dropna().reset_index(drop=True)
print('clean shape:', df.shape)
clean shape: (333, 7)
4 Univariate: Numeric Summaries
describe summarizes numeric columns.
In [1]:
df.describe().round(2)
| bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | |
|---|---|---|---|---|
| count | 333.00 | 333.00 | 333.00 | 333.00 |
| mean | 43.99 | 17.16 | 200.97 | 4207.06 |
| std | 5.47 | 1.97 | 14.02 | 805.22 |
| min | 32.10 | 13.10 | 172.00 | 2700.00 |
| 25% | 39.50 | 15.60 | 190.00 | 3550.00 |
| 50% | 44.50 | 17.30 | 197.00 | 4050.00 |
| 75% | 48.60 | 18.70 | 213.00 | 4775.00 |
| max | 59.60 | 21.50 | 231.00 | 6300.00 |
5 Univariate: Categorical Counts
Count the levels of each categorical column.
In [1]:
for col in ['species','island','sex']:
print(f'\n{col}:'); print(df[col].value_counts())
species: species Adelie 146 Gentoo 119 Chinstrap 68 Name: count, dtype: int64 island: island Biscoe 163 Dream 123 Torgersen 47 Name: count, dtype: int64 sex: sex Male 168 Female 165 Name: count, dtype: int64
6 Univariate: Distributions
Histograms reveal the shape of each measurement.
In [1]:
num_cols = ['bill_length_mm','bill_depth_mm','flipper_length_mm','body_mass_g']
df[num_cols].hist(figsize=(10,8), bins=20, edgecolor='black')
plt.tight_layout()
plt.show()
<cell-expr>:1: UserWarning:
7 Bivariate: Relationships by Species
A scatter colored by species often separates groups cleanly.
In [1]:
plt.figure(figsize=(8,5))
sns.scatterplot(data=df, x='flipper_length_mm', y='body_mass_g', hue='species', style='sex')
plt.title('Body mass vs flipper length by species')
plt.show()
<cell-expr>:1: UserWarning:
8 Bivariate: Boxplots by Group
Boxplots compare distributions across categories.
In [1]:
plt.figure(figsize=(8,5))
sns.boxplot(data=df, x='species', y='body_mass_g', hue='sex')
plt.title('Body mass by species and sex')
plt.show()
<cell-expr>:1: UserWarning:
9 Correlations
The correlation matrix shows how numeric features move together.
In [1]:
corr = df[num_cols].corr().round(2)
corr
| bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | |
|---|---|---|---|---|
| bill_length_mm | 1.00 | -0.23 | 0.65 | 0.59 |
| bill_depth_mm | -0.23 | 1.00 | -0.58 | -0.47 |
| flipper_length_mm | 0.65 | -0.58 | 1.00 | 0.87 |
| body_mass_g | 0.59 | -0.47 | 0.87 | 1.00 |
In [1]:
plt.figure(figsize=(6,5))
sns.heatmap(corr, annot=True, cmap='coolwarm', vmin=-1, vmax=1)
plt.title('Penguins correlations')
plt.show()
<cell-expr>:1: UserWarning:
10 Group Summaries
Group means quantify the differences we see in plots.
In [1]:
df.groupby('species')[num_cols].mean().round(1)
| bill_length_mm | bill_depth_mm | flipper_length_mm | body_mass_g | |
|---|---|---|---|---|
| species | ||||
| Adelie | 38.8 | 18.3 | 190.1 | 3706.2 |
| Chinstrap | 48.8 | 18.4 | 195.8 | 3733.1 |
| Gentoo | 47.6 | 15.0 | 217.2 | 5092.4 |
Case Study: Insights from the Penguin Data
Synthesize the EDA into a short list of findings and a recommendation.
In [1]:
print('Key findings:')
print('- Gentoo penguins are much heavier and have longer flippers than Adelie/Chinstrap.')
print('- Body mass and flipper length are strongly positively correlated (~0.87).')
print('- Bill length separates Chinstrap from Adelie despite similar body mass.')
print('- Males are heavier than females within every species.')
print('\nRecommendation: species can likely be predicted from flipper length, body mass, and bill dimensions.')
Key findings: - Gentoo penguins are much heavier and have longer flippers than Adelie/Chinstrap. - Body mass and flipper length are strongly positively correlated (~0.87). - Bill length separates Chinstrap from Adelie despite similar body mass. - Males are heavier than females within every species. Recommendation: species can likely be predicted from flipper length, body mass, and bill dimensions.
Exercises
- Load the seaborn
diamondsdataset and report its shape and dtypes. - Summarize and visualize the distribution of diamond price.
- Make a boxplot of price by cut.
- Compute the correlation matrix of the numeric columns and plot a heatmap.
- Count the number of diamonds in each color category.
- Plot carat vs price, colored by cut.
- Compute average price by cut and clarity.
- Identify the feature most correlated with price.
- Write three findings and one recommendation from your diamonds EDA.
- Explain why EDA should precede modeling.
Python Data Science: From Foundations to Applications — Chapter 11