Distributions, relationships and correlation charts

Three training groups have forty invented scores each, and two hundred invented flats have paired areas and prices. One average cannot show a score distribution, and two separate averages cannot show which area belongs to which price. A histogramhistogram: A chart that groups numeric observations into intervals and represents their counts or density with adjoining bars. groups scores into bins; a box plotbox plot: A distribution summary that displays quartiles, a median and whiskers under a stated rule, with any separately drawn observations beyond them. compares medians, quartiles and whiskers; an ECDFempirical cumulative distribution function: For each value, the fraction of the observed sample at or below that value. gives the share at or below a value without binning.

For relationships, a scatter plot keeps each pair together. A fitted line summarises linear change, while Pearson correlationPearson correlation: A dimensionless measure of linear association between paired varying measurements, on a scale from minus one to one. measures standardised linear association. Neither proves cause. The examples draw the same score data three ways and the same flats as a scatter plot, so each summary can be compared with the observations it summarises.

Three charts answer different questions about the same scores

The data are invented, from fixed seeds, so that every run gets the same numbers: 40 students in each of three sections, and 200 flats with their area, price, age and distance to a metro station.

"""The invented data of this lesson, made from fixed seeds so that every run gets the same numbers."""
import numpy as np
import pandas as pd


def exam_scores() -> pd.DataFrame:
    """Scores out of 100 for three sections of 40 students each."""
    rng = np.random.default_rng(2026)
    a = rng.normal(64, 10, 40)                                        # one broad group
    b = np.concatenate([rng.normal(46, 6, 20), rng.normal(79, 6, 20)])  # two groups of students
    c = 90 - rng.gamma(2.0, 8.0, 40)                                  # most score high, a long tail below
    scores = np.clip(np.concatenate([a, b, c]), 0, 100).round()
    return pd.DataFrame({"section": np.repeat(["A", "B", "C"], 40), "score": scores})


def flats() -> pd.DataFrame:
    """200 flats: floor area in square feet, price in millions, age in years, km to the nearest metro station."""
    rng = np.random.default_rng(7)
    area = rng.uniform(450, 1800, 200).round()
    age = rng.integers(0, 30, 200)
    metro = rng.uniform(0.2, 6.0, 200).round(1)
    price = (0.8 + 0.0058 * area - 0.04 * age - 0.25 * metro + rng.normal(0, 0.6, 200)).round(2)
    return pd.DataFrame({"area": area, "price": price, "age": age, "metro_km": metro})

import matplotlib.pyplot as plt
import numpy as np
from matplotlib.ticker import MaxNLocator

from lesson_data import exam_scores

plt.style.use("house.mplstyle")
scores = exam_scores()
groups = {name: g["score"].to_numpy() for name, g in scores.groupby("section")}
styles = {"A": ("#6b7280", "-"), "B": ("#1d5fcb", "--"), "C": ("#1c2027", ":")}

fig, (hist, box, ecdf) = plt.subplots(1, 3, figsize=(8.4, 3.2), layout="constrained")
for name, values in groups.items():
    colour, dash = styles[name]
    hist.hist(values, bins=range(20, 101, 8), histtype="step", color=colour, linestyle=dash, linewidth=2, label=name)
    ecdf.ecdf(values, color=colour, linestyle=dash, label=f"Section {name}")
box.boxplot(list(groups.values()), tick_labels=list(groups), orientation="vertical", widths=0.5)
hist.set(title="Histogram", xlabel="Score", ylabel="Students")
hist.yaxis.set_major_locator(MaxNLocator(integer=True))
box.set(title="Box plot", ylabel="Score")
ecdf.set(title="ECDF", xlabel="Score", ylabel="Share at or below")
ecdf.legend(loc="upper left", fontsize=10)
fig.savefig("three-views.svg", metadata={"Date": None})

print("section  median   Q1    Q3   IQR  lowest  below 50")
for name, values in groups.items():
    q1, median, q3 = np.percentile(values, [25, 50, 75])
    print(f"{name:>7} {median:7.1f} {q1:5.1f} {q3:5.1f} {q3 - q1:5.1f} {values.min():7.0f} {np.mean(values < 50):8.0%}")

section  median   Q1    Q3   IQR  lowest  below 50
      A    63.0  59.0  70.2  11.2      45       2%
      B    67.0  49.5  75.2  25.8      41      25%
      C    76.5  67.5  83.0  15.5      53       0%

Distributions show the spread of observations, here forty scores per section. Compare these distributions with three complementary views of the same data. The histogram shows the shape: section A has one hump in the 60s, and section B has two, one in the 40s and one in the high 70s. The box plot compares the sections in one glance: C’s median of 76.5 is the highest, and B’s box, from 49.5 to 75.2, is the widest, but the box says nothing about B’s two groups. The ECDF shows B’s two steep stretches and the gap between them; 25 % of section B is below 50. Reading its lower inverse at 0.5 gives the twentieth ordered observation. For C that is 76, while the usual interpolated sample median averages 76 and 77 to give 76.5. State the percentile convention rather than calling those different calculations identical.

Follow section C’s observations into their different summaries.

The calculation’s name and convention are part of the result, especially when two middle observations differ.

A scatter plot preserves the pair that a fitted line summarises

For relationships between two numbers of each item, the scatter plot is the chart. Here each of the 200 flats is a dot placed by its area and its price, with a straight trend line fitted by least squares:

import matplotlib.pyplot as plt
import numpy as np

from lesson_data import flats

plt.style.use("house.mplstyle")
homes = flats()
slope, intercept = np.polyfit(homes["area"], homes["price"], deg=1)   # highest power first
r = homes["area"].corr(homes["price"])

fig, ax = plt.subplots()
ax.scatter(homes["area"], homes["price"], s=16, alpha=0.6, color="#1d5fcb")
xs = np.array([450, 1800])
ax.plot(xs, intercept + slope * xs, color="#1c2027", linewidth=2)
ax.set(xlabel="Floor area (square feet)", ylabel="Price (millions)")
ax.set_title(f"About {slope * 100:.2f} million more per 100 square feet")
fig.savefig("scatter-trend.svg", metadata={"Date": None})

print(f"trend line: price = {intercept:.2f} + {slope:.5f} x area")
print(f"Pearson correlation r = {r:.2f}, so r squared = {r * r:.2f}")
print(f"the line's price for 1,000 square feet: {intercept + slope * 1000:.2f} million")

trend line: price = -0.72 + 0.00586 x area
Pearson correlation r = 0.95, so r squared = 0.91
the line's price for 1,000 square feet: 5.15 million

np.polyfit(x, y, deg=1) returns the coefficients highest power first, so the slope comes before the intercept: each extra square foot adds 0.00586 million on average, about 0.59 million per 100 square feet. The correlation, r = 0.95, says how strongly the two variables move together linearly after their scales are standardised, from −1 to +1. It does not measure prediction error in millions: changing the currency changes those distances while leaving r unchanged. Three tools help when scatter plots get harder. Transparency (alpha) draws points partly see-through, so places where many overlap look darker. hexbin counts the points in hexagonal cells and shades each cell by its count. Polynomial.fit from numpy.polynomial fits the same line, and NumPy’s documentation recommends it for new code because it is numerically more stable.

Follow the flat observations before interpreting the fitted line’s number.

A high r is not a price-per-square-foot estimate or a measure of error in millions.

A correlation matrix is a map to inspect valid pairs

Correlation charts compare columns in pairs to show which move together. These correlation charts summarise association rather than cause. Their numbers come from DataFrame.corr():

from lesson_data import flats

homes = flats()
print(homes.corr().round(2))
print()
print("strongest with price:", homes.corr()["price"].drop("price").abs().idxmax())

          area  price   age  metro_km
area      1.00   0.95  0.02      0.01
price     0.95   1.00 -0.13     -0.14
age       0.02  -0.13  1.00      0.04
metro_km  0.01  -0.14  0.04      1.00

strongest with price: area

A heatmap colours each cell of this matrix, so a table of 30 columns, 435 pairs, can be scanned in seconds. The diagonal is 1 for nonconstant columns with enough paired observations. A constant or all-missing column instead has undefined correlations, which pandas represents as NaN. The matrix is symmetric, so half of it repeats the other half. Read correlation charts with care: Pearson’s r summarises standardised linear association, so a strong curve can score near 0, one far point can create a large r on its own, and none of it says which column causes which. Here area is the only strong partner of price, while age and the distance to the metro pull the price down only slightly.

seaborn builds on matplotlib and draws each of these statistical charts in one call, with grouping by a column and a legend built in. Its most recent release is 0.13.2, and it still works with pandas 3, so it is worth using when it saves code; the charts in this lesson use matplotlib directly, which does not depend on seaborn’s release pace.

Trace a Pearson cell from row pairs to its meaning.

The valid pair count and the pattern behind the cell matter as much as its displayed colour.

What each distribution chart can hide, reproduced Pro: opens with a Pro, Premium or Ultimate pass

Scatter plots, trend lines and correlation heatmaps without the traps Pro: opens with a Pro, Premium or Ultimate pass

Practice

Read the score summaries and paired flat measurements before answering. Distinguish a plotted percentile from an interpolated median and a correlation from a causal claim.

Interview questions

Key takeaways

• A histogram shows a distribution’s shape, a box plot compares many groups, and an ECDF shows every value and its percentiles without bins.
• A scatter plot shows a relationship; np.polyfit returns the slope first, and r measures standardised linear association rather than prediction error in the measured unit.
• A Pearson correlation chart summarises linear association for each valid pair, so inspect curves, missing pairs and extreme observations before interpreting it.
• seaborn draws these charts in fewer lines on top of matplotlib.