Data Analysis with Python (pandas, polars, DuckDB) Module 11 – Statistics an analyst uses every week
Correlation, trend lines and causation
Pearson and Spearman correlation in pandas, a trend line with numpy.polyfit, why one outlier changes both, and why correlation alone never shows a cause.
What you will learn
- Compute Pearson and Spearman correlations and know when each fits
- Calculate and interpret a straight-line trend with numpy.polyfit
- Explain confounding and why correlation is not causation
- Use residuals to check whether a straight line is reasonable
Before you start
What you’ll learn · Before you start
What you will learn
- Compute Pearson and Spearman correlations and know when each fits
- Calculate and interpret a straight-line trend with numpy.polyfit
- Explain confounding and why correlation is not causation
- Use residuals to check whether a straight line is reasonable
Before you start
On this page
Correlation measures how closely two numeric columns move together, on a scale from −1 to 1. Pearson correlation: A dimensionless measure of linear association between paired varying measurements, on a scale from minus one to one. More in “Distributions, relationships and correlation charts” measures linear association. Spearman correlation: A coefficient of monotonic association obtained from paired ranks; unusual rank pairings can still affect it. applies it to ranks, measuring monotonic association; ranks limit an extreme value’s magnitude effect, but do not make it immune to unusual points. Trend lines
add the size of the relationship: np.polyfit(x, y, 1) returns a slope in real units, such as 2.75 more marks for
every extra hour of study. None of these numbers says why two columns move together. A third variable can drive both,
which is why correlation is not causation, and why a plot comes before any coefficient. This lesson computes all
three on invented data with pandas and NumPy, then shows where each one misleads.
Values and ranks respond differently to one wrong row
The data is invented: the hours 24 students studied in the week before a test, and their marks out of 100. A second run adds one row that a typing slip turned from 4.0 hours into 40.
import pandas as pd
# Invented records of 24 students: hours of study in the week before a test, and the mark out of 100.
students = pd.DataFrame({
"hours": [2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 12, 12, 13, 14, 15, 17],
"marks": [35, 42, 44, 38, 48, 49, 52, 52, 57, 48, 64, 64, 53, 64, 62, 53, 67, 63, 68, 66, 70, 76, 80, 76],
})
print("24 students")
print(students.corr(method="pearson").round(2).to_string())
print(f"spearman: {students['hours'].corr(students['marks'], method='spearman'):.2f}")
# One more row, typed as 40 hours instead of 4.0: a single data-entry slip.
slip = pd.concat([students, pd.DataFrame({"hours": [40], "marks": [41]})], ignore_index=True)
print("\nwith the slip (40 hours, 41 marks)")
for method in ("pearson", "spearman"):
print(f"{method:<9} {slip['hours'].corr(slip['marks'], method=method):.2f}")
print("\nthe ranks Spearman correlates (last five rows):")
print(slip.rank().tail(5).to_string())
print(f"pearson of the ranks: {slip.rank().corr().iloc[0, 1]:.2f}")Output
24 students
hours marks
hours 1.00 0.92
marks 0.92 1.00
spearman: 0.93
with the slip (40 hours, 41 marks)
pearson 0.24
spearman 0.74
the ranks Spearman correlates (last five rows):
hours marks
20 21.0 22.0
21 22.0 23.5
22 23.0 25.0
23 24.0 23.5
24 25.0 3.0
pearson of the ranks: 0.74
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 hours_marks.py
On the 24 original rows both correlations are high, 0.92 and 0.93: students who studied longer scored higher, and the pattern is close to a straight line. The slip changes the story. Pearson’s r falls to 0.24, because it weighs every point by its distance from the mean, and 40 hours is far from everything else. Spearman’s ρ falls only to 0.74. On ranks, the slip is just the student with the most hours and the third-lowest mark: one odd rank among 25, however extreme the value. The last line of the output confirms it: Pearson’s r of the ranks is Spearman’s ρ.
| Criterion | Pearson's r | Spearman's ρ |
|---|---|---|
| Measures | How close the points lie to a straight line | Whether y tends to rise or fall as x rises, in any shape |
| Computed on | The values | The ranks of the values |
| One extreme value | Can change it greatly | Can still change it |
| Curved but steadily rising data | Below 1, though the pattern is perfect | 1 for a perfectly steady rise |
| In pandas | df.corr() or df.corr(method="pearson") | df.corr(method="spearman") |
Use Pearson when the relationship looks straight and the data has no wild values, and Spearman for ranked data, skewed columns or curved but steady patterns. When the two disagree strongly, plot the data: the gap itself says that a few points or a curve are deciding the answer.
Follow the mistaken row through values and ranks:
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- On the 24 invented original rows, Pearson rounds to 0.92 and Spearman to 0.93. Both summarize association rather than a cause.
- Adding the one mistaken 40-hour, 41-mark row changes Pearson to 0.24 and Spearman to 0.74. This is one erroneous observation, not a consistent conversion of all units.
- The slip is rank 25 for hours and rank 3 for marks. Ranks limit the effect of increasing the same extreme magnitude, but a discordant rank can still change Spearman substantially.
Checkpoint
Check: Values and ranks
A slope has units and an observed fitting range
A correlation says how tight a relationship is; trend lines say how steep it is. np.polyfit(x, y, deg=1) fits
the straight line with the smallest sum of squared vertical gaps and returns its coefficients, highest power first:
the slope, then the intercept.
import numpy as np
import pandas as pd
students = pd.DataFrame({
"hours": [2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 12, 12, 13, 14, 15, 17],
"marks": [35, 42, 44, 38, 48, 49, 52, 52, 57, 48, 64, 64, 53, 64, 62, 53, 67, 63, 68, 66, 70, 76, 80, 76],
})
slope, intercept = np.polyfit(students["hours"], students["marks"], deg=1) # highest power first
print(f"marks = {intercept:.1f} + {slope:.2f} x hours")
print(f"each extra hour goes with {slope:.1f} more marks; 10 hours predict {intercept + slope * 10:.0f}")
residuals = students["marks"] - (intercept + slope * students["hours"])
print(f"residuals: mean {round(residuals.mean(), 2) + 0.0:.2f}, largest {residuals.abs().max():.1f} marks")
print(f"r squared: {students['hours'].corr(students['marks']) ** 2:.2f}")
# The same fit with the data-entry slip (40 hours, 41 marks) added.
hours = np.append(students["hours"], 40)
marks = np.append(students["marks"], 41)
slope_slip, intercept_slip = np.polyfit(hours, marks, deg=1)
print(f"with the slip: marks = {intercept_slip:.1f} + {slope_slip:.2f} x hours")Output
marks = 35.0 + 2.75 x hours each extra hour goes with 2.8 more marks; 10 hours predict 63 residuals: mean 0.00, largest 9.7 marks r squared: 0.84 with the slip: marks = 53.4 + 0.40 x hours
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 trend_line.py
The rounded line is marks = 35.0 + 2.75 × hours, about 2.8 more marks per extra hour. At 10 hours the full-precision fit rounds to 63 marks; displayed coefficients give 62.5. A residual: An observed value minus its fitted prediction under a stated model, retaining the outcome’s units. is actual minus predicted mark; the largest absolute residual is 9.7. r², 0.84, is the share of the variation in marks that the line accounts for. The slip drags the slope down to 0.40, because a least-squares line, like Pearson’s r, gives an extreme point great weight. Read a slope only inside the range of the data, 2 to 17 hours here: for 40 hours the line would predict 145 marks on a test out of 100.
India-specific
Rent and property listings in Indian cities quote prices per square foot, so a trend line of monthly rent against area reads naturally as “₹20 more rent for every extra square foot”. Report the slope in those units, with the range of areas it was fitted on.
Compare interpolation with extrapolation:
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- polyfit returns slope then intercept. The rounded fitted equation is 35.0+2.75×hours over the observed 2-to-17-hour range.
- At 10 hours the full-precision fitted line rounds to 63 marks. The displayed rounded coefficients give 62.5; their rounding explains the difference.
- At 40 hours it predicts 145 marks on a test out of 100, exposing the danger of extrapolating that straight line.
Checkpoint
Check: Slope, units and fit range
The same rounded coefficient can hide different shapes
Four datasets of 12 points each, with r rounded to 0.75 and their least-squares lines
Text description of the diagram
Each panel shows 12 points and its fitted dashed straight line. Pearson's correlation rounds to 0.75 in all four.
- A line with noise: the points scatter evenly around a rising straight line. Here the line describes the data well.
- A curve that bends back: the points rise steeply, level off and then fall. The straight line runs below the points in the middle and above them at both ends.
- One far point: eleven points form a cloud with little observed linear association. A point far to the right and higher up dominates the pooled coefficient.
- Two groups of six points: one cluster low on the left, the other higher on the right. Within each cluster there is little observed linear association; group separation dominates the pooled coefficient.
All four invented datasets have r rounded to 0.75, and the same coefficient tells four different stories, the
point Francis Anscombe made in 1973 with four datasets of his own. Only the first is a straight line with noise. The
second is a curve that rises and falls back, the third is a cloud with no trend plus one far point, and the fourth is
two groups with no trend inside either. A coefficient cannot tell these apart; a scatter plot does it at a glance, and
the residuals of a trend line do it in numbers. In pandas, students.plot.scatter(x="hours", y="marks") draws one
with matplotlib.
Compare what the matching number hides:
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- The noisy line has a Pearson coefficient that rounds to 0.75. The recorded output and scatter points show the shape the coefficient omits.
- A curve that bends back and a cloud plus one far point also round to 0.75. They require different diagnoses despite that matching display.
- The two-cluster dataset also rounds to 0.75. Plotting groups and the joint shape is necessary before interpreting a pooled line.
Key idea
Checkpoint
Check: Shape beyond a coefficient
Association needs a design before it can identify a cause
A confounder at work. Temperature drives both sales series
Text description of the diagram
Three boxes. At the top is the day's top temperature. Arrows lead from it to two boxes below: ice-cream cups sold, and air conditioners sold. Hotter days sell more of both.
A dashed line without arrowheads joins the two sales boxes. It stands for their strong correlation: on days when one shop sells many ice creams, the other sells many air conditioners. Neither causes the other; both follow the temperature, the confounder.
Two columns can correlate strongly without either one affecting the other. On hot days an ice-cream parlour sells more cups and an appliance shop sells more air conditioners, so over a summer the two sales series rise and fall together. Temperature is a confounder: A variable relevant to both an exposure and an outcome that can mix an observed association with a different causal explanation.: a third variable that drives both. Other routes to a correlation without the causation you might read into it:
- Reverse causation: the effect runs the other way, as when extra classes go with low marks because struggling students are the ones sent to them.
- Shared trends: two series that both grow over time correlate strongly, whatever they measure.
- Selection: the rows were filtered on something related to both columns, which creates or hides a pattern.
Causation needs a design that rules these out, such as an A/B test with random assignment, or an analysis that holds the confounders fixed and says which ones it could not measure.
Trace how the proposed common cause affects both columns.
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- Temperature is the proposed common cause in the synthetic example. Its effect can reach both measured sales columns.
- Both ice-cream and air-conditioner sales respond to that driver, producing association without one sales series causing the other.
- A coefficient alone cannot choose among confounding, reverse direction, trends or selection. A causal interpretation needs a justified design and its assumptions.
Checkpoint
Check: Association and causes
Residuals and the four shapes, checked with numbers Pro: opens with a Pro, Premium or Ultimate pass
A trend line is only as good as the pattern it summarises, and residuals are how to check that pattern without looking. This part rebuilds the four datasets of the figure, computes the checks that tell them apart, and says what to do with each shape.
In this part:
- Four rounded correlations match, while residuals differ
- What to do with each shape, and the numbers that justify it
- Why one row tilted the line: the slip’s share of the sums, worked out
3 diagrams · 1 runnable example (Python) · about 7 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Confounders, shared trends and selection, reproduced in code Pro: opens with a Pro, Premium or Ultimate pass
A correlation between two columns can come entirely from a third variable, from time itself or from the way the rows were chosen. All three are reproduced here in code that runs, with the checks that expose them, and an exercise asks you to judge three claims about a table of flats.
In this part:
- A hidden third variable reproduced: temperature behind two sales series
- Shared growth reproduced: 1,000 unrelated pairs of trending series
- Selection reproduced: a filter that creates a correlation
- Your turn: three claims about 40 flats, checked with numbers
4 diagrams · 6 runnable examples (Python) · about 9 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Practise: decide from the stated evidence
Practice that adapts to you
Practise this lesson
12 questions about 11 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
-
Fresher to mid · Warm-up
When would you use Spearman's correlation instead of Pearson's?
Use Spearman when the relationship rises or falls steadily but not in a straight line, when the data are ranks or ordered categories, or when a few extreme values could dominate. Pearson's r measures how close the points lie to a straight line, so a single outlier or a curve can pull it far from what most of the data shows: in the lesson, one typing slip took it from 0.92 to 0.24. Spearman's ρ is Pearson's r computed on the ranks, so it asks only whether one column tends to rise with the other, and the same slip moved it only to 0.74. Pearson keeps its place when the relationship is straight and the values matter, for example before fitting a straight trend line. When the two disagree a lot, plot the data before reporting either.
-
Fresher to mid
Across 200 cities, the number of app notifications sent and the number of purchases correlate at 0.8. Marketing wants to double notifications everywhere. What do you check before agreeing, and what would you propose instead?
-
Mid to senior
A trend line of monthly sales against advertising spend, fitted on 18 months, has a slope of 3.1. The finance head wants to triple the advertising budget and expects three times the extra sales. What do you tell them?
-
Fresher to mid
Monthly active users and the average support response time have both risen for 30 months, and they correlate at 0.93. A manager concludes that growth is slowing support down. How do you test that claim?
Key takeaways
- Pearson measures linear association; Spearman measures monotonic association on ranks, while unusual points can affect both.
np.polyfit(x, y, 1)returns the slope and the intercept; report the slope in units, inside the range of the data.- Datasets with the same correlation can have very different shapes: plot first.
- Correlation is not causation: confounders, reverse causation, shared trends and selection all create correlations.
Terms in this lesson
- confounder
- A variable relevant to both an exposure and an outcome that can mix an observed association with a different causal explanation.
- residual
- An observed value minus its fitted prediction under a stated model, retaining the outcome’s units.
- Spearman correlation
- A coefficient of monotonic association obtained from paired ranks; unusual rank pairings can still affect it.
References
- pandas.DataFrame.corr (pandas development team (NumFOCUS))
- pandas.Series.corr (pandas development team (NumFOCUS))
- numpy.polyfit (NumPy)
- numpy.corrcoef (NumPy)
- Scatter plot (NIST/SEMATECH e-Handbook of Statistical Methods) (National Institute of Standards and Technology)
- scipy.stats.spearmanr (SciPy)
- scipy.stats.pearsonr (SciPy)
- Graphs in Statistical Analysis (The American Statistician (American Statistical Association))
- pandas.DataFrame.plot.scatter (pandas development team (NumFOCUS))
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress