Data Analysis with Python (pandas, polars, DuckDB) Module 8 – Dates and time series
Growth rates, lags and seasonality
Compute month-on-month and year-on-year growth with shift and pct_change, handle missing values as pandas 3 does, and build weekday and hour profiles.
What you will learn
- Compute period-over-period and year-over-year growth with shift and pct_change
- Explain how pandas 3 handles missing values in pct_change
- Compute and interpret a weekday seasonal index
- Check period coverage and calendar comparison keys before reporting growth
Before you start
What you’ll learn · Before you start
What you will learn
- Compute period-over-period and year-over-year growth with shift and pct_change
- Explain how pandas 3 handles missing values in pct_change
- Compute and interpret a weekday seasonal index
- Check period coverage and calendar comparison keys before reporting growth
Before you start
On this page
A rise from 400 orders to 500 is 25%, but nine days against a whole month is not a comparable growth rate. A
growth rate: A relative change computed as the current value divided by a stated earlier base, minus one; a percentage display multiplies that fraction by 100. divides the current value by an earlier base and subtracts one. A
lag: An earlier observation paired with a current one under a stated offset, such as the previous row or the same calendar month a year earlier. places that earlier observation beside the current one: one month for month-on-month
growth, twelve months for year-on-year growth on monthly data. shift and pct_change count rows by default, so
calendar comparisons need the intended period keys and gaps retained in position.
seasonality: A pattern that recurs at a position in a calendar or other cycle, such as weekday or month; a period comparison can mix this pattern with growth. is a repeating pattern by calendar position, such as weekday or hour. Matching months a year apart can control a stable month pattern, but it does not guarantee matched holidays or business coverage. Keep a running month out of a whole-month headline, and leave missing values explicit unless a declared replacement is justified.
Pair the intended periods before dividing
Growth rates express a change relative to a specified earlier base, rather than the number of orders added. Growth rates are comparable only when the source units, period keys and observed coverage match the question being asked.
The examples use two years of synthetic daily orders from an online grocery shop, exported on the ninth day of a month, so the last month is still running.
"""Daily orders of a synthetic online grocery shop, from a September to the 9th day of an October two years later."""
import numpy as np
import pandas as pd
def daily_orders():
days = pd.date_range("2024-09-01", "2026-10-09", freq="D")
months = (days.year - 2024) * 12 + days.month - 9 # 0 for the first September
weekday = np.array([1.00, 0.95, 0.95, 1.00, 1.10, 1.30, 1.25])[days.dayofweek]
rng = np.random.default_rng(12)
noise = rng.normal(1.0, 0.04, len(days))
return pd.Series(np.round(200 * 1.015 ** months * weekday * noise).astype(int), index=days, name="orders") """Flag partial months and keep every calendar position before computing whole-month growth."""
import pandas as pd
from orders import daily_orders
def monthly_table(orders):
"""Monthly totals and coverage from one observation per naive local midnight."""
index = orders.index
if (not isinstance(index, pd.DatetimeIndex) or index.tz is not None
or index.hasnans or index.has_duplicates or not index.equals(index.normalize())):
raise ValueError("use unique date-only keys on the declared reporting calendar")
table = pd.DataFrame({
"orders": orders.resample("MS").sum(min_count=1),
"days": orders.resample("MS").count(),
})
table["complete"] = table["days"] == table.index.days_in_month
table["mom"] = table["orders"].pct_change()
table["yoy"] = table["orders"].pct_change(12)
return table
def whole_month_report(monthly):
"""Compute rates on the full calendar grid before selecting complete headline months."""
whole_totals = monthly["orders"].where(monthly["complete"])
full = monthly.loc[monthly["complete"]].copy()
full["mom"] = whole_totals.pct_change().reindex(full.index)
full["yoy"] = whole_totals.pct_change(12).reindex(full.index)
return full
if __name__ == "__main__":
orders = daily_orders()
monthly = monthly_table(orders)
print(f"{len(monthly)} months, the first and the last three:")
shown = pd.concat([monthly.head(1), monthly.tail(3)])
print(shown.to_string(formatters={"mom": "{:+.1%}".format, "yoy": "{:+.1%}".format}))
full = whole_month_report(monthly)
last = full.iloc[-1]
print(f"\nLast complete month: {full.index[-1]:%b %Y}, {last['orders']:,} orders, MoM {last['mom']:+.1%}, YoY {last['yoy']:+.1%}")
earlier = full.index[-1] - pd.DateOffset(years=1)
if earlier in monthly.index and monthly.loc[earlier, "complete"]:
print("YoY compares it with", f"{earlier:%b %Y}:", f"{monthly.loc[earlier, 'orders']:,}", "orders")
else:
print("YoY base is missing or incomplete at the required calendar month")
middle_gap = pd.Series(10, index=pd.date_range("2026-01-01", "2026-03-31", freq="D"))
middle_gap = middle_gap.drop(pd.Timestamp("2026-02-15"))
controls = whole_month_report(monthly_table(middle_gap))
print("\nOne February day is missing; March's adjacent-month rate is missing:",
bool(pd.isna(controls.loc[pd.Timestamp("2026-03-01"), "mom"]))) Output
26 months, the first and the last three:
orders days complete mom yoy
2024-09-01 6486 30 True NaN NaN
2026-08-01 9556 31 True +3.5% +20.9%
2026-09-01 9147 30 True -4.3% +18.9%
2026-10-01 2785 9 False -69.6% -65.5%
Last complete month: Sep 2026, 9,147 orders, MoM -4.3%, YoY +18.9%
YoY compares it with Sep 2025: 7,696 orders
One February day is missing; March's adjacent-month rate is missing: True
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 monthly_growth.py
Monthly positions, their comparison bases and the running month
Text description of the diagram
The final fourteen monthly positions of the synthetic input run from the earlier September through the later October, displayed in chronological rows of two cards. A dashed divider marks the December-to-January calendar boundary. The prior-month comparison links the later August's 9,556 to September's 9,147. The twelve-position comparison links the earlier September's 7,696 to that same later September. The final October has only nine supplied days and is outlined with dashes and labelled running. The links describe calendar positions, not a license to drop an unknown monthly row.
resample("MS").sum() turns days into months, and counting the days in each month next to its total shows at once
that the last month holds only 9. Its month-on-month rate, −69.6 %, and its year-on-year rate, −65.5 %, describe the
export date, not the shop. September is the latest whole month: 4.3 % below August and 18.9 %
above the September a year earlier. Before selecting headline rows, the program marks every incomplete month’s
whole-period total as unknown on the full calendar grid. Its final drill removes one February day and keeps March’s
adjacent-month rate missing; dropping February first would incorrectly compare March with January.
pct_change(12) is the year-on-year rate because the series has exactly one row
per month, so twelve rows back is twelve months back. pandas builds lags with shift: shift(1) moves every value
one row down, so each row also holds the month before. Lags are just as useful on their own: orders.shift(12) puts
last year’s value beside this year’s in the same row, which is often clearer in a report than a percentage.
Keep the incomplete month on the calendar grid, then inspect the base selected for each headline rate.
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- Name the source calendar. The synthetic daily series becomes one monthly row per calendar month. Its last exported month contains only nine observed days, so it is not a whole-month headline.
- Compare coverage first. The running month shows −69.6% month on month and −65.5% year on year. These ratios are computed, but they compare partial coverage with complete months.
- Use the intended period keys. The latest whole September has 9,147 orders, −4.3% from August and +18.9% from the same September a year earlier. Keep incomplete whole-month totals unknown before the lag, and select complete headline rows afterwards.
Key idea
Checkpoint
Check: Calendar lags and growth
Keep missing periods in their calendar position
"""pct_change in pandas 3 when a value is missing: no silent filling."""
import pandas as pd
orders = pd.Series([100.0, None, 110.0], index=["Jan", "Feb", "Mar"])
print("pct_change(): ", orders.pct_change().tolist())
try:
orders.pct_change(fill_method="ffill")
except ValueError as error:
print("fill_method='ffill': ", "ValueError:", error)
print("ffill().pct_change():", orders.ffill().pct_change().round(3).tolist())
print("shift(1) by hand: ", (orders / orders.shift(1) - 1).tolist()) Output
pct_change(): [nan, nan, nan] fill_method='ffill': ValueError: fill_method must be None; got fill_method='ffill'. ffill().pct_change(): [nan, 0.0, 0.1] shift(1) by hand: [nan, nan, nan]
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pct_change_missing.py
In pandas 3, pct_change() does not fill missing values first: the month after a gap compares with a missing value
and is missing too. The old fill_method="ffill" now raises a ValueError: the pandas 3 documentation says the
argument must be None and will be removed. If carrying the last known value forward is right for your data, say so
with ffill().pct_change(), which reports 0 % for the missing month and 10 % for the next. Often it is not justified: a
missing month of orders might be a missing export, and a growth rate built on an invented value is worse than a gap.
The same result comes from the definition, orders / orders.shift(1) - 1, which is worth knowing because it works the
same way in every tool. Growth rates inherit gaps in their comparison values. Keep missing calendar periods in position; dropping rows
before calculating a calendar rate would change which periods are compared. A last-observation comparison can be
valid too, but it must be labelled as that different metric.
Retain the unknown middle month and contrast a missing comparison with an explicit carried-value policy.
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- Retain the missing position. The middle row represents an unknown month. Removing its position would pair the third month's 110 with the first month's 100 as if they were adjacent months.
- Read pandas 3's result. pct_change does not fill first: the missing current value prevents one rate and the missing prior value prevents the next. All three output positions are missing in this input.
- Separate policy from measurement. Forward filling invents a carried value and yields zero then ten percent. It is a declared transformation, not evidence of what happened in the unknown month.
Key idea
Checkpoint
Check: Missing comparison values
State the denominator of a seasonal index
Seasonality describes recurring differences by a position in a calendar cycle, such as the weekday. Seasonality can coexist with growth, changing exposure and unusual events, so a repeating profile is a descriptive comparison rather than a causal explanation.
"""A weekday profile: the average orders of each weekday, as a share of the overall daily average."""
import pandas as pd
from orders import daily_orders
orders = daily_orders()["2026-07-01":"2026-09-30"] # one recent quarter, 13 weeks and 1 day
profile = orders.groupby(orders.index.day_name()).agg(days="count", mean="mean")
profile = profile.reindex(["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"])
profile["index"] = profile["mean"] / orders.mean()
print(profile.round({"mean": 1, "index": 2}).to_string())
print(f"\nBusiest day: {profile['index'].idxmax()}, {profile['index'].max() - 1:+.0%} against an average day") Output
days mean index Monday 13 281.4 0.93 Tuesday 13 264.2 0.87 Wednesday 14 269.4 0.89 Thursday 13 282.2 0.93 Friday 13 310.2 1.02 Saturday 13 366.3 1.21 Sunday 13 354.2 1.17 Busiest day: Saturday, +21% against an average day
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 weekday_profile.py
A seasonal profile averages a value by its position in the cycle, here the weekday, and divides by the overall
average, so an index above 1 marks a day busier than average. Saturday’s 1.21 means 21 % more orders than an average
day, and Tuesday’s 0.87 means 13 % fewer. The days column matters: the quarter starts on a Wednesday, so Wednesdays
appear 14 times and every other day 13, and a profile built on uneven counts should say so. Month-of-year profiles use month positions instead of weekday positions, and more observed cycles help reveal
whether an unusual month dominates. Hour profiles summarize by hour on the reporting clock. They need comparable
observed coverage per hour, rather than treating an unobserved hour as a measured zero.
Count the observed weekdays first, then compare each mean with the stated overall daily denominator.
The drawing shows the whole scene; its 3 steps are listed below.
Every step as text
- Count the supplied days. The input has fourteen Wednesdays and thirteen of every other weekday. Each weekday mean therefore has its own observed-day denominator.
- Keep mean and exposure separate. A weekday profile averages the supplied daily orders by weekday, rather than treating its total as evidence of a larger typical day.
- Read the relative level. Saturday's mean divided by the overall daily mean is 1.21 after rounding, about 21% above an average observed day. Tuesday's 0.87 is about 13% below that same denominator.
Key idea
Checkpoint
Check: Seasonal indices
Growth that misleads: tiny bases, running months and missing months Pro: opens with a Pro, Premium or Ultimate pass
A growth rate is a ratio, and a ratio misleads when its base is tiny, when the two periods are not the same length, or when the rows are not the months they seem to be. This part reproduces each case and the check that catches it.
In this part:
- Report the base beside a relative change
- Match exposure before comparing a running month
- Restore calendar positions before applying row lags
3 diagrams · 3 runnable examples (Python) · about 7 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Seasonal profiles that hold up: weekday by hour, and a festival that moves Pro: opens with a Pro, Premium or Ultimate pass
Seasonal patterns decide staffing, stock and marketing, so a profile has to separate the pattern from noise and from the calendar’s own quirks. This part builds a weekday-by-hour profile and handles a festival whose month changes from year to year.
In this part:
- Compare the same completed weekday-hour band
- Align a moving festival with its own season
3 diagrams · 3 runnable examples (Python) · about 6 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Compound growth and a monthly KPI table, with numbers Pro: opens with a Pro, Premium or Ultimate pass
Monthly rates are easy to compute and easy to combine wrongly: they multiply, they do not add, and their simple average overstates growth. This part works through compounding with numbers, then builds a monthly KPI table that refuses partial and missing months before it computes anything.
In this part:
- Multiply growth factors over matched periods
- Validate the expected daily calendar before rates
- Match each dashboard column to its question
3 diagrams · 2 runnable examples (Python) · about 7 minutes
Opening this part with your pass…
This part opens with any MySmartCoPilot pass.
- Pro pass · 1 week₹79$2.99
- 30 days₹179$5.99
- 1 year₹1,699$45
One payment; a pass never renews. The price shown is the total you pay: no fees or charges are added at checkout.
Passes can’t be bought right now, so this part can’t be opened at the moment. The free part above stands on its own. If you have a pass, enter your licence key.
It also opens every other in-depth part in Learn and every Pro tool. Sales are final, except where the law gives you a right to withdraw, for example in the EU and the UK (refund policy).
Practise whole-month growth
The exercise computes month-on-month and year-on-year growth with the standard library, refusing the month that is still running and any month with missing days.
Exercise · Medium · Python
Month-on-month and year-on-year growth for whole months
A shop exports one row per day: the day as text in YYYY-MM-DD form and the number of orders. The export may have been taken in the middle of a month, and some months may be missing altogether. Write monthly_growth(rows) in kpi.py, where rows is a list of (day, orders) pairs in any order, with at most one row per day.
A month counts only when it is whole, with a row for every one of its days. Return one tuple per whole month, in calendar order: (month, total, mom, yoy), with the month as "YYYY-MM". mom is the change from the calendar month before and yoy the change from the same month a year earlier, both in per cent and rounded to 1 decimal place, so a rise from 310 to 336 orders is 8.4. Give None instead when the month you compare with is not a whole month in the data, or when its total is 0.
Starter code · kpi.py
from calendar import monthrange
from datetime import date
def monthly_growth(rows):
"""[(month, total, mom, yoy), ...] for every whole month, in calendar order.
rows is a list of (day, orders) pairs, such as ("2026-07-14", 31).
mom and yoy are changes in per cent, rounded to 1 decimal place, or None.
"""
# Replace this line with your code.
return [] The sample tests · test_kpi.py
from calendar import monthrange
from kpi import monthly_growth
def month(year, number, per_day, days=None):
"""Rows for the first `days` days of a month (all of them by default), each with `per_day` orders."""
last = monthrange(year, number)[1] if days is None else days
return [(f"{year}-{number:02d}-{d:02d}", per_day) for d in range(1, last + 1)]
def test_partial_month_is_left_out():
"""leaves out the month that is still running"""
rows = month(2026, 1, 10) + month(2026, 2, 12) + month(2026, 3, 15, days=9)
assert monthly_growth(rows) == [("2026-01", 310, None, None), ("2026-02", 336, 8.4, None)]
def test_year_on_year():
"""compares each month with the same month a year earlier"""
rows = [r for m in range(1, 13) for r in month(2025, m, 10)] + month(2026, 1, 11)
result = monthly_growth(rows)
assert len(result) == 13
assert result[-1] == ("2026-01", 341, 10.0, 10.0)
assert all(yoy is None for _, _, _, yoy in result[:12])
def test_missing_month():
"""does not compare a month with the one two months back when the month between is missing"""
rows = month(2026, 1, 10) + month(2026, 3, 10)
assert monthly_growth(rows) == [("2026-01", 310, None, None), ("2026-03", 310, None, None)]
def test_zero_base():
"""gives None, not an error, when the month before had no orders"""
rows = month(2026, 4, 0) + month(2026, 5, 4)
assert monthly_growth(rows) == [("2026-04", 0, None, None), ("2026-05", 124, None, None)]
def test_rows_in_any_order():
"""sorts the months itself"""
rows = list(reversed(month(2026, 6, 20) + month(2026, 7, 21)))
assert monthly_growth(rows) == [("2026-06", 600, None, None), ("2026-07", 651, 8.5, None)]
def test_missing_day_does_not_shift_the_comparison():
"""An incomplete February is excluded, but March is not compared with January."""
february = month(2026, 2, 12)
february = [row for row in february if row[0] != "2026-02-15"]
rows = month(2026, 1, 10) + february + month(2026, 3, 15)
assert monthly_growth(rows) == [("2026-01", 310, None, None), ("2026-03", 465, None, None)]
def test_incomplete_prior_year_has_no_yoy():
"""A known partial prior-year month is not a whole-month comparison base."""
earlier = [row for row in month(2025, 1, 10) if row[0] != "2025-01-15"]
assert monthly_growth(earlier + month(2026, 1, 11)) == [("2026-01", 341, None, None)] A hint
Add the orders up per (year, month) key and count the rows of each key at the same time. A month is whole when its count equals monthrange(year, month)[1], the number of days in it. The month before (year, month) is (year, month - 1), except in January, where it is (year - 1, 12); the same month a year earlier is (year - 1, month).
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Practice that adapts to you
Practise this lesson
7 questions about 11 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
-
Fresher to mid · Warm-up
What is the difference between
shift(1)andpct_change()in pandas?Default shift(1) places the preceding row's value beside the current row; the first position has no prior value. pct_change computes current divided by that lag minus one, so 400 to 500 gives 0.25, displayed as 25%. A twelve-row shift represents the same calendar month a year earlier only when the row positions retain the monthly grid. pandas 3 does not fill unknown values first. Preserve missing calendar positions for a calendar metric; dropping a row changes the pair to a last-observation comparison. A forward fill is an explicit replacement policy rather than a recovered measurement, and needs justification.
-
Fresher to mid
Orders in a city the company entered last month grew 200 % month on month, the highest of all cities, and the regional manager wants to announce it. What do you check, and how would you report the city?
-
Fresher to mid
Nine days into October, a dashboard shows revenue down 68 % from September and orders down 70 %, and a director asks what went wrong. What do you tell them, and how do you change the dashboard?
-
Mid to senior
A sweets chain's October sales are 19 % below last October, and November's are 53 % above last November. The finance team asks whether the business is shrinking or booming. How do you answer, and what comparison do you set up for next year?
Key takeaways
- A growth rate compares a current value with a stated earlier base; pct_change returns a fraction rather than percentage units.
- Calendar growth needs the intended period keys and missing periods retained in their positions.
- pandas 3 does not silently fill missing comparison values, so a replacement must have a declared meaning.
- Whole-month headlines require comparable coverage; a yearly match does not remove every calendar or business difference.
- A seasonal profile compares observed averages by cycle position with a stated overall average and exposure.
Terms in this lesson
- growth rate
- A relative change computed as the current value divided by a stated earlier base, minus one; a percentage display multiplies that fraction by 100.
- lag
- An earlier observation paired with a current one under a stated offset, such as the previous row or the same calendar month a year earlier.
- seasonality
- A pattern that recurs at a position in a calendar or other cycle, such as weekday or month; a period comparison can mix this pattern with growth.
References
- pandas.Series.pct_change (pandas development team (NumFOCUS))
- pandas.Series.shift (pandas development team (NumFOCUS))
- Time series / date functionality (pandas user guide) (pandas development team (NumFOCUS))
- pandas.pivot_table (pandas development team (NumFOCUS))
- What's new in 3.0.0 (pandas) (pandas development team (NumFOCUS))
- Holiday orders of the Department of Personnel and Training (Department of Personnel and Training, Government of India)
- Calendar month lengths (Python Software Foundation)
- assert and optimized execution (Python Software Foundation)
- Central-office holiday calendar: first simulation year (Department of Personnel and Training, Government of India)
- Central-office holiday calendar: second simulation year (Department of Personnel and Training, Government of India)
- Central-office holiday calendar: subsequent year (Department of Personnel and Training, Government of India)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress