Data Analysis with Python (pandas, polars, DuckDB) Module 7 – Reshaping: tidy data, melt, pivot and MultiIndex
Tidy data: long vs wide, melt and pivot
Reshape pandas tables between wide and long with melt and pivot, fix the "duplicate entries" error, and choose the layout each analysis needs.
What you will learn
- Explain the three tidy-data rules and recognise the common messy layouts
- Convert wide tables to long ones with melt and back with pivot
- Handle the duplicate entries that make pivot fail
- Split column names that hold two variables, such as maths_t1
- Choose the long or the wide layout for analysis, charts and reports
What you’ll learn
What you will learn
- Explain the three tidy-data rules and recognise the common messy layouts
- Convert wide tables to long ones with melt and back with pivot
- Handle the duplicate entries that make pivot fail
- Split column names that hold two variables, such as maths_t1
- Choose the long or the wide layout for analysis, charts and reports
On this page
Tidy data is a way of laying out a table so that every variable has its own column, every observation has its own
row, and every kind of observation has its own table. A wide table breaks those rules when its column headers are
values rather than names of variables: one column per month, per subject or per year. A long table holds the same
numbers with one row per measurement and a column that says what was measured. In pandas, melt turns a wide table
into a long one, and pivot turns a long table back into a wide one; pivot_table does the same job as pivot but
combines rows that land in the same cell. Group-by, charts and joins work best on long data, while people reading a
report usually want it wide, so analysts move between the two layouts all the time.
This lesson reshapes one small table of rain-gauge readings in both directions, with every run recorded. It melts a
table with a column per month into one reading per row, answers three questions on the long table, pivots it back,
reproduces the error pivot raises when two rows compete for one cell, and fixes that with a rule instead of an
average. It then splits column names that hold two variables, shows the same reshapes in polars and DuckDB, and ends
with an exercise on exam marks. You should already be able to load a CSV file into a DataFrame and run a groupby.
The three rules of tidy data
Hadley Wickham’s paper “Tidy Data” in the Journal of Statistical Software gave the layout its name and its three rules. Put in plain words:
- Each variable is a column. A column holds one kind of value: a gauge name, a year, a month or a reading in millimetres, never a mixture.
- Each observation is a row. One row holds everything recorded about one measurement: which gauge, which year, which month and how much rain.
- Each kind of observational unit is a table. Facts about gauges, such as their location and height, belong in a table of gauges; the readings belong in a table of readings. A join brings them together when a question needs both.
The paper also describes the five most common ways real tables break these rules, and each has a standard pandas remedy:
- Column headers are values, not variable names, such as one column per month.
meltfixes it, and it is the case this lesson starts with. - One column holds two variables, such as
maths_t1naming both a subject and a term. Melt first, then split the old names withstr.splitorstr.extract, or usepd.wide_to_long. - Variables sit in both rows and columns, such as a sheet with one row per gauge and measure (minimum, maximum) and one column per day. Melt the day columns, then pivot the measure values into columns.
- Several kinds of observational unit share one table, such as gauge locations repeated on every reading. Move the
repeated facts into their own table with
drop_duplicatesand keep a key to join on. - One kind of unit is spread over several tables, such as one file per year. Stack them with
pd.concat.
Whether a table is tidy depends on the question. For “how much rain fell each month?”, the month is a variable, and a
table with a column per month is wide. For a report that compares the four months of one year side by side, the same
wide table is exactly the right shape. The rules describe the layout that analysis code wants; reports and
spreadsheets often want something else, and pivot produces it on demand.
melt: from wide to long
Here are the readings as they might arrive from a spreadsheet: three gauges, two years, and one column per month from June to September. The readings are invented for this lesson:
gauge,year,JUN,JUL,AUG,SEP
Airport,2024,182.4,401.0,436.2,210.5
Airport,2025,140.9,455.3,298.0,260.1
Lakeside,2024,201.7,433.8,380.4,190.2
Lakeside,2025,165.0,470.2,322.6,240.8
Hill Fort,2024,240.3,520.6,410.9,230.0
Hill Fort,2025,198.5,560.1,350.7,275.4 melt keeps the columns you name in id_vars, which identify a reading, and turns every other column into rows. The
old column headers go into a new column named by var_name, and the cells go into a column named by value_name:
import pandas as pd
# Three rain gauges, two years, one column per month (invented readings, in millimetres).
wide = pd.read_csv("rain_wide.csv")
print(wide)
print()
long = wide.melt(
id_vars=["gauge", "year"], # columns that say which reading this is: kept as they are
var_name="month", # the old column headers become the values of this column
value_name="rain_mm", # the old cells become the values of this column
)
print(f"wide: {wide.shape[0]} rows x {wide.shape[1]} columns")
print(f"long: {long.shape[0]} rows x {long.shape[1]} columns")
print(long.head(8))
print()
print(long.dtypes) Output
gauge year JUN JUL AUG SEP
0 Airport 2024 182.4 401.0 436.2 210.5
1 Airport 2025 140.9 455.3 298.0 260.1
2 Lakeside 2024 201.7 433.8 380.4 190.2
3 Lakeside 2025 165.0 470.2 322.6 240.8
4 Hill Fort 2024 240.3 520.6 410.9 230.0
5 Hill Fort 2025 198.5 560.1 350.7 275.4
wide: 6 rows x 6 columns
long: 24 rows x 4 columns
gauge year month rain_mm
0 Airport 2024 JUN 182.4
1 Airport 2025 JUN 140.9
2 Lakeside 2024 JUN 201.7
3 Lakeside 2025 JUN 165.0
4 Hill Fort 2024 JUN 240.3
5 Hill Fort 2025 JUN 198.5
6 Airport 2024 JUL 401.0
7 Airport 2025 JUL 455.3
gauge str
year int64
month str
rain_mm float64
dtype: object
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 melt_rain.py
One table, two layouts. melt goes from wide to long, pivot goes back
Text description of the diagram
The upper box is the wide layout: one row per gauge and year, 6 rows and 6 columns. Its columns are gauge, year, JUN, JUL, AUG and SEP, so the first row reads Airport, 2024, 182.4, 401.0, 436.2 and 210.5.
The lower box is the long, tidy layout: one row per reading, 24 rows and 4 columns. Its columns are gauge, year, month and rain_mm, so the first row of the wide table becomes four rows: Airport, 2024, JUN, 182.4; Airport, 2024, JUL, 401.0; Airport, 2024, AUG, 436.2; and Airport, 2024, SEP, 210.5.
An arrow from wide to long is labelled melt, which turns the month headers into values of the month column. An arrow from long to wide is labelled pivot, which turns the month values back into column headers.
The arithmetic is worth checking every time you melt: each of the 6 rows had 4 month columns, so the long table has
6 × 4 = 24 rows, and the two identifier columns are repeated on each of them. A different count means a column you
expected to keep was melted, or one you wanted to melt was named in id_vars. Three more details matter in practice:
- Without
value_vars, every column not inid_varsis melted. Name the value columns explicitly when the table also holds columns such as a comment or a total that should not become readings. - The rows come out column by column: all June readings first, then all July readings. Sort the long table if the order matters to you.
- The result gets a fresh index from 0 to 23, because
ignore_index=Trueis the default. Passignore_index=Falseto keep the original row labels.
Why the long layout pays off
Each of these questions is one group-by on the long table:
import pandas as pd
wide = pd.read_csv("rain_wide.csv")
long = wide.melt(id_vars=["gauge", "year"], var_name="month", value_name="rain_mm")
# 1. Season total for each gauge and year: one group-by on the long table.
totals = long.groupby(["gauge", "year"])["rain_mm"].sum()
print(totals)
print()
# 2. The wettest month of each gauge and year: the row with the largest reading per group.
wettest = long.loc[long.groupby(["gauge", "year"])["rain_mm"].idxmax()]
print(wettest.to_string(index=False))
print()
# 3. The average of each month over every gauge and year: group by the column melt created.
print(long.groupby("month", sort=False)["rain_mm"].mean().round(1))
print()
# The season total from the wide table works too, but every month column has to be named.
print(wide.set_index(["gauge", "year"])[["JUN", "JUL", "AUG", "SEP"]].sum(axis=1).round(1)) Output
gauge year
Airport 2024 1230.1
2025 1154.3
Hill Fort 2024 1401.8
2025 1384.7
Lakeside 2024 1206.1
2025 1198.6
Name: rain_mm, dtype: float64
gauge year month rain_mm
Airport 2024 AUG 436.2
Airport 2025 JUL 455.3
Hill Fort 2024 JUL 520.6
Hill Fort 2025 JUL 560.1
Lakeside 2024 JUL 433.8
Lakeside 2025 JUL 470.2
month
JUN 188.1
JUL 473.5
AUG 366.5
SEP 234.5
Name: rain_mm, dtype: float64
gauge year
Airport 2024 1230.1
2025 1154.3
Lakeside 2024 1206.1
2025 1198.6
Hill Fort 2024 1401.8
2025 1384.7
dtype: float64
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 long_questions.py
The season total, the wettest month and the average of each month all come from the same two columns, month and
rain_mm. The last result shows the wide alternative: it works, but the code has to name every month column. With
a column per month of a 20-year record, that is 240 names, and every new month means changing the code, while the
long table simply gains rows. groupby("month", sort=False) kept the months in the order they first appear in the
data, which here is the calendar order; with the default sort=True they would come out alphabetically.
pivot: from long back to wide
pivot is the reverse of melt. index names the columns that identify a row of the result, columns names the
column whose values become the new column headers, and values names the column that fills the cells:
import pandas as pd
long = pd.read_csv("rain_wide.csv").melt(id_vars=["gauge", "year"], var_name="month", value_name="rain_mm")
# Back to one row per gauge and year, one column per month.
wide = long.pivot(index=["gauge", "year"], columns="month", values="rain_mm")
print(wide) # the new columns are sorted as text
print()
wide = wide[["JUN", "JUL", "AUG", "SEP"]] # calendar order again
print(wide)
print()
# A plain table for a CSV file or a spreadsheet: the index becomes columns again.
flat = wide.reset_index()
flat.columns.name = None # "month" was the name of the column labels
print(flat)
print()
# Another layout from the same long table: season totals, gauges down, years across.
report = long.pivot_table(index="gauge", columns="year", values="rain_mm", aggfunc="sum")
print(report) Output
month AUG JUL JUN SEP
gauge year
Airport 2024 436.2 401.0 182.4 210.5
2025 298.0 455.3 140.9 260.1
Hill Fort 2024 410.9 520.6 240.3 230.0
2025 350.7 560.1 198.5 275.4
Lakeside 2024 380.4 433.8 201.7 190.2
2025 322.6 470.2 165.0 240.8
month JUN JUL AUG SEP
gauge year
Airport 2024 182.4 401.0 436.2 210.5
2025 140.9 455.3 298.0 260.1
Hill Fort 2024 240.3 520.6 410.9 230.0
2025 198.5 560.1 350.7 275.4
Lakeside 2024 201.7 433.8 380.4 190.2
2025 165.0 470.2 322.6 240.8
gauge year JUN JUL AUG SEP
0 Airport 2024 182.4 401.0 436.2 210.5
1 Airport 2025 140.9 455.3 298.0 260.1
2 Hill Fort 2024 240.3 520.6 410.9 230.0
3 Hill Fort 2025 198.5 560.1 350.7 275.4
4 Lakeside 2024 201.7 433.8 380.4 190.2
5 Lakeside 2025 165.0 470.2 322.6 240.8
year 2024 2025
gauge
Airport 1230.1 1154.3
Hill Fort 1401.8 1384.7
Lakeside 1206.1 1198.6
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pivot_rain.py
Four things in that output are easy to miss:
- The months came back as AUG, JUL, JUN, SEP.
pivotsorts the new column labels, and text sorts alphabetically. Select the columns in the order you want, as the second step does, or turnmonthinto an ordered categorical before pivoting so that its sort order is the calendar order. - The result has the two identifier columns as a two-level row index, and the column labels carry the name
month.reset_index()turns the index back into ordinary columns, and settingcolumns.name = Noneremoves the leftover label, which gives a plain table to save as CSV. - The same long table produced a second layout without any further reshaping:
pivot_tablewithaggfunc="sum"added up the four months of each gauge and year and placed the years across. - Because
meltandpivotare exact inverses on data with one value per cell, the flat table equals the original wide file, apart from the order of the rows.
When pivot fails: duplicate entries
pivot needs every combination of index and column values to appear at most once, because each combination becomes
exactly one cell. Real data breaks that assumption more often than you would expect: a record sent twice, a correction
appended instead of replacing the original, or a join earlier in the pipeline that matched one row to two. Here the
Lakeside gauge’s July reading for 2025 arrives a second time, corrected from 470.2 to 407.2:
import pandas as pd
long = pd.read_csv("rain_wide.csv").melt(id_vars=["gauge", "year"], var_name="month", value_name="rain_mm")
# Lakeside's July 2025 reading arrives a second time, corrected from 470.2 to 407.2.
correction = pd.DataFrame({"gauge": ["Lakeside"], "year": [2025], "month": ["JUL"], "rain_mm": [407.2]})
readings = pd.concat([long, correction], ignore_index=True)
try:
readings.pivot(index=["gauge", "year"], columns="month", values="rain_mm")
except ValueError as err:
print(f"pivot failed: {err}")
print()
keys = ["gauge", "year", "month"]
print(readings[readings.duplicated(keys, keep=False)]) # every row whose key appears more than once
print()
# How many readings landed in each cell: anything above 1 is a duplicate.
print(readings.pivot_table(index=["gauge", "year"], columns="month", values="rain_mm", aggfunc="size"))
print()
# pivot_table with a mean "works", and blends the wrong reading with its correction.
mean = readings.pivot_table(index=["gauge", "year"], columns="month", values="rain_mm", aggfunc="mean")
print(f"mean of both readings: {mean.loc[('Lakeside', 2025), 'JUL']:.1f}")
# The rule for this data: a later reading replaces an earlier one.
latest = readings.drop_duplicates(keys, keep="last")
fixed = latest.pivot(index=["gauge", "year"], columns="month", values="rain_mm")
print(f"latest reading: {fixed.loc[('Lakeside', 2025), 'JUL']:.1f}")
print(f"rows: {len(readings)} readings, {len(latest)} after dropping the replaced one") Output
pivot failed: Index contains duplicate entries, cannot reshape
gauge year month rain_mm
9 Lakeside 2025 JUL 470.2
24 Lakeside 2025 JUL 407.2
month AUG JUL JUN SEP
gauge year
Airport 2024 1 1 1 1
2025 1 1 1 1
Hill Fort 2024 1 1 1 1
2025 1 1 1 1
Lakeside 2024 1 1 1 1
2025 1 2 1 1
mean of both readings: 438.7
latest reading: 407.2
rows: 25 readings, 24 after dropping the replaced one
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pivot_duplicates.py
pivot refuses with ValueError: Index contains duplicate entries, cannot reshape, and that refusal is useful: it
tells you the data does not mean what you assumed. Three steps resolve it:
- Find the duplicates.
duplicated(keys, keep=False)marks every row whose key appears more than once, so you see both versions side by side, not only the second one. - Count the rows per cell.
pivot_table(..., aggfunc="size")shows how many rows would land in each cell. Every number above 1 is a cell that needs a decision. - Apply the rule the data calls for. Here a later reading replaces an earlier one, so
drop_duplicates(keys, keep="last")keeps the correction, andpivotworks again.
The tempting fix is pivot_table, which never raises on duplicates because it aggregates them. With its default mean,
the Lakeside cell became 438.7, a number that neither the gauge nor the correction ever reported. Aggregate only when
several rows per cell are expected and the aggregate means something, such as a sum of daily readings into a monthly
total; when duplicates are mistakes, remove them by a rule you can state.
| Criterion | pivot | pivot_table |
|---|---|---|
| Repeats | Raises an error | Combined by aggfunc |
| Cells | Values as they are | Mean, unless you set aggfunc |
| Types | Any | What aggfunc accepts |
| Extras | None | fill_value, margins |
| When to choose | One value per cell | Totals, means, counts |
Long or wide: choosing a layout
Neither layout is better in general; each suits different jobs. A practical rule is to keep data long while you clean, check and analyse it, and to pivot to wide at the end, for the table a person will read:
| Criterion | Long | Wide |
|---|---|---|
| Group-by | One column | A column per value |
| New data | More rows | New columns, new code |
| Charts | What plots expect | One line per column |
| Reading | Hard by eye | Side by side |
| Models | Pivot first | A row per case |
| When to choose | Cleaning, analysis, charts | Reports, sheets, models |
Column names that hold two variables
Exam sheets often have a column per subject and term, such as maths_t1 and science_t2. Each name holds two
variables, and a tidy table needs a column for each. Melt the marks into rows, then split the old column names:
import pandas as pd
# One row per student; each column name holds two variables, the subject and the term.
marks = pd.DataFrame({
"roll_no": ["R01", "R02", "R03"],
"maths_t1": [78, 64, 91],
"maths_t2": [82, 70, 88],
"science_t1": [69, 75, 85],
"science_t2": [74, 72, 90],
})
long = marks.melt(id_vars="roll_no", var_name="column", value_name="marks")
long[["subject", "term"]] = long["column"].str.split("_", expand=True) # "maths_t1" -> "maths", "t1"
tidy = long.drop(columns="column")[["roll_no", "subject", "term", "marks"]]
print(tidy.sort_values(["roll_no", "subject", "term"]).to_string(index=False))
print()
# wide_to_long does both steps when the names follow a "stub_suffix" pattern.
both = pd.wide_to_long(marks, stubnames=["maths", "science"], i="roll_no", j="term", sep="_", suffix=r"t\d")
print(both)
print()
# A report layout: one row per student and subject, the terms side by side.
report = tidy.pivot(index=["roll_no", "subject"], columns="term", values="marks").reset_index()
report.columns.name = None
report["change"] = report["t2"] - report["t1"]
print(report.to_string(index=False)) Output
roll_no subject term marks
R01 maths t1 78
R01 maths t2 82
R01 science t1 69
R01 science t2 74
R02 maths t1 64
R02 maths t2 70
R02 science t1 75
R02 science t2 72
R03 maths t1 91
R03 maths t2 88
R03 science t1 85
R03 science t2 90
maths science
roll_no term
R01 t1 78 69
R02 t1 64 75
R03 t1 91 85
R01 t2 82 74
R02 t2 70 72
R03 t2 88 90
roll_no subject t1 t2 change
R01 maths 78 82 4
R01 science 69 74 5
R02 maths 64 70 6
R02 science 75 72 -3
R03 maths 91 88 -3
R03 science 85 90 5
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 two_variables.py
str.split("_", expand=True) splits every name at the underscore and returns the parts as columns, which are assigned
to subject and term in one step. When the names follow a fixed stub-and-suffix pattern, pd.wide_to_long does
the melt and the split at once: it keeps one column per stub (here maths and science) and turns the suffix into a
term level of the index, which is a halfway layout, still a column per subject. For names that do not split on a
single character, such as MathsTerm1, use str.extract with a regular expression. The last step pivots the terms
back into columns, because comparing a student’s two terms is easiest side by side, and computes the change.
The same reshapes in polars and DuckDB
polars and DuckDB both reshape with their own names for the same two operations:
import duckdb
import polars as pl
# The same wide table, reshaped by polars and by DuckDB's SQL.
wide = pl.read_csv("rain_wide.csv")
long = wide.unpivot(index=["gauge", "year"], variable_name="month", value_name="rain_mm")
print(f"polars unpivot: {long.height} rows")
back = long.pivot(on="month", index=["gauge", "year"], values="rain_mm")
print(back) # polars keeps the order in which the months first appear
print()
sql_long = duckdb.sql("UNPIVOT 'rain_wide.csv' ON JUN, JUL, AUG, SEP INTO NAME month VALUE rain_mm")
print(f"DuckDB UNPIVOT: {len(sql_long.fetchall())} rows")
totals = duckdb.sql("""
PIVOT (UNPIVOT 'rain_wide.csv' ON JUN, JUL, AUG, SEP INTO NAME month VALUE rain_mm)
ON year USING round(sum(rain_mm), 1) GROUP BY gauge ORDER BY gauge
""")
print(totals) Output
polars unpivot: 24 rows shape: (6, 6) ┌───────────┬──────┬───────┬───────┬───────┬───────┐ │ gauge ┆ year ┆ JUN ┆ JUL ┆ AUG ┆ SEP │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═══════════╪══════╪═══════╪═══════╪═══════╪═══════╡ │ Airport ┆ 2024 ┆ 182.4 ┆ 401.0 ┆ 436.2 ┆ 210.5 │ │ Airport ┆ 2025 ┆ 140.9 ┆ 455.3 ┆ 298.0 ┆ 260.1 │ │ Lakeside ┆ 2024 ┆ 201.7 ┆ 433.8 ┆ 380.4 ┆ 190.2 │ │ Lakeside ┆ 2025 ┆ 165.0 ┆ 470.2 ┆ 322.6 ┆ 240.8 │ │ Hill Fort ┆ 2024 ┆ 240.3 ┆ 520.6 ┆ 410.9 ┆ 230.0 │ │ Hill Fort ┆ 2025 ┆ 198.5 ┆ 560.1 ┆ 350.7 ┆ 275.4 │ └───────────┴──────┴───────┴───────┴───────┴───────┘ DuckDB UNPIVOT: 24 rows ┌───────────┬────────┬────────┐ │ gauge │ 2024 │ 2025 │ │ varchar │ double │ double │ ├───────────┼────────┼────────┤ │ Airport │ 1230.1 │ 1154.3 │ │ Hill Fort │ 1401.8 │ 1384.7 │ │ Lakeside │ 1206.1 │ 1198.6 │ └───────────┴────────┴────────┘
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 other_engines.py
In polars, unpivot(index=…) is melt(id_vars=…) and pivot(on=…, index=…, values=…) is pivot with the header
column named on. Unlike pandas, polars kept the months in the order they first appeared instead of sorting them.
DuckDB’s UNPIVOT … ON … INTO NAME … VALUE … and PIVOT … ON … USING … GROUP BY … are SQL statements, here run
directly on the CSV file. PIVOT always aggregates, like pivot_table: the USING clause names the aggregate,
without one DuckDB counts the rows, and first() passes a single value through unchanged. polars raises an error on
duplicates as pandas does, and its aggregate_function argument plays the role of aggfunc.
Version note
In polars, melt is deprecated in favour of unpivot, which takes index instead of id_vars and on instead of
value_vars, and polars 2.0 removes melt altogether, according to its upgrade guide. unpivot behaves the same
on polars 1.33.1, 1.44.2 and the release candidate 2.0.0rc2, so it is the name to use in new code.
Your turn: reshape an exam sheet
A school exports end-of-term marks with one row per student and a column per subject and term. Write two functions:
to_tidy(wide), which returns one row per student, subject and term with the columns roll_no, subject, term
and marks, and to_report(tidy), which returns one row per student and subject with the columns roll_no,
subject, t1, t2 and change:
roll_no,maths_t1,science_t1,english_t1,maths_t2,science_t2,english_t2
R01,78,69,81,82,74,85
R02,64,75,70,70,72,68
R03,91,85,77,88,90,80
R04,55,62,73,61,66,75 import pandas as pd
def to_tidy(wide):
"""One row per student, subject and term, with the columns roll_no, subject, term and marks."""
return wide # replace this line: melt, then split the old column names into subject and term
def to_report(tidy):
"""One row per student and subject, with the columns roll_no, subject, t1, t2 and change (t2 - t1)."""
return tidy # replace this line: pivot the terms into columns, then add the change The checker calls both functions on that sheet and runs seven checks; on the untouched starting point, all seven fail:
import pandas as pd
from reshape_marks import to_report, to_tidy
wide = pd.read_csv("exam_wide.csv")
def check(name, test):
try:
ok = bool(test())
except Exception: # a missing column, for example
ok = False
print("PASS" if ok else "FAIL", name)
return ok
def tidy():
return to_tidy(wide.copy())
def report():
return to_report(tidy())
def mark(roll_no, subject, term):
t = tidy()
return t.loc[(t["roll_no"] == roll_no) & (t["subject"] == subject) & (t["term"] == term), "marks"].item()
results = [
check("tidy: 24 rows (4 students x 3 subjects x 2 terms)", lambda: len(tidy()) == 24),
check("tidy: columns roll_no, subject, term, marks", lambda: list(tidy().columns) == ["roll_no", "subject", "term", "marks"]),
check("tidy: R03 science t2 is 90", lambda: mark("R03", "science", "t2") == 90),
check("tidy: one row per student, subject and term", lambda: not tidy().duplicated(["roll_no", "subject", "term"]).any()),
check("report: 12 rows and the columns roll_no, subject, t1, t2, change", lambda: report().shape == (12, 5) and list(report().columns) == ["roll_no", "subject", "t1", "t2", "change"]),
check("report: R04 maths changed by +6", lambda: report().set_index(["roll_no", "subject"]).loc[("R04", "maths"), "change"] == 6),
check("report: a flat table, t1 and t2 as plain columns", lambda: report().index.nlevels == 1 and report().columns.nlevels == 1 and "t1" in report().columns),
]
print(f"{sum(results)} of {len(results)} checks pass") Output
FAIL tidy: 24 rows (4 students x 3 subjects x 2 terms) FAIL tidy: columns roll_no, subject, term, marks FAIL tidy: R03 science t2 is 90 FAIL tidy: one row per student, subject and term FAIL report: 12 rows and the columns roll_no, subject, t1, t2, change FAIL report: R04 maths changed by +6 FAIL report: a flat table, t1 and t2 as plain columns 0 of 7 checks pass
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 check_marks.py
Work in a folder that holds just these three files and edit reshape_marks.py alone. Each run of
python check_marks.py reports every check, and the job is done at 7 of 7 checks pass. If you are stuck: melt
with id_vars="roll_no" comes first, then str.split("_", expand=True) on the old column names; the report is a
pivot with a two-column index followed by reset_index(), which leaves no MultiIndex behind.
Check yourself
5 questions about this lesson. Every answer and why it is right is on the page, behind “Show the answer”. Your score stays in this browser.
Interview questions
Fresher · What is the difference between pivot and pivot_table in pandas?
Both turn long data into a wide table: the values of one column become the new column headers, and the values of
another fill the cells. pivot only rearranges. Each combination of index and column values may appear once, or it
raises “Index contains duplicate entries, cannot reshape”, and it copies values of any type unchanged. pivot_table
aggregates: rows that land in the same cell are combined with aggfunc, which is the mean unless you choose another,
and it adds report options such as fill_value and margins. Use pivot when each cell should hold exactly one
value, because its error tells you when that assumption is wrong. Use pivot_table for summaries, and choose the
aggfunc on purpose: a mean can quietly blend a duplicate with the value it was meant to replace.
Key takeaways
- Tidy data has a column per variable, a row per observation and a table per kind of observation. Column headers that are values, such as months or subjects, are a common way for a table to break the rules.
melt(id_vars=…, var_name=…, value_name=…)turns wide into long; check that the long table has as many rows as the wide one had rows times melted columns.pivot(index=…, columns=…, values=…)turns long into wide. It sorts the new column labels, so put month and other ordered labels back in order, and usereset_index()for a flat table.- “Index contains duplicate entries” means two rows want the same cell. Find them with
duplicated(keep=False), count them withpivot_table(aggfunc="size"), and remove them by a stated rule before reaching for an average. - Keep data long for cleaning, analysis and charts; pivot to wide for reports, spreadsheets and model inputs.
- polars names the operations
unpivotandpivot, and DuckDB hasUNPIVOTandPIVOTstatements.
References
- Tidy Data (Journal of Statistical Software, volume 59, issue 10) (Journal of Statistical Software)
- Reshaping and pivot tables (pandas user guide) (pandas development team (NumFOCUS))
- pandas.melt (pandas API reference) (pandas development team (NumFOCUS))
- pandas.DataFrame.pivot (pandas API reference) (pandas development team (NumFOCUS))
- pandas.DataFrame.pivot_table (pandas API reference) (pandas development team (NumFOCUS))
- pandas.wide_to_long (pandas API reference) (pandas development team (NumFOCUS))
- Unpivots (Polars user guide) (Polars)
- Pivots (Polars user guide) (Polars)
- Upgrading to Polars 2 (deprecations and removals) (Polars)
- UNPIVOT statement (DuckDB documentation) (DuckDB Foundation)
- PIVOT statement (DuckDB documentation) (DuckDB Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress