Your country

Tools that support it use your country for local currency, number formats, units and paper size. Your choice is saved only in this browser.

Type a name or a two-letter code. Use the up and down arrow keys to move through the countries, Enter to choose one and Escape to close.

Data Analysis with Python (pandas, polars, DuckDB)  Module 7 – Reshaping: tidy data, melt, pivot and MultiIndex

Tidy data: long vs wide, melt and pivot

Reshape pandas tables between wide and long with melt and pivot, fix the "duplicate entries" error, and choose the layout each analysis needs.

  • Beginner
  • 30 minutes
  • Examples run with Python 3.14.8, pandas 3.0.6, numpy 2.5.3, pyarrow 25.0.1, polars 1.44.2 and duckdb 1.5.6
  • By MySmartCoPilot
Free: 30 minutes

What you will learn

  • Explain the three tidy-data rules and recognise the common messy layouts
  • Convert wide tables to long ones with melt and back with pivot
  • Handle the duplicate entries that make pivot fail
  • Split column names that hold two variables, such as maths_t1
  • Choose the long or the wide layout for analysis, charts and reports
What you’ll learn

What you will learn

  • Explain the three tidy-data rules and recognise the common messy layouts
  • Convert wide tables to long ones with melt and back with pivot
  • Handle the duplicate entries that make pivot fail
  • Split column names that hold two variables, such as maths_t1
  • Choose the long or the wide layout for analysis, charts and reports
On this page

Tidy data is a way of laying out a table so that every variable has its own column, every observation has its own row, and every kind of observation has its own table. A wide table breaks those rules when its column headers are values rather than names of variables: one column per month, per subject or per year. A long table holds the same numbers with one row per measurement and a column that says what was measured. In pandas, melt turns a wide table into a long one, and pivot turns a long table back into a wide one; pivot_table does the same job as pivot but combines rows that land in the same cell. Group-by, charts and joins work best on long data, while people reading a report usually want it wide, so analysts move between the two layouts all the time.

This lesson reshapes one small table of rain-gauge readings in both directions, with every run recorded. It melts a table with a column per month into one reading per row, answers three questions on the long table, pivots it back, reproduces the error pivot raises when two rows compete for one cell, and fixes that with a rule instead of an average. It then splits column names that hold two variables, shows the same reshapes in polars and DuckDB, and ends with an exercise on exam marks. You should already be able to load a CSV file into a DataFrame and run a groupby.

The three rules of tidy data

Hadley Wickham’s paper “Tidy Data” in the Journal of Statistical Software gave the layout its name and its three rules. Put in plain words:

  1. Each variable is a column. A column holds one kind of value: a gauge name, a year, a month or a reading in millimetres, never a mixture.
  2. Each observation is a row. One row holds everything recorded about one measurement: which gauge, which year, which month and how much rain.
  3. Each kind of observational unit is a table. Facts about gauges, such as their location and height, belong in a table of gauges; the readings belong in a table of readings. A join brings them together when a question needs both.

The paper also describes the five most common ways real tables break these rules, and each has a standard pandas remedy:

  • Column headers are values, not variable names, such as one column per month. melt fixes it, and it is the case this lesson starts with.
  • One column holds two variables, such as maths_t1 naming both a subject and a term. Melt first, then split the old names with str.split or str.extract, or use pd.wide_to_long.
  • Variables sit in both rows and columns, such as a sheet with one row per gauge and measure (minimum, maximum) and one column per day. Melt the day columns, then pivot the measure values into columns.
  • Several kinds of observational unit share one table, such as gauge locations repeated on every reading. Move the repeated facts into their own table with drop_duplicates and keep a key to join on.
  • One kind of unit is spread over several tables, such as one file per year. Stack them with pd.concat.

Whether a table is tidy depends on the question. For “how much rain fell each month?”, the month is a variable, and a table with a column per month is wide. For a report that compares the four months of one year side by side, the same wide table is exactly the right shape. The rules describe the layout that analysis code wants; reports and spreadsheets often want something else, and pivot produces it on demand.

melt: from wide to long

Here are the readings as they might arrive from a spreadsheet: three gauges, two years, and one column per month from June to September. The readings are invented for this lesson:

rain_wide.csv CSV · rain_wide.csv
gauge,year,JUN,JUL,AUG,SEP
Airport,2024,182.4,401.0,436.2,210.5
Airport,2025,140.9,455.3,298.0,260.1
Lakeside,2024,201.7,433.8,380.4,190.2
Lakeside,2025,165.0,470.2,322.6,240.8
Hill Fort,2024,240.3,520.6,410.9,230.0
Hill Fort,2025,198.5,560.1,350.7,275.4

melt keeps the columns you name in id_vars, which identify a reading, and turns every other column into rows. The old column headers go into a new column named by var_name, and the cells go into a column named by value_name:

One reading per row Python · melt_rain.py
import pandas as pd

# Three rain gauges, two years, one column per month (invented readings, in millimetres).
wide = pd.read_csv("rain_wide.csv")
print(wide)
print()

long = wide.melt(
    id_vars=["gauge", "year"],  # columns that say which reading this is: kept as they are
    var_name="month",  # the old column headers become the values of this column
    value_name="rain_mm",  # the old cells become the values of this column
)
print(f"wide: {wide.shape[0]} rows x {wide.shape[1]} columns")
print(f"long: {long.shape[0]} rows x {long.shape[1]} columns")
print(long.head(8))
print()
print(long.dtypes)

Output

       gauge  year    JUN    JUL    AUG    SEP
0    Airport  2024  182.4  401.0  436.2  210.5
1    Airport  2025  140.9  455.3  298.0  260.1
2   Lakeside  2024  201.7  433.8  380.4  190.2
3   Lakeside  2025  165.0  470.2  322.6  240.8
4  Hill Fort  2024  240.3  520.6  410.9  230.0
5  Hill Fort  2025  198.5  560.1  350.7  275.4

wide: 6 rows x 6 columns
long: 24 rows x 4 columns
       gauge  year month  rain_mm
0    Airport  2024   JUN    182.4
1    Airport  2025   JUN    140.9
2   Lakeside  2024   JUN    201.7
3   Lakeside  2025   JUN    165.0
4  Hill Fort  2024   JUN    240.3
5  Hill Fort  2025   JUN    198.5
6    Airport  2024   JUL    401.0
7    Airport  2025   JUL    455.3

gauge          str
year         int64
month          str
rain_mm    float64
dtype: object

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 melt_rain.py

The rain readings as a wide table with a column per month and as a long table with a row per reading; melt and pivot convert between them.Wide: a row per gauge and year6 rows × 6 columnsgauge · year · JUN · JUL · AUG · SEPAirport · 2024 · 182.4 · 401.0 · 436.2 · 210.5Long (tidy): a row per reading24 rows × 4 columnsgauge · year · month · rain_mmAirport · 2024 · JUN · 182.4Airport · 2024 · JUL · 401.0Airport · 2024 · AUG · 436.2Airport · 2024 · SEP · 210.5meltpivot

One table, two layouts. melt goes from wide to long, pivot goes back

Text description of the diagram

The upper box is the wide layout: one row per gauge and year, 6 rows and 6 columns. Its columns are gauge, year, JUN, JUL, AUG and SEP, so the first row reads Airport, 2024, 182.4, 401.0, 436.2 and 210.5.

The lower box is the long, tidy layout: one row per reading, 24 rows and 4 columns. Its columns are gauge, year, month and rain_mm, so the first row of the wide table becomes four rows: Airport, 2024, JUN, 182.4; Airport, 2024, JUL, 401.0; Airport, 2024, AUG, 436.2; and Airport, 2024, SEP, 210.5.

An arrow from wide to long is labelled melt, which turns the month headers into values of the month column. An arrow from long to wide is labelled pivot, which turns the month values back into column headers.

The arithmetic is worth checking every time you melt: each of the 6 rows had 4 month columns, so the long table has 6 × 4 = 24 rows, and the two identifier columns are repeated on each of them. A different count means a column you expected to keep was melted, or one you wanted to melt was named in id_vars. Three more details matter in practice:

  • Without value_vars, every column not in id_vars is melted. Name the value columns explicitly when the table also holds columns such as a comment or a total that should not become readings.
  • The rows come out column by column: all June readings first, then all July readings. Sort the long table if the order matters to you.
  • The result gets a fresh index from 0 to 23, because ignore_index=True is the default. Pass ignore_index=False to keep the original row labels.

Why the long layout pays off

Each of these questions is one group-by on the long table:

Three questions, one group-by each Python · long_questions.py
import pandas as pd

wide = pd.read_csv("rain_wide.csv")
long = wide.melt(id_vars=["gauge", "year"], var_name="month", value_name="rain_mm")

# 1. Season total for each gauge and year: one group-by on the long table.
totals = long.groupby(["gauge", "year"])["rain_mm"].sum()
print(totals)
print()

# 2. The wettest month of each gauge and year: the row with the largest reading per group.
wettest = long.loc[long.groupby(["gauge", "year"])["rain_mm"].idxmax()]
print(wettest.to_string(index=False))
print()

# 3. The average of each month over every gauge and year: group by the column melt created.
print(long.groupby("month", sort=False)["rain_mm"].mean().round(1))
print()

# The season total from the wide table works too, but every month column has to be named.
print(wide.set_index(["gauge", "year"])[["JUN", "JUL", "AUG", "SEP"]].sum(axis=1).round(1))

Output

gauge      year
Airport    2024    1230.1
           2025    1154.3
Hill Fort  2024    1401.8
           2025    1384.7
Lakeside   2024    1206.1
           2025    1198.6
Name: rain_mm, dtype: float64

    gauge  year month  rain_mm
  Airport  2024   AUG    436.2
  Airport  2025   JUL    455.3
Hill Fort  2024   JUL    520.6
Hill Fort  2025   JUL    560.1
 Lakeside  2024   JUL    433.8
 Lakeside  2025   JUL    470.2

month
JUN    188.1
JUL    473.5
AUG    366.5
SEP    234.5
Name: rain_mm, dtype: float64

gauge      year
Airport    2024    1230.1
           2025    1154.3
Lakeside   2024    1206.1
           2025    1198.6
Hill Fort  2024    1401.8
           2025    1384.7
dtype: float64

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 long_questions.py

The season total, the wettest month and the average of each month all come from the same two columns, month and rain_mm. The last result shows the wide alternative: it works, but the code has to name every month column. With a column per month of a 20-year record, that is 240 names, and every new month means changing the code, while the long table simply gains rows. groupby("month", sort=False) kept the months in the order they first appear in the data, which here is the calendar order; with the default sort=True they would come out alphabetically.

CSV Transpose & Unpivot Unpivot a wide CSV file into long rows, or pivot long rows back to a wide table, in the browser.

pivot: from long back to wide

pivot is the reverse of melt. index names the columns that identify a row of the result, columns names the column whose values become the new column headers, and values names the column that fills the cells:

Back to a month per column, and a season report Python · pivot_rain.py
import pandas as pd

long = pd.read_csv("rain_wide.csv").melt(id_vars=["gauge", "year"], var_name="month", value_name="rain_mm")

# Back to one row per gauge and year, one column per month.
wide = long.pivot(index=["gauge", "year"], columns="month", values="rain_mm")
print(wide)  # the new columns are sorted as text
print()

wide = wide[["JUN", "JUL", "AUG", "SEP"]]  # calendar order again
print(wide)
print()

# A plain table for a CSV file or a spreadsheet: the index becomes columns again.
flat = wide.reset_index()
flat.columns.name = None  # "month" was the name of the column labels
print(flat)
print()

# Another layout from the same long table: season totals, gauges down, years across.
report = long.pivot_table(index="gauge", columns="year", values="rain_mm", aggfunc="sum")
print(report)

Output

month             AUG    JUL    JUN    SEP
gauge     year                            
Airport   2024  436.2  401.0  182.4  210.5
          2025  298.0  455.3  140.9  260.1
Hill Fort 2024  410.9  520.6  240.3  230.0
          2025  350.7  560.1  198.5  275.4
Lakeside  2024  380.4  433.8  201.7  190.2
          2025  322.6  470.2  165.0  240.8

month             JUN    JUL    AUG    SEP
gauge     year                            
Airport   2024  182.4  401.0  436.2  210.5
          2025  140.9  455.3  298.0  260.1
Hill Fort 2024  240.3  520.6  410.9  230.0
          2025  198.5  560.1  350.7  275.4
Lakeside  2024  201.7  433.8  380.4  190.2
          2025  165.0  470.2  322.6  240.8

       gauge  year    JUN    JUL    AUG    SEP
0    Airport  2024  182.4  401.0  436.2  210.5
1    Airport  2025  140.9  455.3  298.0  260.1
2  Hill Fort  2024  240.3  520.6  410.9  230.0
3  Hill Fort  2025  198.5  560.1  350.7  275.4
4   Lakeside  2024  201.7  433.8  380.4  190.2
5   Lakeside  2025  165.0  470.2  322.6  240.8

year         2024    2025
gauge                    
Airport    1230.1  1154.3
Hill Fort  1401.8  1384.7
Lakeside   1206.1  1198.6

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pivot_rain.py

Four things in that output are easy to miss:

  • The months came back as AUG, JUL, JUN, SEP. pivot sorts the new column labels, and text sorts alphabetically. Select the columns in the order you want, as the second step does, or turn month into an ordered categorical before pivoting so that its sort order is the calendar order.
  • The result has the two identifier columns as a two-level row index, and the column labels carry the name month. reset_index() turns the index back into ordinary columns, and setting columns.name = None removes the leftover label, which gives a plain table to save as CSV.
  • The same long table produced a second layout without any further reshaping: pivot_table with aggfunc="sum" added up the four months of each gauge and year and placed the years across.
  • Because melt and pivot are exact inverses on data with one value per cell, the flat table equals the original wide file, apart from the order of the rows.
Pivot Table Generator Build sums, counts and averages by rows and columns from a CSV or Excel file without writing code.

When pivot fails: duplicate entries

pivot needs every combination of index and column values to appear at most once, because each combination becomes exactly one cell. Real data breaks that assumption more often than you would expect: a record sent twice, a correction appended instead of replacing the original, or a join earlier in the pipeline that matched one row to two. Here the Lakeside gauge’s July reading for 2025 arrives a second time, corrected from 470.2 to 407.2:

A corrected reading breaks pivot Python · pivot_duplicates.py
import pandas as pd

long = pd.read_csv("rain_wide.csv").melt(id_vars=["gauge", "year"], var_name="month", value_name="rain_mm")

# Lakeside's July 2025 reading arrives a second time, corrected from 470.2 to 407.2.
correction = pd.DataFrame({"gauge": ["Lakeside"], "year": [2025], "month": ["JUL"], "rain_mm": [407.2]})
readings = pd.concat([long, correction], ignore_index=True)

try:
    readings.pivot(index=["gauge", "year"], columns="month", values="rain_mm")
except ValueError as err:
    print(f"pivot failed: {err}")
print()

keys = ["gauge", "year", "month"]
print(readings[readings.duplicated(keys, keep=False)])  # every row whose key appears more than once
print()

# How many readings landed in each cell: anything above 1 is a duplicate.
print(readings.pivot_table(index=["gauge", "year"], columns="month", values="rain_mm", aggfunc="size"))
print()

# pivot_table with a mean "works", and blends the wrong reading with its correction.
mean = readings.pivot_table(index=["gauge", "year"], columns="month", values="rain_mm", aggfunc="mean")
print(f"mean of both readings: {mean.loc[('Lakeside', 2025), 'JUL']:.1f}")

# The rule for this data: a later reading replaces an earlier one.
latest = readings.drop_duplicates(keys, keep="last")
fixed = latest.pivot(index=["gauge", "year"], columns="month", values="rain_mm")
print(f"latest reading:        {fixed.loc[('Lakeside', 2025), 'JUL']:.1f}")
print(f"rows: {len(readings)} readings, {len(latest)} after dropping the replaced one")

Output

pivot failed: Index contains duplicate entries, cannot reshape

       gauge  year month  rain_mm
9   Lakeside  2025   JUL    470.2
24  Lakeside  2025   JUL    407.2

month           AUG  JUL  JUN  SEP
gauge     year                    
Airport   2024    1    1    1    1
          2025    1    1    1    1
Hill Fort 2024    1    1    1    1
          2025    1    1    1    1
Lakeside  2024    1    1    1    1
          2025    1    2    1    1

mean of both readings: 438.7
latest reading:        407.2
rows: 25 readings, 24 after dropping the replaced one

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pivot_duplicates.py

pivot refuses with ValueError: Index contains duplicate entries, cannot reshape, and that refusal is useful: it tells you the data does not mean what you assumed. Three steps resolve it:

  1. Find the duplicates. duplicated(keys, keep=False) marks every row whose key appears more than once, so you see both versions side by side, not only the second one.
  2. Count the rows per cell. pivot_table(..., aggfunc="size") shows how many rows would land in each cell. Every number above 1 is a cell that needs a decision.
  3. Apply the rule the data calls for. Here a later reading replaces an earlier one, so drop_duplicates(keys, keep="last") keeps the correction, and pivot works again.

The tempting fix is pivot_table, which never raises on duplicates because it aggregates them. With its default mean, the Lakeside cell became 438.7, a number that neither the gauge nor the correction ever reported. Aggregate only when several rows per cell are expected and the aggregate means something, such as a sum of daily readings into a monthly total; when duplicates are mistakes, remove them by a rule you can state.

pivot or pivot_table?
Criterion pivotpivot_table
Repeats Raises an errorCombined by aggfunc
Cells Values as they areMean, unless you set aggfunc
Types AnyWhat aggfunc accepts
Extras Nonefill_value, margins
When to choose One value per cellTotals, means, counts

Long or wide: choosing a layout

Neither layout is better in general; each suits different jobs. A practical rule is to keep data long while you clean, check and analyse it, and to pivot to wide at the end, for the table a person will read:

Long or wide?
Criterion LongWide
Group-by One columnA column per value
New data More rowsNew columns, new code
Charts What plots expectOne line per column
Reading Hard by eyeSide by side
Models Pivot firstA row per case
When to choose Cleaning, analysis, chartsReports, sheets, models

Column names that hold two variables

Exam sheets often have a column per subject and term, such as maths_t1 and science_t2. Each name holds two variables, and a tidy table needs a column for each. Melt the marks into rows, then split the old column names:

Subject and term out of the column names Python · two_variables.py
import pandas as pd

# One row per student; each column name holds two variables, the subject and the term.
marks = pd.DataFrame({
    "roll_no": ["R01", "R02", "R03"],
    "maths_t1": [78, 64, 91],
    "maths_t2": [82, 70, 88],
    "science_t1": [69, 75, 85],
    "science_t2": [74, 72, 90],
})

long = marks.melt(id_vars="roll_no", var_name="column", value_name="marks")
long[["subject", "term"]] = long["column"].str.split("_", expand=True)  # "maths_t1" -> "maths", "t1"
tidy = long.drop(columns="column")[["roll_no", "subject", "term", "marks"]]
print(tidy.sort_values(["roll_no", "subject", "term"]).to_string(index=False))
print()

# wide_to_long does both steps when the names follow a "stub_suffix" pattern.
both = pd.wide_to_long(marks, stubnames=["maths", "science"], i="roll_no", j="term", sep="_", suffix=r"t\d")
print(both)
print()

# A report layout: one row per student and subject, the terms side by side.
report = tidy.pivot(index=["roll_no", "subject"], columns="term", values="marks").reset_index()
report.columns.name = None
report["change"] = report["t2"] - report["t1"]
print(report.to_string(index=False))

Output

roll_no subject term  marks
    R01   maths   t1     78
    R01   maths   t2     82
    R01 science   t1     69
    R01 science   t2     74
    R02   maths   t1     64
    R02   maths   t2     70
    R02 science   t1     75
    R02 science   t2     72
    R03   maths   t1     91
    R03   maths   t2     88
    R03 science   t1     85
    R03 science   t2     90

              maths  science
roll_no term                
R01     t1       78       69
R02     t1       64       75
R03     t1       91       85
R01     t2       82       74
R02     t2       70       72
R03     t2       88       90

roll_no subject  t1  t2  change
    R01   maths  78  82       4
    R01 science  69  74       5
    R02   maths  64  70       6
    R02 science  75  72      -3
    R03   maths  91  88      -3
    R03 science  85  90       5

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 two_variables.py

str.split("_", expand=True) splits every name at the underscore and returns the parts as columns, which are assigned to subject and term in one step. When the names follow a fixed stub-and-suffix pattern, pd.wide_to_long does the melt and the split at once: it keeps one column per stub (here maths and science) and turns the suffix into a term level of the index, which is a halfway layout, still a column per subject. For names that do not split on a single character, such as MathsTerm1, use str.extract with a regular expression. The last step pivots the terms back into columns, because comparing a student’s two terms is easiest side by side, and computes the change.

The same reshapes in polars and DuckDB

polars and DuckDB both reshape with their own names for the same two operations:

unpivot and pivot in polars, UNPIVOT and PIVOT in DuckDB Python · other_engines.py
import duckdb
import polars as pl

# The same wide table, reshaped by polars and by DuckDB's SQL.
wide = pl.read_csv("rain_wide.csv")

long = wide.unpivot(index=["gauge", "year"], variable_name="month", value_name="rain_mm")
print(f"polars unpivot: {long.height} rows")
back = long.pivot(on="month", index=["gauge", "year"], values="rain_mm")
print(back)  # polars keeps the order in which the months first appear
print()

sql_long = duckdb.sql("UNPIVOT 'rain_wide.csv' ON JUN, JUL, AUG, SEP INTO NAME month VALUE rain_mm")
print(f"DuckDB UNPIVOT: {len(sql_long.fetchall())} rows")
totals = duckdb.sql("""
    PIVOT (UNPIVOT 'rain_wide.csv' ON JUN, JUL, AUG, SEP INTO NAME month VALUE rain_mm)
    ON year USING round(sum(rain_mm), 1) GROUP BY gauge ORDER BY gauge
""")
print(totals)

Output

polars unpivot: 24 rows
shape: (6, 6)
┌───────────┬──────┬───────┬───────┬───────┬───────┐
│ gauge     ┆ year ┆ JUN   ┆ JUL   ┆ AUG   ┆ SEP   │
│ ---       ┆ ---  ┆ ---   ┆ ---   ┆ ---   ┆ ---   │
│ str       ┆ i64  ┆ f64   ┆ f64   ┆ f64   ┆ f64   │
╞═══════════╪══════╪═══════╪═══════╪═══════╪═══════╡
│ Airport   ┆ 2024 ┆ 182.4 ┆ 401.0 ┆ 436.2 ┆ 210.5 │
│ Airport   ┆ 2025 ┆ 140.9 ┆ 455.3 ┆ 298.0 ┆ 260.1 │
│ Lakeside  ┆ 2024 ┆ 201.7 ┆ 433.8 ┆ 380.4 ┆ 190.2 │
│ Lakeside  ┆ 2025 ┆ 165.0 ┆ 470.2 ┆ 322.6 ┆ 240.8 │
│ Hill Fort ┆ 2024 ┆ 240.3 ┆ 520.6 ┆ 410.9 ┆ 230.0 │
│ Hill Fort ┆ 2025 ┆ 198.5 ┆ 560.1 ┆ 350.7 ┆ 275.4 │
└───────────┴──────┴───────┴───────┴───────┴───────┘

DuckDB UNPIVOT: 24 rows
┌───────────┬────────┬────────┐
│   gauge   │  2024  │  2025  │
│  varchar  │ double │ double │
├───────────┼────────┼────────┤
│ Airport   │ 1230.1 │ 1154.3 │
│ Hill Fort │ 1401.8 │ 1384.7 │
│ Lakeside  │ 1206.1 │ 1198.6 │
└───────────┴────────┴────────┘

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 other_engines.py

In polars, unpivot(index=…) is melt(id_vars=…) and pivot(on=…, index=…, values=…) is pivot with the header column named on. Unlike pandas, polars kept the months in the order they first appeared instead of sorting them. DuckDB’s UNPIVOT … ON … INTO NAME … VALUE … and PIVOT … ON … USING … GROUP BY … are SQL statements, here run directly on the CSV file. PIVOT always aggregates, like pivot_table: the USING clause names the aggregate, without one DuckDB counts the rows, and first() passes a single value through unchanged. polars raises an error on duplicates as pandas does, and its aggregate_function argument plays the role of aggfunc.

Version note

In polars, melt is deprecated in favour of unpivot, which takes index instead of id_vars and on instead of value_vars, and polars 2.0 removes melt altogether, according to its upgrade guide. unpivot behaves the same on polars 1.33.1, 1.44.2 and the release candidate 2.0.0rc2, so it is the name to use in new code.

Your turn: reshape an exam sheet

A school exports end-of-term marks with one row per student and a column per subject and term. Write two functions: to_tidy(wide), which returns one row per student, subject and term with the columns roll_no, subject, term and marks, and to_report(tidy), which returns one row per student and subject with the columns roll_no, subject, t1, t2 and change:

exercise/exam_wide.csv CSV · exercise/exam_wide.csv
roll_no,maths_t1,science_t1,english_t1,maths_t2,science_t2,english_t2
R01,78,69,81,82,74,85
R02,64,75,70,70,72,68
R03,91,85,77,88,90,80
R04,55,62,73,61,66,75
exercise/reshape_marks.py (your starting point) Python · exercise/reshape_marks.py
import pandas as pd


def to_tidy(wide):
    """One row per student, subject and term, with the columns roll_no, subject, term and marks."""
    return wide  # replace this line: melt, then split the old column names into subject and term


def to_report(tidy):
    """One row per student and subject, with the columns roll_no, subject, t1, t2 and change (t2 - t1)."""
    return tidy  # replace this line: pivot the terms into columns, then add the change

The checker calls both functions on that sheet and runs seven checks; on the untouched starting point, all seven fail:

exercise/check_marks.py, run on the starting point Python · exercise/check_marks.py
import pandas as pd

from reshape_marks import to_report, to_tidy

wide = pd.read_csv("exam_wide.csv")


def check(name, test):
    try:
        ok = bool(test())
    except Exception:  # a missing column, for example
        ok = False
    print("PASS" if ok else "FAIL", name)
    return ok


def tidy():
    return to_tidy(wide.copy())


def report():
    return to_report(tidy())


def mark(roll_no, subject, term):
    t = tidy()
    return t.loc[(t["roll_no"] == roll_no) & (t["subject"] == subject) & (t["term"] == term), "marks"].item()


results = [
    check("tidy: 24 rows (4 students x 3 subjects x 2 terms)", lambda: len(tidy()) == 24),
    check("tidy: columns roll_no, subject, term, marks", lambda: list(tidy().columns) == ["roll_no", "subject", "term", "marks"]),
    check("tidy: R03 science t2 is 90", lambda: mark("R03", "science", "t2") == 90),
    check("tidy: one row per student, subject and term", lambda: not tidy().duplicated(["roll_no", "subject", "term"]).any()),
    check("report: 12 rows and the columns roll_no, subject, t1, t2, change", lambda: report().shape == (12, 5) and list(report().columns) == ["roll_no", "subject", "t1", "t2", "change"]),
    check("report: R04 maths changed by +6", lambda: report().set_index(["roll_no", "subject"]).loc[("R04", "maths"), "change"] == 6),
    check("report: a flat table, t1 and t2 as plain columns", lambda: report().index.nlevels == 1 and report().columns.nlevels == 1 and "t1" in report().columns),
]
print(f"{sum(results)} of {len(results)} checks pass")

Output

FAIL tidy: 24 rows (4 students x 3 subjects x 2 terms)
FAIL tidy: columns roll_no, subject, term, marks
FAIL tidy: R03 science t2 is 90
FAIL tidy: one row per student, subject and term
FAIL report: 12 rows and the columns roll_no, subject, t1, t2, change
FAIL report: R04 maths changed by +6
FAIL report: a flat table, t1 and t2 as plain columns
0 of 7 checks pass

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 check_marks.py

Work in a folder that holds just these three files and edit reshape_marks.py alone. Each run of python check_marks.py reports every check, and the job is done at 7 of 7 checks pass. If you are stuck: melt with id_vars="roll_no" comes first, then str.split("_", expand=True) on the old column names; the report is a pivot with a two-column index followed by reset_index(), which leaves no MultiIndex behind.

Check yourself

5 questions about this lesson. Every answer and why it is right is on the page, behind “Show the answer”. Your score stays in this browser.

  1. Question 1 of 5 A sheet has one row per student and the columns roll_no, maths, science and english, each holding marks. Which tidy-data rule does it break?

    Choose one answer.

    Show the answer to question 1

    Answer: Each variable is a column: the subject is a variable, but its values are spread over the column headers

    Besides roll_no, the sheet records two variables, the subject and the mark. The subject has no column of its own: its values (maths, science, english) are the headers, so one row holds three observations. melt gives each observation its own row, with a subject column and a marks column.

  2. Question 2 of 5 A table has 50 rows and 14 columns: district, year and one column per month. How many rows does melt(id_vars=["district", "year"]) return?

    Type a number.

    Show the answer to question 2

    Answer: 600 rows

    Every one of the 12 month columns becomes a row for each of the 50 original rows: 50 × 12 = 600. The two id_vars are repeated on every one of them.

  3. Question 3 of 5 pivot(index="gauge", columns="month", values="rain_mm") raises "ValueError: Index contains duplicate entries, cannot reshape". What does it mean?

    Choose one answer.

    Show the answer to question 3

    Answer: At least two rows have the same gauge and the same month, so one cell would need two values

    A gauge appearing on many rows is normal: it has a reading per month. The error is about pairs: one gauge and one month must lead to one cell. Find the rows with duplicated(["gauge", "month"], keep=False), then decide which one is right; reset_index() changes nothing, because the duplicates are in the data.

  4. Question 4 of 5 After pivot, the month columns come out in the order AUG, JUL, JUN, SEP. Why?

    Choose one answer.

    Show the answer to question 4

    Answer: pivot sorts the new column labels, and text sorts alphabetically

    The long table listed June first, but pivot sorts the labels that become columns. Select the columns in calendar order afterwards, or make the month an ordered categorical before pivoting, so that sorting follows the calendar.

  5. Question 5 of 5 Which of these are easier with the long layout, one row per gauge, year and month?

    Choose every answer that is right.

    Show the answer to question 5

    Answer:

    • The average rainfall of each month over all gauges
    • Adding the readings of a new month without changing any code
    • Joining the readings to a table of monthly averages on the month

    The first three need one column to group, extend or join on, which the long layout has. A table for people to read is the wide layout, and pivot makes it from the long table at the end.

Interview questions

Fresher · What is the difference between pivot and pivot_table in pandas?

Both turn long data into a wide table: the values of one column become the new column headers, and the values of another fill the cells. pivot only rearranges. Each combination of index and column values may appear once, or it raises “Index contains duplicate entries, cannot reshape”, and it copies values of any type unchanged. pivot_table aggregates: rows that land in the same cell are combined with aggfunc, which is the mean unless you choose another, and it adds report options such as fill_value and margins. Use pivot when each cell should hold exactly one value, because its error tells you when that assumption is wrong. Use pivot_table for summaries, and choose the aggfunc on purpose: a mean can quietly blend a duplicate with the value it was meant to replace.

Key takeaways

  • Tidy data has a column per variable, a row per observation and a table per kind of observation. Column headers that are values, such as months or subjects, are a common way for a table to break the rules.
  • melt(id_vars=…, var_name=…, value_name=…) turns wide into long; check that the long table has as many rows as the wide one had rows times melted columns.
  • pivot(index=…, columns=…, values=…) turns long into wide. It sorts the new column labels, so put month and other ordered labels back in order, and use reset_index() for a flat table.
  • “Index contains duplicate entries” means two rows want the same cell. Find them with duplicated(keep=False), count them with pivot_table(aggfunc="size"), and remove them by a stated rule before reaching for an average.
  • Keep data long for cleaning, analysis and charts; pivot to wide for reports, spreadsheets and model inputs.
  • polars names the operations unpivot and pivot, and DuckDB has UNPIVOT and PIVOT statements.

References

Related tools

Report a problem with this lesson

Quick answers and tool search

Type to search tools or to get a quick answer, for example 18% of 2500. Use the up and down arrow keys to move through the results, Enter to choose, and Escape to close.