pandas to polars and back: migration and interop

On a fifteen-line shop export, moving code from pandas to polars and back means translating idioms, converting frames, and proving that the result did not change. Most pandas operations have a direct polars method: a boolean mask becomes filter, assign becomes with_columns, groupby().agg becomes group_by().agg, transform becomes an expression with over(), merge becomes join and melt becomes unpivot. Code that relies on the index, such as set_index, loc or alignment by label, has no direct equivalent, because polars has no index: it becomes key columns, filters and joins. Frames cross through an Arrow interchange boundaryArrow interchange boundary: A transfer of typed column data between libraries using Arrow representations; values, nulls, types and key semantics still need a contract., with pl.from_pandas one way and to_pandas back: to_pandas needs the pyarrow package, and so does from_pandas for any column that is not a plain NumPy one, such as text. pl.sql runs SQL over polars frames, and Frame equivalenceframe equivalence: Agreement on the intended rows, values, column order, row order and required types, tested under an explicit comparison policy. is checked with assert_frame_equal under an explicit value, order and type policy.

Port choices expose index and missing-value rules

The examples use 15 order lines of a clothing shop in sales.csv, with the usual flaws of an export: city names in mixed case with stray spaces, a missing city, a missing status, a repeated order and missing discounts.

Most of a port from pandas to polars is mechanical: one line of pandas becomes one step of a polars chain, and the chain reads top to bottom. Three behaviours change results rather than syntax, and the right-hand column of the table names them. A missing value in a condition counts as “not true”, so filter drops the row where the original object-valued pandas examples keep it. Nullable pandas Boolean masks can drop unknown conditions too; state the dtype and policy. A missing group key forms a group of its own, where pandas leaves it out. And polars keeps no group or join order unless it is asked to, so a result needs an explicit sort before it is compared or published.

Follow the input and compare the result with the recorded example.

Conversion preserves values while types and index can change

import pandas as pd
import polars as pl

# A pandas frame with the types an analyst meets: a nullable integer, a float with NaN, text, a category, a timestamp.
pdf = pd.DataFrame({
    "qty": pd.array([2, None, 5], dtype="Int64"),
    "price": [899.0, float("nan"), 349.0],
    "city": ["Pune", None, "Thane"],
    "kind": pd.Categorical(["kurta", "saree", "kurta"]),
    "at": pd.to_datetime(["2026-04-02 10:00", "2026-04-05 11:30", None]),
}, index=pd.Index(["a", "b", "c"], name="row"))
print("pandas:", {c: str(t) for c, t in pdf.dtypes.items()})

# pandas to polars, through Arrow: NaN becomes null, and the index is dropped unless you ask for it.
p = pl.from_pandas(pdf)
print("polars:", p.schema)
print("with the index:", pl.from_pandas(pdf, include_index=True).columns)

# polars to pandas: an integer column with a null comes back as float64 unless Arrow-backed types are asked for.
print("back:  ", {c: str(t) for c, t in p.to_pandas().dtypes.items()})
print("arrow: ", {c: str(t) for c, t in p.to_pandas(use_pyarrow_extension_array=True).dtypes.items()})
print("qty after the round trip:", p.to_pandas()["qty"].tolist())

pandas: {'qty': 'Int64', 'price': 'float64', 'city': 'str', 'kind': 'category', 'at': 'datetime64[us]'}
polars: Schema({'qty': Int64, 'price': Float64, 'city': String, 'kind': Categorical, 'at': Datetime(time_unit='us', time_zone=None)})
with the index: ['row', 'qty', 'price', 'city', 'kind', 'at']
back:   {'qty': 'float64', 'price': 'float64', 'city': 'str', 'kind': 'category', 'at': 'datetime64[us]'}
arrow:  {'qty': 'int64[pyarrow]', 'price': 'double[pyarrow]', 'city': 'large_string[pyarrow]', 'kind': 'dictionary<values=large_string, indices=int64, ordered=0>[pyarrow]', 'at': 'timestamp[us][pyarrow]'}
qty after the round trip: [2.0, nan, 5.0]

pl.from_pandas turns NaN in a float column into null, keeps the nullable Int64, maps pandas’ str to String and a category to Categorical, and drops the index unless include_index=True keeps it as a column. The way back changes one type silently: an integer column with a null comes back as float64, with NaN in place of the null, because NumPy integers cannot be missing. to_pandas(use_pyarrow_extension_array=True) preserves Arrow-backed representations and nulls for these columns; inspect the resulting schema rather than assuming every dtype is identical. Convert once, at the edge of a pipeline, and say in the code which types you expect back.

Follow the input and compare the result with the recorded example.

SQL and expressions query the same in-scope frames

import polars as pl

sales = pl.read_csv("sales.csv").with_columns(
    pl.col("city").str.strip_chars().str.to_titlecase(),
    amount=pl.col("qty") * pl.col("unit_price"),
)

# pl.sql finds polars (and pandas) frames by their variable name; it returns a LazyFrame unless eager=True.
by_sql = pl.sql("""
    SELECT city, COUNT(*) AS orders, SUM(amount) AS revenue
    FROM sales
    WHERE status = 'Delivered' AND city IS NOT NULL
    GROUP BY city
    ORDER BY revenue DESC
""").collect()

# The same query as expressions.
by_expr = (
    sales.filter((pl.col("status") == "Delivered") & pl.col("city").is_not_null())
    .group_by("city")
    .agg(orders=pl.len(), revenue=pl.col("amount").sum())
    .sort("revenue", descending=True)
)
print(by_sql)
print("same values as the expressions:", by_sql.equals(by_expr.cast({"orders": pl.Int64})))

# SQLContext registers frames under names of your choice.
ctx = pl.SQLContext(orders=sales, eager=True)
print(ctx.tables(), ctx.execute("SELECT product, SUM(qty) AS units FROM orders GROUP BY product ORDER BY product").rows())

shape: (3, 3)
┌────────┬────────┬─────────┐
│ city   ┆ orders ┆ revenue │
│ ---    ┆ ---    ┆ ---     │
│ str    ┆ i64    ┆ i64     │
╞════════╪════════╪═════════╡
│ Mumbai ┆ 3      ┆ 8742    │
│ Pune   ┆ 3      ┆ 6796    │
│ Nagpur ┆ 4      ┆ 6444    │
└────────┴────────┴─────────┘
same values as the expressions: True
['orders'] [('Dupatta', 10), ('Kurta', 12), ('Saree', 5)]

pl.sql finds polars and pandas frames by their variable names and returns a LazyFrame unless eager=True asks for a DataFrame. SQLContext registers frames under names of your choice, which keeps larger programs readable. Both run through the same engine as expressions, so the SQL report and the expression report give the same values; only COUNT(*) comes back as Int64 where pl.len() gives UInt32, which the comparison casts away. SQL is often the shortest path when the code being ported was itself SQL, or when the people who maintain it think in SQL.

Follow the input and compare the result with the recorded example.

Frame equality detects the missing city revenue

import pandas as pd
import polars as pl
from polars.testing import assert_frame_equal

import report_pandas
import report_polars

from_pandas = pl.from_pandas(report_pandas.report("sales.csv"))
from_polars = report_polars.report("sales.csv")
print(from_pandas)
print(from_polars)
try:
    assert_frame_equal(from_pandas, from_polars, check_dtypes=False)
    print("the two reports agree")
except AssertionError as e:
    print("AssertionError:", str(e).splitlines()[0])

shape: (3, 3)
┌────────┬────────┬─────────┐
│ city   ┆ orders ┆ revenue │
│ ---    ┆ ---    ┆ ---     │
│ str    ┆ i64    ┆ i64     │
╞════════╪════════╪═════════╡
│ Mumbai ┆ 3      ┆ 8742    │
│ Nagpur ┆ 4      ┆ 6444    │
│ Pune   ┆ 3      ┆ 6796    │
└────────┴────────┴─────────┘
shape: (4, 3)
┌────────┬────────┬─────────┐
│ city   ┆ orders ┆ revenue │
│ ---    ┆ ---    ┆ ---     │
│ str    ┆ u32    ┆ i64     │
╞════════╪════════╪═════════╡
│ Mumbai ┆ 3      ┆ 8742    │
│ Nagpur ┆ 4      ┆ 6444    │
│ Pune   ┆ 3      ┆ 6796    │
│ null   ┆ 1      ┆ 1396    │
└────────┴────────┴─────────┘
AssertionError: DataFrames are different (height (row count) mismatch)

import pandas as pd


def report(path):
    df = pd.read_csv(path)
    df["city"] = df["city"].str.strip().str.title()
    df = df[df["status"] == "Delivered"]
    df = df.assign(amount=df["qty"] * df["unit_price"])
    return df.groupby("city", as_index=False).agg(orders=("order_id", "size"), revenue=("amount", "sum")).sort_values("city", ignore_index=True)


if __name__ == "__main__":
    print(report("sales.csv"))

import polars as pl


def report(path):
    return (
        pl.read_csv(path)
        .with_columns(pl.col("city").str.strip_chars().str.to_titlecase())
        .filter(pl.col("status") == "Delivered")
        .with_columns(amount=pl.col("qty") * pl.col("unit_price"))
        .group_by("city")
        .agg(orders=pl.len(), revenue=pl.col("amount").sum())
        .sort("city", nulls_last=True)
    )


if __name__ == "__main__":
    print(report("sales.csv"))

assert_frame_equal compares two polars frames column by column and stops at the first difference; check_dtypes=False lets integer widths differ, since pandas counts in int64 and polars in UInt32. The test fails, and it is right to: the polars report has a fourth row, the order with no city, which pandas’ group-by left out. Fill the city with “Unknown” in both versions, or pass dropna=False to pandas, and the test passes. Without the test, the two reports would have disagreed by 1,396 rupees in silence.

Which library a team should use depends on the work more than on speed. pandas has the widest ecosystem and suits small and medium tables that feed plotting, statistics or machine-learning libraries. polars suits long pipelines of expressions on large tables and lazy reads of big files. DuckDB suits work that is mostly SQL or mostly files on disk. The three exchange data through Arrow, so mixing them is normal.

Follow the input and compare the result with the recorded example.

A 12-step cleaning script ported and proved, with the traps reproduced Pro: opens with a Pro, Premium or Ultimate pass

What crosses the boundary between pandas and polars, and at what cost Pro: opens with a Pro, Premium or Ultimate pass

Where polars SQL differs, old code tested, and the choice of library Pro: opens with a Pro, Premium or Ultimate pass

Practise the stated transformations

Interview questions

Key takeaways

• Most pandas idioms map to one polars method; code built on the index becomes key columns, filters and joins.
• pl.from_pandas and to_pandas go through Arrow: NaN becomes null on the way in, and an integer column with a null becomes float64 on the way back unless Arrow types are kept.
• pl.sql and SQLContext run SQL over polars frames with the same engine as expressions.
• Prove every port with assert_frame_equal on real inputs, and sort before comparing.