Data types in pandas 3: str, nullable and Arrow

Every pandas column has one data type, its dtype, and the dtype decides three things: which values the column can hold, how a missing value is marked and how much memory the column takes. pandas 3 has three families of dtypes. The NumPy dtypes (int64, float64, bool, datetime64) are the defaults for numbers, flags and dates. Their missing marker is NaN, a floating-point value (NaT for dates), so an integer column with one gap becomes float64. The nullable dtypes (Int64, Float64, boolean, string) keep integers and flags intact and mark gaps with pd.NA. The Arrow-backed dtypes (int64[pyarrow], string[pyarrow] and others) store each column in the Apache Arrow format. Text also has a default of its own, str, new in pandas 3: a column of strings is no longer object. This lesson shows each family with its gaps and conversions, and measures what each one costs on a 100,000-row table.

What pandas 3 infers

When you build a Series or read a file without saying which types you want, pandas looks at the values and picks a dtype. Here are its choices for eight small columns:

import pandas as pd

samples = {
    "text": ["Asha", "Ravi", None],
    "whole numbers": [12, 7, 30],
    "numbers with a gap": [12, None, 30],
    "decimals": [12.5, 7.0, 30.25],
    "flags": [True, False, True],
    "flags with a gap": [True, None, False],
    "mixed values": ["A12", 7, 3.5],
    "dates as text": ["2026-01-05", "2026-02-10", None],
}
for label, values in samples.items():
    print(f"{label:20} {pd.Series(values).dtype}")

parsed = pd.to_datetime(pd.Series(samples["dates as text"]))
print(f"{'dates, parsed':20} {parsed.dtype}")

text                 str
whole numbers        int64
numbers with a gap   float64
decimals             float64
flags                bool
flags with a gap     object
mixed values         object
dates as text        str
dates, parsed        datetime64[us]

• Text becomes str. Before pandas 3.0 the same column was object, a column of arbitrary Python objects.
• Whole numbers become int64, but one gap turns them into float64: 12 and 30 are stored as 12.0 and 30.0, with NaN in between. That is the most common surprise in real data, where a single empty cell is enough.
• Flags become bool, but a flag column with a gap falls back to object, which holds True, None and False as separate Python objects and supports none of the fast boolean operations.
• Mixed values (text and numbers in one column) also become object, the one dtype that can hold anything.
• Dates written as text stay text. They become dates only when you parse them, and to_datetime then gives datetime64[us]: pandas 3 parses strings to microsecond resolution by default, where earlier versions always used nanoseconds.

str: the text dtype of pandas 3

The new default for text is a dedicated string dtype. It prints as str, and it behaves like the old object columns for everything that matters day to day, with two deliberate differences:

import numpy as np
import pandas as pd

names = pd.Series(["Asha", "Ravi", None])
print(names)
print(repr(names.dtype), "storage:", names.dtype.storage)
print(names.isna().tolist(), names.str.upper().tolist())

try:
    names[0] = 5  # a str column holds text or missing values only
except TypeError as error:
    print("TypeError:", error)

# astype(str) keeps a missing value missing instead of writing the text "nan"
prices = pd.Series([1.5, np.nan])
print(prices.astype(str).tolist(), prices.astype(str).dtype)

0    Asha
1    Ravi
2     NaN
dtype: str
<StringDtype(na_value=nan)> storage: pyarrow
[False, False, True] ['ASHA', 'RAVI', nan]
TypeError: Invalid value '5' for dtype 'str'. Value should be a string or missing value, got 'int' instead.
['1.5', nan] str

• It stores only strings or missing values, and a missing value is NaN, the same marker as in float64 columns. Putting a number into it raises TypeError, where an object column would have accepted anything. A wrong value now fails at the moment it is written, instead of surfacing much later as a mixed column.
• Its storage is pyarrow when the PyArrow package is installed, as here, and python (an array of Python string objects) when it is not. Both print the same values; they differ in memory and speed, which the last section measures.

The str accessor works as before (names.str.upper() keeps the gap a gap), and astype(str) now keeps a missing value missing. Earlier versions turned NaN into the three-letter text "nan", which then looked like a real value.

Finding text columns after the upgrade

Code written for pandas 2 often found text columns by checking for object. In pandas 3 that check quietly misses them:

import pandas as pd

df = pd.DataFrame({
    "name": [" Asha ", "Ravi"],
    "code": ["007", "019"],
    "qty": [1, 2],
    "note": ["gift", 3],  # a mixed column: text and a number
})
print(df.dtypes)

# Code written for pandas 2 often looked for text like this:
print("dtype == object:", [c for c in df.columns if df[c].dtype == object])

# In pandas 3 the text columns are str, so ask the question directly:
print("is_string_dtype:", [c for c in df.columns if pd.api.types.is_string_dtype(df[c])])
print("select_dtypes:", df.select_dtypes(include=["object", "string"]).columns.tolist())

name       str
code       str
qty      int64
note    object
dtype: object
dtype == object: ['note']
is_string_dtype: ['name', 'code']
select_dtypes: ['name', 'code', 'note']

dtype == object now finds only note, the one column that mixes text and a number, so a cleaning step built on it would skip every real text column without an error. pd.api.types.is_string_dtype asks the question you mean and works in pandas 2 and 3. select_dtypes(include=["object", "string"]) also works in both versions, and it includes mixed columns, which is useful when you are hunting for them.

Nullable dtypes: Int64, Float64 and boolean

The nullable dtypes are spelt with a capital letter (Int64, Float64) or in full (boolean, string). The number and flag types store their values in a NumPy array plus a separate mask that records which positions are missing, so a gap no longer forces a change of type:

import numpy as np
import pandas as pd

counts = pd.Series([12, None, 30])
print(counts.dtype, counts.tolist())  # the gap turned the integers into floats

kept = pd.Series([12, None, 30], dtype="Int64")
print(kept)
print(kept.sum(), kept.mean())

flags = pd.Series([True, None, False], dtype="boolean")
print(flags)

# In Float64, NaN and NA are the same missing value.
amounts = pd.Series([1.5, None, np.nan], dtype="Float64")
print(amounts.tolist(), amounts.isna().sum())

for target, values in [("int64", [1.0, np.nan]), ("Int64", [1.5, 2.0])]:
    try:
        pd.Series(values).astype(target)
    except (TypeError, ValueError) as error:
        print(f"astype({target!r}):", type(error).__name__, "-", error)

float64 [12.0, nan, 30.0]
0      12
1    <NA>
2      30
dtype: Int64
42 21.0
0     True
1     <NA>
2    False
dtype: boolean
[1.5, <NA>, <NA>] 2
astype('int64'): IntCastingNaNError - Cannot convert non-finite values (NA or inf) to integer.Replace or remove non-finite values or cast to an integer typethat supports these values (e.g. 'Int64')
astype('Int64'): TypeError - cannot safely cast non-equivalent float64 to int64

• Int64 keeps 12 and 30 as integers and shows the gap as <NA>. Sums and means skip it, as they skip NaN.
• boolean gives flags a real three-state type: True, False and <NA>.
• In Float64, NaN and pd.NA are the same missing value: pandas 3 converts a NaN you put into a nullable float column to <NA>, so both entries above count as missing.
• astype refuses conversions that would change values. A NaN cannot become a NumPy int64, and 1.5 cannot become an Int64 without rounding, so both raise instead of guessing. Round first (round()) when rounding is what you want.

The nullable text dtype is called string, in lower case. It is not the same as str: string marks gaps with pd.NA, str with NaN. You get string from convert_dtypes() or by asking for it.

NaN or pd.NA: how missing values behave

The two markers mean the same thing to a person, a value that is not there, but they follow different rules in comparisons and logic:

import numpy as np
import pandas as pd

print("NaN == NaN:", np.nan == np.nan)
print("NA == 1:", pd.NA == 1)
print("NA | True:", pd.NA | True, "  NA & False:", pd.NA & False, "  NA | False:", pd.NA | False)

try:
    if pd.NA:
        pass
except TypeError as error:
    print("if pd.NA:", error)

with_nan = pd.Series([2.0, np.nan, 5.0])
with_na = pd.Series([2, None, 5], dtype="Int64")
print(with_nan > 1)
print(with_na > 1)
print("sums:", with_nan.sum(), with_na.sum(), "  without skipping:", with_nan.sum(skipna=False), with_na.sum(skipna=False))

NaN == NaN: False
NA == 1: <NA>
NA | True: True   NA & False: False   NA | False: <NA>
if pd.NA: boolean value of NA is ambiguous
0     True
1    False
2     True
dtype: bool
0    True
1    <NA>
2    True
dtype: boolean
sums: 7.0 7   without skipping: nan <NA>

• NaN is a floating-point number that is not equal to anything, itself included, so ==, < and > with it are always False (and != is always True).
• pd.NA means “unknown”. A comparison with it is <NA> too, because the answer is unknown. Logic follows the same idea: NA | True is True whatever the unknown value is, NA & False is False, and NA | False stays unknown.
• Asking Python whether pd.NA is true, as an if does, raises TypeError instead of guessing.
• Both are skipped by sum and mean unless you pass skipna=False, and then each spreads into the result.

The difference matters most in filters. A comparison on a float64 column treats the gap as False, while on an Int64 column the gap stays <NA>, and a filter drops rows whose mask is <NA>:

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "as_float64": [2.0, np.nan, 5.0],
    "as_Int64": pd.array([2, None, 5], dtype="Int64"),
})
for column in df.columns:
    above = df[column] > 1
    print(f"{column:10} above 1: {df[above].index.tolist()}   not above 1: {df[~above].index.tolist()}")

as_float64 above 1: [0, 2]   not above 1: [1]
as_Int64   above 1: [0, 2]   not above 1: []

With float64, row 1 is “not above 1”, because the comparison said False and ~ turned it into True. With Int64, row 1 is in neither result, because ~ of an unknown is still unknown. If you split a table into two parts with a condition and its opposite, nullable columns can drop rows from both parts, so count the gaps first with isna().sum(), or fill them on purpose before you split.

Choosing types when you read: dtype, convert_dtypes and dtype_backend

The best moment to fix a type is when the data is read, before any information is lost. The same five-column CSV read four ways:

import io

import pandas as pd

CSV = """id,code,price,paid,city
1,007,10.5,yes,Pune
2,019,,no,Delhi
,120,7.25,yes,
"""

default = pd.read_csv(io.StringIO(CSV))
print(default)
print(default.dtypes, "\n")

told = pd.read_csv(io.StringIO(CSV), dtype={"id": "Int64", "code": "str"})
print(told[["id", "code"]], "\n")

nullable = pd.read_csv(io.StringIO(CSV)).convert_dtypes()
print(nullable.dtypes, "\n")

arrow = pd.read_csv(io.StringIO(CSV), dtype_backend="pyarrow")
print(arrow.dtypes)

    id  code  price paid   city
0  1.0     7  10.50  yes   Pune
1  2.0    19    NaN   no  Delhi
2  NaN   120   7.25  yes    NaN
id       float64
code       int64
price    float64
paid         str
city         str
dtype: object 

     id code
0     1  007
1     2  019
2  <NA>  120 

id         Int64
code       Int64
price    Float64
paid      string
city      string
dtype: object 

id        int64[pyarrow]
code      int64[pyarrow]
price    double[pyarrow]
paid     string[pyarrow]
city     string[pyarrow]
dtype: object

• The default reader made id a float64 because of its gap, and it read the product codes 007 and 019 as the numbers 7 and 19. The leading zeros are gone, and no later conversion can bring them back.
• dtype= tells the reader what you already know: id is a nullable integer and code is text. Both survive intact. Use this for every identifier, postcode, phone number and code column.
• convert_dtypes() picks the best nullable dtype for each column after reading: Int64, Float64 and, for text, string. It cannot restore the zeros, because they were never read.
• dtype_backend="pyarrow" makes every column Arrow-backed. It needs PyArrow installed, and some NumPy-based code expects NumPy dtypes, so try it on your own pipeline before you switch everything.

What each choice costs, measured

The dtype also decides memory. This example builds 100,000 seeded order rows (an ID with gaps, a city, a quantity, a price and a yes or no flag), reads them five ways and measures each frame with memory_usage(deep=True), which counts the text itself and not just the pointers to it:

import io

import numpy as np
import pandas as pd

# 100,000 seeded order rows: an id with 1 % gaps, a city, a quantity, a price and a yes/no flag.
rng = np.random.default_rng(7)
n = 100_000
ids = np.arange(1, n + 1).astype(str)
ids[rng.random(n) < 0.01] = ""
city = rng.choice(["Pune", "Delhi", "Chennai", "Kolkata", "Mumbai", "Jaipur"], n)
qty = rng.integers(1, 6, n).astype(str)
price = np.round(rng.uniform(50, 900, n), 2).astype(str)
paid = rng.choice(["yes", "no"], n, p=[0.8, 0.2])
lines = ["order_id,city,qty,price,paid"] + [",".join(row) for row in zip(ids, city, qty, price, paid)]
csv = "\n".join(lines) + "\n"

def read(**options):
    return pd.read_csv(io.StringIO(csv), **options)

as_objects = {"city": object, "paid": object}
frames = {
    "default (str on Arrow)": read(),
    "nullable (convert_dtypes)": read().convert_dtypes(),
    "Arrow (dtype_backend)": read(dtype_backend="pyarrow"),
    "text as Python objects": read().astype(as_objects),
    "city and paid as category": read().astype({"city": "category", "paid": "category"}),
}
print(f"{'how it was read':27} {'total bytes':>12} {'city bytes':>11}")
for name, df in frames.items():
    used = df.memory_usage(deep=True, index=False)
    print(f"{name:27} {used.sum():>12,} {used['city']:>11,}")

how it was read              total bytes  city bytes
default (str on Arrow)         4,864,018   1,383,798
nullable (convert_dtypes)      5,164,018   1,383,798
Arrow (dtype_backend)          4,076,518     983,798
text as Python objects        13,064,018   5,483,798
city and paid as category      2,600,106     100,084

• Stored as Python objects, the way text columns were stored before pandas 3, the table takes 13.06 MB, and the city column alone 5.48 MB, about 55 bytes for each short city name.
• The default str on Arrow storage brings city down to 1.38 MB and the table to 4.86 MB.
• dtype_backend="pyarrow" saves another 0.4 MB on city. Its Arrow strings use 4-byte offsets where str uses 8-byte ones, and the 4 bytes saved on each of 100,000 rows are exactly that difference.
• The nullable dtypes cost 0.3 MB more than the default, 1 byte per value for each of the three number columns’ masks.
• category stores each of the six city names once and a small code per row, and city shrinks to 0.10 MB. The categorical-data lesson covers when that is worth it.

Practise: fix the dtypes of an export

An order export has four columns: order_id, with one missing value; product_code, with leading zeros that matter; gift_wrap, written as Y, N or left empty; and quantity. Read it so that order_id stays an integer despite the gap, product_code keeps its zeros and gift_wrap becomes a nullable boolean with <NA> for the empty cell.

All three can be decided in the read_csv call: dtype= names the types, and true_values and false_values tell the reader which words mean True and False.

import io

import pandas as pd

EXPORT = """order_id,product_code,gift_wrap,quantity
1001,00420,Y,2
1002,07311,,1
,00420,N,3
1004,10005,Y,1
"""

orders = pd.read_csv(
    io.StringIO(EXPORT),
    dtype={"order_id": "Int64", "product_code": "str", "gift_wrap": "boolean"},
    true_values=["Y"],
    false_values=["N"],
)
print(orders)
print(orders.dtypes)

   order_id product_code  gift_wrap  quantity
0      1001        00420       True         2
1      1002        07311       <NA>         1
2      <NA>        00420      False         3
3      1004        10005       True         1
order_id          Int64
product_code        str
gift_wrap       boolean
quantity          int64
dtype: object

order_id is Int64 with <NA> for the missing ID, product_code keeps 00420, and gift_wrap is boolean, with <NA> where the export left the cell empty. quantity had no problems, so the default int64 stays.

Interview questions

Fresher · What is the difference between NaN and pd.NA in pandas?

Both mark a missing value, in different dtypes. NaN is a floating-point value from NumPy: it is the missing marker of float64 columns and of the default str text dtype, and it is not equal to anything, so comparing it with ==, < or > is False. Because it is a float, an integer column with a NaN becomes float64. pd.NA is pandas’ own marker for the nullable dtypes (Int64, Float64, boolean, string) and the Arrow-backed ones. It means “unknown”: comparisons with it return <NA>, logic follows three-valued rules (NA | True is True), and bool(pd.NA) raises TypeError. Integers keep their type. Both are skipped by sums by default. In pandas 3, NaN placed in a nullable float column is treated as NA.

Key takeaways

• pandas 3 infers str for text, int64 and float64 for numbers, bool for flags and object only for mixed values. One gap turns integers into float64 and flags into object.
• str holds only strings or NaN, raises TypeError for anything else, and is stored with PyArrow when it is installed. Find text columns with pd.api.types.is_string_dtype, not dtype == object.
• Nullable dtypes (Int64, Float64, boolean, string) keep their type and mark gaps as pd.NA, which means “unknown” in comparisons and logic.
• A filter drops rows whose mask is <NA>, so a condition and its opposite can both miss the gaps.
• Decide the types of IDs and codes when you read, with dtype=: lost leading zeros cannot be recovered later.
• Text stored as Python objects took 2.7 times the memory of the default str in the measured table; category was smaller still for repeated values.