New columns, assign and Copy-on-Write

A new column in pandas is a single assignment: df["total"] = df["qty"] * df["price"] computes the whole column at once, with no loop. assign does the same inside a method chain and returns a new DataFrame, and the pd.col expressions of pandas 3 let it refer to columns of the frame being built. The bigger change in pandas 3 is Copy-on-Write: any DataFrame or Series that you get from another one, by indexing it or by calling a method on it, behaves as a copy of it. To change a DataFrame you change that DataFrame directly, in one step, for example with df.loc[mask, "col"] = value. The rule ends the old uncertainty about views and copies and removes the SettingWithCopyWarning, but it also means that chained assignment, df["col"][mask] = value, and df["col"].fillna(0, inplace=True) never change df; only a warning says so. This lesson creates columns, shows the rule at work, including the memory it shares, and rewrites the patterns that stopped working.

Creating columns

import numpy as np
import pandas as pd

orders = pd.DataFrame({
    "order": ["K1", "K2", "K3", "K4"],
    "tier": ["premium", "basic", "premium", "basic"],
    "qty": [2, 1, 5, 3],
    "price": [450.0, 1200.0, 99.0, 250.0],
})

orders["total"] = orders["qty"] * orders["price"]  # one expression for the whole column
orders["discount"] = np.where(orders["tier"] == "premium", 0.10, 0.0)  # if, row by row
orders["to_pay"] = (orders["total"] * (1 - orders["discount"])).round(2)
orders["bulk"] = orders["qty"] >= 3  # a comparison is already a column of True and False
orders["label"] = orders["order"] + " (" + orders["tier"] + ")"  # text joins element by element
print(orders)
print(orders.dtypes)

  order     tier  qty   price   total  discount  to_pay   bulk         label
0    K1  premium    2   450.0   900.0       0.1   810.0  False  K1 (premium)
1    K2    basic    1  1200.0  1200.0       0.0  1200.0  False    K2 (basic)
2    K3  premium    5    99.0   495.0       0.1   445.5   True  K3 (premium)
3    K4    basic    3   250.0   750.0       0.0   750.0   True    K4 (basic)
order           str
tier            str
qty           int64
price       float64
total       float64
discount    float64
to_pay      float64
bulk           bool
label           str
dtype: object

• orders["qty"] * orders["price"] multiplies the two columns row by row and returns a new Series, which becomes the total column. Operations on whole columns are vectorised: they run in compiled code, not in a Python loop.
• np.where(condition, if_true, if_false) is a row-by-row “if”. Here premium orders get a 10 % discount and the others none.
• Results chain like ordinary arithmetic: total * (1 - discount), rounded to two decimals.
• A comparison is already a column of True and False, so qty >= 3 makes the bulk flag in one step. There is no need to fill a column row by row.
• + joins text columns element by element; the result is str, like its inputs.

Assigning to a column name that does not exist creates it; assigning to one that exists replaces it.

The Copy-on-Write rule

The classic way to get this wrong is to give a value to a subset of rows in two indexing steps. Here is the same “premium customers get a 10 % discount” written three ways:

import numpy as np
import pandas as pd

customers = pd.DataFrame({
    "name": ["Asha", "Ravi", "Meena", "John"],
    "tier": ["premium", "basic", "premium", "basic"],
    "discount": [0.0, 0.0, 0.0, 0.0],
})
premium = customers["tier"] == "premium"

# 1. Chained assignment: two indexing steps, so the value lands in a temporary copy.
chained = customers.copy()
chained[premium]["discount"] = 0.1
print("chained:", chained["discount"].tolist())

# 2. One loc step on the frame itself: rows and column together.
with_loc = customers.copy()
with_loc.loc[premium, "discount"] = 0.1
print("loc:    ", with_loc["discount"].tolist())

# 3. assign returns a new frame and leaves the original as it was.
assigned = customers.assign(discount=np.where(premium, 0.1, 0.0))
print("assign: ", assigned["discount"].tolist(), " original:", customers["discount"].tolist())

chained: [0.0, 0.0, 0.0, 0.0]
loc:     [0.1, 0.0, 0.1, 0.0]
assign:  [0.1, 0.0, 0.1, 0.0]  original: [0.0, 0.0, 0.0, 0.0]

three_discounts.py:13: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  chained[premium]["discount"] = 0.1

• Chained assignment first selects the premium rows (chained[premium], a new object) and then sets ["discount"] on that object. Under Copy-on-Write the new object behaves as a copy, so the 0.1 went into a temporary copy that was thrown away. The frame is unchanged, and pandas prints a ChainedAssignmentError warning, shown above as standard error, that suggests loc.
• loc with rows and column together is one indexing step on the frame itself, so it changes the frame.
• assign never changes anything: it returns a new frame with the column replaced, and customers keeps its zeros.

Before pandas 3.0, whether a selection was a view of the original data or a copy depended on the exact operation, so the same chained assignment could work in one place and do nothing in another, and pandas tried to warn with SettingWithCopyWarning. pandas 3 made the behaviour consistent, with no way to switch it off: every derived object behaves as a copy, chained assignment never works, and that old warning no longer exists. The defensive .copy() calls that people added to silence it are no longer needed.

Lazy copies: how pandas avoids copying

“Behaves as a copy” does not mean “is copied”. pandas shares data between objects for as long as it can, and copies only at the moment one of them is written to. That is the “copy on write” in the name:

import numpy as np
import pandas as pd

df = pd.DataFrame({"amount": [1200.0, 800.0, 1500.0], "qty": [2, 1, 3]})

col = df["amount"]  # a lazy copy: no data is copied yet
print("shares memory:", np.shares_memory(col.to_numpy(), df["amount"].to_numpy()))

col.iloc[0] = 1.0  # the first write makes col copy its data
print("col:", col.tolist(), " df:", df["amount"].tolist())
print("shares memory:", np.shares_memory(col.to_numpy(), df["amount"].to_numpy()))

renumbered = df.reset_index(drop=True)  # also lazy: shares data until one side changes
print("reset_index shares memory:", np.shares_memory(renumbered["qty"].to_numpy(), df["qty"].to_numpy()))

values = df["amount"].to_numpy()  # read-only, so it cannot change df behind its back
try:
    values[0] = 5.0
except ValueError as error:
    print("ValueError:", error)

shares memory: True
col: [1.0, 800.0, 1500.0]  df: [1200.0, 800.0, 1500.0]
shares memory: False
reset_index shares memory: True
ValueError: assignment destination is read-only

• col = df["amount"] copies nothing: np.shares_memory confirms that the two objects use the same array.
• The first write, col.iloc[0] = 1.0, gives col its own copy of the data. df keeps 1200.0, and the two no longer share memory.
• Many methods return lazy copies too: reset_index(drop=True) shares the qty data with df until either side is changed.
• to_numpy() returns a read-only array when it shares data with the frame, so NumPy code cannot change the frame behind its back: writing to it raises “assignment destination is read-only”. Make a copy of the array if you need to change it.

One consequence is worth knowing for large frames. A lazy copy has to be copied for real if both objects are still alive when one of them is written to. df2 = df.reset_index(drop=True) followed by a write to df2 copies every column that is written to, because df still holds the originals. Reassigning to the same name, df = df.reset_index(drop=True), lets the old frame go, so the write needs no copy.

inplace and the copy keyword

Methods called on a column with inplace=True are chained assignment in disguise:

import numpy as np
import pandas as pd

df = pd.DataFrame({"city": ["Pune", "Delhi", "Kochi"], "amount": [1200.0, np.nan, 640.0]})

df["amount"].fillna(0, inplace=True)  # works on a temporary copy of the column
print("after inplace on df['amount']:", df["amount"].tolist())

df["amount"] = df["amount"].fillna(0)  # supported: assign the result back
print("after assigning back:        ", df["amount"].tolist())

other = pd.DataFrame({"amount": [np.nan, 5.0]})
other.fillna({"amount": 0}, inplace=True)  # supported: inplace on the frame itself
print("fillna({column: value}):      ", other["amount"].tolist())

after inplace on df['amount']: [1200.0, nan, 640.0]
after assigning back:         [1200.0, 0.0, 640.0]
fillna({column: value}):       [0.0, 5.0]

inplace_trap.py:6: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment using an inplace method.
Such inplace method never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' instead, to perform the operation inplace on the original object, or try to avoid an inplace operation using 'df[col] = df[col].method(value)'.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html
  df["amount"].fillna(0, inplace=True)  # works on a temporary copy of the column

df["amount"] is a new object, so filling it in place fills a temporary copy, and pandas warns again. Two supported forms do the job: assign the result back, df["amount"] = df["amount"].fillna(0), or call the method on the frame itself with a column-to-value dict, df.fillna({"amount": 0}, inplace=True). In pandas 3, fillna, replace, where, mask, clip, ffill, bfill and interpolate with inplace=True also return the object itself instead of None, so the old bug df = df.fillna(0, inplace=True), which used to set df to None, now does what it says.

The copy keyword of astype, rename, reindex, merge and several other methods has no effect any more and is deprecated:

import pandas as pd

df = pd.DataFrame({"qty": [2, 1, 3]})

small = df.astype({"qty": "int8"}, copy=False)  # the copy keyword no longer does anything
print(small["qty"].dtype)

snapshot = df.copy()  # an explicit, eager copy still exists when you want one
snapshot.loc[0, "qty"] = 99
print(df["qty"].tolist(), snapshot["qty"].tolist())

int8
[2, 1, 3] [99, 1, 3]

copy_keyword.py:5: Pandas4Warning: The copy keyword is deprecated and will be removed in a future version. Copy-on-Write is active in pandas since 3.0 which utilizes a lazy copy mechanism that defers copies until necessary. Use .copy() to make an eager copy if necessary.
  small = df.astype({"qty": "int8"}, copy=False)  # the copy keyword no longer does anything

The warning class is Pandas4Warning, which pandas uses for changes that will be enforced in pandas 4.0. Delete copy=False wherever you find it. When you really want an independent copy straight away, for example a snapshot to compare against later, .copy() still makes one.

Readable chains with assign, pipe and query

Because every step returns a new object and leaves the input alone, pandas code can be written as one chain of steps, read from top to bottom:

import pandas as pd

orders = pd.DataFrame({
    "order": ["K1", "K2", "K3", "K4", "K5", "K6"],
    "status": ["paid", "cancelled", "paid", "paid", "refunded", "paid"],
    "city": [" pune", "Delhi", "Kochi ", "pune", "Delhi", "Delhi"],
    "qty": [2, 1, 5, 3, 1, 4],
    "price": [450.0, 1200.0, 99.0, 250.0, 300.0, 150.0],
})


def log_rows(df, step):
    """A pipe step: print how many rows are left, then pass the frame on unchanged."""
    print(f"{step:<14}{len(df)} rows")
    return df


report = (
    orders.pipe(log_rows, "start")
    .query("status == 'paid'")
    .pipe(log_rows, "paid only")
    .assign(
        city=pd.col("city").str.strip().str.title(),
        total=pd.col("qty") * pd.col("price"),
    )
    .loc[lambda d: d["total"] >= 500]
    .pipe(log_rows, "total >= 500")
    .sort_values("total", ascending=False)
    .drop(columns=["status"])
)
print(report.to_string(index=False))
print("original untouched:", orders["city"].tolist()[:2])

start         6 rows
paid only     4 rows
total >= 500  3 rows
order  city  qty  price  total
   K1  Pune    2  450.0  900.0
   K4  Pune    3  250.0  750.0
   K6 Delhi    4  150.0  600.0
original untouched: [' pune', 'Delhi']

• query keeps the paid orders; loc with a function keeps totals of 500 or more.
• assign cleans the city names and adds the total, using pd.col because the chain has no variable name for the frame at that point. Before pandas 3 the same step needed lambda d: d["qty"] * d["price"].
• pipe(log_rows, "paid only") calls your own function with the frame and passes on whatever it returns. Here it prints the row count after each filter, which is the quickest way to find the step where rows go missing.
• The original orders frame is untouched at the end, so the chain can be rerun safely while you work on it.

A chain is easiest to read with one step per line. Once a chain grows past about ten steps, or a step needs a comment to explain it, split it into named functions and join them with pipe.

Practise: fix four Copy-on-Write bugs

This script cleans a small sales table. It runs without an error, but three of its changes do nothing and the fourth changes the wrong object. Find all four, then rewrite the script so it produces the right table with no warnings:

import pandas as pd

sales = pd.DataFrame({
    "region": ["north", "south", "north", "east", "south"],
    "units": [12, None, 7, 30, 4],
    "price": [100.0, 80.0, 120.0, 60.0, None],
})

sales["units"].fillna(0, inplace=True)  # 1. a missing unit count means none were sold
sales[sales["region"] == "east"]["price"] = 55.0  # 2. a new price for the east region
sales["price"][4] = 90.0  # 3. a price that was missing for one row

north = sales[sales["region"] == "north"]
north["units"] = north["units"] * 2  # 4. the north figures were reported in pairs

sales["revenue"] = sales["units"] * sales["price"]
print(sales)

  region  units  price  revenue
0  north   12.0  100.0   1200.0
1  south    NaN   80.0      NaN
2  north    7.0  120.0    840.0
3   east   30.0   60.0   1800.0
4  south    4.0    NaN      NaN

refactor_before.py:9: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment using an inplace method.
Such inplace method never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' instead, to perform the operation inplace on the original object, or try to avoid an inplace operation using 'df[col] = df[col].method(value)'.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html
  sales["units"].fillna(0, inplace=True)  # 1. a missing unit count means none were sold
refactor_before.py:10: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  sales[sales["region"] == "east"]["price"] = 55.0  # 2. a new price for the east region
refactor_before.py:11: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  sales["price"][4] = 90.0  # 3. a price that was missing for one row

The three warnings point at lines 9, 10 and 11. The fourth bug has no warning at all: north is a new object, so doubling its units is allowed, and nothing about sales changes. Under Copy-on-Write, changing a subset never changes the frame it came from.

import warnings

import pandas as pd

# Any chained assignment left in this script now stops it instead of only warning.
warnings.simplefilter("error", pd.errors.ChainedAssignmentError)

sales = pd.DataFrame({
    "region": ["north", "south", "north", "east", "south"],
    "units": [12, None, 7, 30, 4],
    "price": [100.0, 80.0, 120.0, 60.0, None],
})

sales["units"] = sales["units"].fillna(0)  # 1. assign the filled column back
sales.loc[sales["region"] == "east", "price"] = 55.0  # 2. rows and column in one loc step
sales.loc[4, "price"] = 90.0  # 3. one cell, by label
sales.loc[sales["region"] == "north", "units"] *= 2  # 4. change sales itself, not a subset of it

sales["revenue"] = sales["units"] * sales["price"]
print(sales)

  region  units  price  revenue
0  north   24.0  100.0   2400.0
1  south    0.0   80.0      0.0
2  north   14.0  120.0   1680.0
3   east   30.0   55.0   1650.0
4  south    4.0   90.0    360.0

Each change is now one step on sales itself: the filled column is assigned back, both price fixes use loc, and the north units are doubled with loc and *= on sales. The first line after the imports turns any remaining ChainedAssignmentError into an exception, so a chained assignment that slips back in would stop the script instead of only printing a warning. Revenue is now 0 for the south row with no units, and every row has a price.

Interview questions

Fresher · What does Copy-on-Write change in pandas 3?

With Copy-on-Write, any DataFrame or Series derived from another, by indexing (including selecting one column) or by a method, behaves as a copy. Changing it never changes the original, and the only way to change an object is to modify it directly, for example df.loc[mask, "col"] = value. So chained assignment such as df["col"][mask] = value, and df["col"].fillna(0, inplace=True), have no effect and raise a ChainedAssignmentError warning. SettingWithCopyWarning is gone, and defensive .copy() calls are unnecessary. Under the hood, pandas shares data between objects and copies it only when one of them is written to, which saves memory and time. Arrays from to_numpy() that share data are read-only, and the copy keyword of methods is deprecated. It is always on in pandas 3.

Key takeaways

• Create columns with whole-column expressions: arithmetic, comparisons, np.where and text joins, never a Python loop over rows.
• Copy-on-Write: every object derived from another behaves as a copy. Change a frame directly, in one step, with df.loc[rows, column] = value or df[column] = ....
• Chained assignment and inplace=True on a column never change the frame and only warn. Changing a subset, such as north = df[...], is allowed but never touches df, without any warning.
• Copies are lazy: data is shared until one side is written to, and to_numpy() may return a read-only array.
• copy=False is deprecated and does nothing; .copy() still makes an eager copy when you need one.
• assign, pd.col, query and pipe let you write analysis as one readable chain that leaves its input untouched.