Outliers, impossible values and rule checks

An outlier is a value far from the rest of its column: a 4,000-unit order among orders of two or three, a delivery that took nine hours. An impossible value breaks a rule of the domain: marks of 105 out of 100, an age of −3, a delivery before its order. The two look alike in a chart and need opposite treatment. An impossible value is an error, to be corrected at the source or excluded by a stated rule. An outlier is a question: often it is real, and removing it removes part of what happened.

So the order matters. Check the rules of the domain first. Then flag unusual values with a statistical method, the IQR fences or robust z-scores, and review what the flags find. Decide to keep, correct, cap or exclude, for a stated reason, and record every exclusion with its count, so that anyone can see how the result was made and undo it.

Rules first

A rule says what values can exist: marks between 0 and 100, ages between 0 and 120, quantities above 0 except on returns, no dates in the future. Rules come from the domain, not from the data, and a statistical method cannot replace them, as the next example shows on twelve exam marks:

import pandas as pd

exam = pd.read_csv("marks.csv")
marks = exam["marks"]

# A domain rule: marks run from 0 to 100.
breaks_rule = ~marks.between(0, 100)

# A statistical flag: beyond the IQR fences, Q1 - 1.5 IQR and Q3 + 1.5 IQR.
q1, q3 = marks.quantile([0.25, 0.75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
beyond_fences = (marks < low) | (marks > high)

print(f"Q1 {q1}, Q3 {q3}, IQR {iqr}: fences {low} and {high}")
exam["breaks rule"] = breaks_rule
exam["beyond fences"] = beyond_fences
print(exam[breaks_rule | beyond_fences | (marks >= 95)].to_string(index=False))

Q1 66.0, Q3 82.75, IQR 16.75: fences 40.875 and 107.875
student  marks  breaks rule  beyond fences
    A05     99        False          False
    A06    105         True          False
    A08     -3         True           True

student,marks
A01,67
A02,72
A03,58
A04,81
A05,99
A06,105
A07,74
A08,-3
A09,63
A10,88
A11,70
A12,77

The rule between(0, 100) catches both 105 and −3. The IQR fences catch only −3: the upper fence is 107.875, so 105 looks ordinary to the statistics although no mark can be above 100. And the legitimate 99 is caught by neither, as it should be. A check written as a rule is also easier to explain than a threshold computed from the data.

between() is inclusive at both ends by default and returns False for a missing value, so ~marks.between(0, 100) counts a gap as a rule break. Decide which you mean: test marks.isna() separately, with the methods of the missing-data lesson, and keep missing values out of the violations.

Flagging unusual values: IQR fences and robust z-scores

Two methods flag values that are far from the rest and resist the extremes they look for.

• IQR fences flag values below Q1 − 1.5 × IQR or above Q3 + 1.5 × IQR, where Q1 and Q3 are the quartiles and IQR is their difference. The NIST/SEMATECH e-Handbook of Statistical Methods calls values beyond these inner fences mild outliers and values beyond 3 × IQR extreme outliers.
• Robust z-scores, also called modified z-scores, are 0.6745 × (x − median) / MAD, where MAD is the median of the absolute deviations from the median. The handbook reports Iglewicz and Hoaglin’s recommendation to label scores above 3.5 in absolute value as potential outliers.

Both rest on medians and quartiles, which a few extreme values barely move. That is the point: the classic z-score, built from the mean and the standard deviation, is pushed around by the very values it is meant to find.

import numpy as np
import pandas as pd

# Invented delivery times in minutes: ordinary days, six late deliveries, and four during a flood.
rng = np.random.default_rng(3)
ordinary = rng.normal(42, 6, 190).round()
late = [70, 74, 78, 82, 86, 90]
flood = [530, 545, 560, 575]
minutes = pd.Series(np.concatenate([ordinary, late, flood]))

# The classic z-score uses the mean and standard deviation, which the extreme values themselves inflate.
z = (minutes - minutes.mean()) / minutes.std()

# The robust (modified) z-score uses the median and the median absolute deviation, MAD.
median = minutes.median()
mad = (minutes - median).abs().median()
robust = 0.6745 * (minutes - median) / mad

print(f"mean {minutes.mean():.1f}, std {minutes.std():.1f} | median {median:.1f}, MAD {mad:.1f}")
print(f"std without the four flood deliveries: {minutes.iloc[:-4].std():.1f}")
print("flagged with |z| > 3:", int((z.abs() > 3).sum()), "| with |robust z| > 3.5:", int((robust.abs() > 3.5).sum()))
table = pd.DataFrame({"minutes": minutes, "z": z.round(2), "robust z": robust.round(1)})
print(table[robust.abs() > 3.5].to_string())

mean 53.7, std 72.0 | median 43.0, MAD 4.0
std without the four flood deliveries: 9.0
flagged with |z| > 3: 4 | with |robust z| > 3.5: 10
     minutes     z  robust z
190     70.0  0.23       4.6
191     74.0  0.28       5.2
192     78.0  0.34       5.9
193     82.0  0.39       6.6
194     86.0  0.45       7.3
195     90.0  0.50       7.9
196    530.0  6.61      82.1
197    545.0  6.82      84.6
198    560.0  7.03      87.2
199    575.0  7.24      89.7

Four deliveries during a flood, above 500 minutes, push the standard deviation from 9.0 without them to 72. With a cut-off of 3, the classic z-score flags only those four; the six late deliveries, 70 to 90 minutes against a median of 43, get scores of 0.23 to 0.50 and look normal. This is masking: large outliers hide smaller ones. The median (43) and the MAD (4) hardly notice the flood, so the robust z-score flags all ten. pandas has no built-in MAD, so compute it as (s - s.median()).abs().median().

Both methods break down on a column where most values are identical:

import pandas as pd

# Most shifts are exactly 8 hours, so more than half the deviations from the median are 0.
hours = pd.Series([8, 8, 8, 8, 8, 8, 9, 7.5, 12])
median = hours.median()
mad = (hours - median).abs().median()
print("median", median, "| MAD", mad)
print("robust z:", (0.6745 * (hours - median) / mad).tolist())

# The IQR rule breaks down the same way: Q1 = Q3, so every value that is not 8 is "beyond the fences".
q1, q3 = hours.quantile([0.25, 0.75])
print("Q1", q1, "| Q3", q3, "| beyond the fences:", hours[(hours < q1) | (hours > q3)].tolist())

# For a column like this, count the values and apply a rule from the domain instead.
print(hours.value_counts().sort_index().to_dict())
print("longer than a 10-hour shift:", hours[hours > 10].tolist())

median 8.0 | MAD 0.0
robust z: [nan, nan, nan, nan, nan, nan, inf, -inf, inf]
Q1 8.0 | Q3 8.0 | beyond the fences: [9.0, 7.5, 12.0]
{7.5: 1, 8.0: 6, 9.0: 1, 12.0: 1}
longer than a 10-hour shift: [12.0]

When more than half the shifts last exactly 8 hours, the MAD is 0 and every robust z-score is nan or infinite. Q1 and Q3 are both 8, so the fences flag every value that is not 8, including a normal 9-hour shift. For a column like this, look at the counts of distinct values and use a rule from the domain, here a shift longer than 10 hours, instead of a statistical flag.

On real data, a flag is not a verdict

The UCI Online Retail II data shows how far apart flags and errors can be. Its quantity column has Q1 = 1 and Q3 = 10, so the IQR fences are −12.5 and 23.5, and 116,489 of its 1,067,371 rows (10.9 %) fall outside them. Most are ordinary wholesale orders: the column is skewed, with many small orders and a long tail of large ones, and fences computed on the raw scale flag the whole tail.

The negative quantities tell a different story. Of 22,950 rows with a quantity below 0, 19,493 sit on invoices whose number starts with C, which marks a cancellation, and the other 3,457 all have a price of 0, which points to stock adjustments rather than sales. Neither group is an outlier to delete: each is a kind of row that a sales analysis excludes by rule, with the rule and the counts written down. (Measured with pandas 3.0.6 on the full dataset.)

Clip, winsorise, exclude or keep

Clipping replaces values beyond a limit with the limit, s.clip(lower, upper). Winsorising is clipping at percentiles, often the 1st and 99th. Both change the data, so use them only with a reason you can state, such as a model that is distorted by extreme values, and never for a total, because the extreme values really happened:

import numpy as np
import pandas as pd

# The deliveries of robust_z.py, with the flood marked.
rng = np.random.default_rng(3)
deliveries = pd.DataFrame({"minutes": np.concatenate([rng.normal(42, 6, 190).round(), [70, 74, 78, 82, 86, 90], [530, 545, 560, 575]])})
deliveries["flood"] = deliveries.index >= 196

# Winsorising: clip at the 1st and 99th percentiles. It changes the values themselves.
low, high = deliveries["minutes"].quantile([0.01, 0.99])
clipped = deliveries["minutes"].clip(low, high)
print(f"percentiles {low:.1f} and {high:.1f} | values changed: {int((clipped != deliveries['minutes']).sum())}")

summary = pd.DataFrame({
    "all, as recorded": deliveries["minutes"].agg(["mean", "median", "sum"]),
    "winsorised": clipped.agg(["mean", "median", "sum"]),
    "flood excluded": deliveries.loc[~deliveries["flood"], "minutes"].agg(["mean", "median", "sum"]),
})
print(summary.round(1).to_string())

percentiles 27.0 and 545.1 | values changed: 3
        all, as recorded  winsorised  flood excluded
mean                53.7        53.5            43.5
median              43.0        43.0            43.0
sum              10734.0     10691.3          8524.0

Winsorising at the 1st and 99th percentiles changes only 3 values, and the mean barely moves, from 53.7 to 53.5: the flood deliveries are 2 % of the rows, so the 99th percentile lies among them. Excluding the flood deliveries moves the mean to 43.5 and the total from 10,734 to 8,524 minutes. The median is 43 in all three columns, which is why a median, or a percentile such as the 90th, often describes typical performance better than a mean after any of these choices.

The deliveries during the flood are real, so the honest report keeps them and says what they did: “43.5 minutes on average excluding four deliveries during the flood (53.7 including them)”. Report with and without, and let the reader see the effect.

Keep an exclusions log

Every row you leave out should leave a trace: which rule removed it, with its key and value. The log turns a cleaning step into something a colleague can check, and its counts belong next to the results.

import pandas as pd

orders = pd.DataFrame({
    "invoice": ["1001", "1002", "C1003", "1004", "1005", "1006", "C1007", "1008"],
    "qty": [2, 1, -2, -5, 3, 4000, -1, 0],
    "price": [250.0, 99.0, 250.0, 0.0, 120.0, 1.5, 99.0, 75.0],
})

# Each rule names what it excludes and why. A row can break more than one rule.
rules = {
    "cancellation invoice (starts with C)": orders["invoice"].str.startswith("C"),
    "negative quantity on a normal invoice": (orders["qty"] < 0) & ~orders["invoice"].str.startswith("C"),
    "zero quantity": orders["qty"] == 0,
}
log = pd.concat(
    [orders.loc[hit, ["invoice", "qty"]].assign(rule=name) for name, hit in rules.items()],
    ignore_index=True,
)
print(log.to_string(index=False))

excluded = pd.concat(rules.values(), axis=1).any(axis=1)
kept = orders[~excluded]
print(log["rule"].value_counts().to_string())
print("rows:", len(orders), "| excluded:", int(excluded.sum()), "| kept:", len(kept))
assert len(kept) + int(excluded.sum()) == len(orders)

# 4,000 units at 1.5 is unusual, not impossible: it stays, flagged for review, not excluded.
print(kept.assign(review=kept["qty"] > 1000).to_string(index=False))

invoice  qty                                  rule
  C1003   -2  cancellation invoice (starts with C)
  C1007   -1  cancellation invoice (starts with C)
   1004   -5 negative quantity on a normal invoice
   1008    0                         zero quantity
rule
cancellation invoice (starts with C)     2
negative quantity on a normal invoice    1
zero quantity                            1
rows: 8 | excluded: 4 | kept: 4
invoice  qty  price  review
   1001    2  250.0   False
   1002    1   99.0   False
   1005    3  120.0   False
   1006 4000    1.5    True

Each rule is a named boolean mask, so the log lists every row a rule excludes, and a row that breaks two rules would appear twice in the log but once in the count of excluded rows. The assert checks that kept and excluded rows add up to the whole table. The 4,000-unit order at 1.5 per unit breaks no rule: it is unusual, so it stays, marked for review.

Exercise

import statistics


def iqr_fences(values, k=1.5):
    """Return (Q1 - k * IQR, Q3 + k * IQR) with pandas' default (inclusive) quartiles."""
    # Replace these lines with your code.
    q1, _, q3 = statistics.quantiles(values, n=4)
    return q1, q3


def robust_z(values):
    """Return 0.6745 * (x - median) / MAD for each value, rounded to 2 places; ValueError when MAD is 0."""
    # Replace these lines with your code.
    return [0.0 for _ in values]


def flag_outliers(values, method):
    """Return one boolean per value: beyond the IQR fences ("iqr") or |robust z| > 3.5 ("mad")."""
    # Replace these lines with your code.
    return [False for _ in values]

from outliers import flag_outliers, iqr_fences, robust_z

MARKS = [67, 72, 58, 81, 99, 105, 74, -3, 63, 88, 70, 77]


def raises(f, *args):
    try:
        f(*args)
    except ValueError:
        return True
    return False


def test_fences_like_pandas():
    """uses inclusive quartiles, as pandas does by default"""
    assert iqr_fences(MARKS) == (40.875, 107.875)
    assert iqr_fences([1, 2, 3, 4]) == (-0.5, 5.5)


def test_fences_k():
    """widens the fences with a larger k"""
    assert iqr_fences(MARKS, k=3) == (15.75, 133.0)


def test_iqr_flags():
    """flags only the values beyond the fences"""
    assert flag_outliers(MARKS, "iqr") == [v == -3 for v in MARKS]


def test_value_on_a_fence():
    """keeps a value that sits exactly on a fence"""
    values = [1, 2, 3, 4, 7]
    assert iqr_fences(values) == (-1.0, 7.0)
    assert flag_outliers(values, "iqr") == [False] * 5


def test_robust_z():
    """computes modified z-scores from the median and MAD"""
    assert robust_z([10, 11, 12, 11, 10, 50]) == [-0.67, 0.0, 0.67, 0.0, -0.67, 26.31]
    assert robust_z(MARKS)[7] == -5.7


def test_mad_flags():
    """flags values whose robust z-score is above 3.5 in absolute value"""
    assert flag_outliers([10, 11, 12, 11, 10, 50], "mad") == [False, False, False, False, False, True]
    assert flag_outliers(MARKS, "mad") == [v == -3 for v in MARKS]


def test_skewed_column():
    """flags the long right tail of a skewed column with both methods"""
    values = [1, 1, 2, 2, 2, 3, 3, 4, 5, 40]
    assert flag_outliers(values, "iqr") == [False] * 9 + [True]
    assert flag_outliers(values, "mad") == [False] * 9 + [True]


def test_mad_zero():
    """refuses to compute robust z-scores when MAD is 0"""
    assert raises(robust_z, [8, 8, 8, 8, 9, 12])
    assert raises(flag_outliers, [8, 8, 8, 8, 9, 12], "mad")


def test_unknown_method():
    """raises ValueError for a method it does not know"""
    assert raises(flag_outliers, MARKS, "zscore")

Interview questions

What is the difference between an outlier and an impossible value, and why does it matter?

An impossible value breaks a rule of the domain: marks above 100, a negative age, a delivery before its order. It is an error, so it is corrected at the source or excluded by that rule, with a count. An outlier is a value far from the others, flagged by a method such as the IQR fences or a robust z-score, and it is often real: a bulk order, a flood day. Treating the two alike causes both kinds of mistake. Deleting outliers removes real events and makes the data look calmer than it was, while relying only on statistics misses impossible values that happen to sit near the rest, such as 105 among marks in the 60s to 90s. So check rules first, flag second, review what is flagged, and report results with and without the outliers you keep.

Key takeaways

• Impossible values break a rule of the domain; outliers are far from the rest. Check rules first, flag second.
• between() is inclusive and returns False for a missing value: handle gaps separately from rule breaks.
• IQR fences (Q1 − 1.5 × IQR, Q3 + 1.5 × IQR) and robust z-scores (0.6745 × (x − median) / MAD, above 3.5) resist the values they look for; the classic z-score lets large outliers mask smaller ones.
• When MAD or IQR is 0, statistical flags break down: count distinct values and use a domain rule.
• On skewed columns, fences flag whole tails of real data. A flag starts a review; it is not a verdict.
• Clip or winsorise only with a stated reason, never for totals, and report results with and without the extremes.
• Keep an exclusions log of rule, key and value, and check that kept and excluded rows add up.