MySmartCoPilot
Ask Go Pro Your pass Pass ends in 7 days

Your country

Tools that support it use your country for local currency, number formats, units and paper size. Your choice is saved only in this browser.

Data Analysis with Python (pandas, polars, DuckDB)  Module 9 – Text extraction and categorical data

Extracting fields from text with regular expressions

Extract PIN codes, phone numbers and bank codes from text with pandas str.extract, check each value against the rules and a directory, and count failures.

  • Intermediate
  • 30 minutes
  • Examples run with Python 3.14.8, pandas 3.0.6, numpy 2.5.3, pyarrow 25.0.1, polars 1.44.2 and duckdb 1.5.6
  • By MySmartCoPilot

What you will learn

  • Test and extract patterns with str.contains, str.extract and str.extractall
  • Write and validate patterns for Indian formats such as PIN codes, mobile numbers and IFSC codes
  • Explain which methods treat a pattern as a regular expression by default in pandas, polars and DuckDB
  • Count and inspect extraction failures, then check extracted values against a reference list
What you’ll learn

What you will learn

  • Test and extract patterns with str.contains, str.extract and str.extractall
  • Write and validate patterns for Indian formats such as PIN codes, mobile numbers and IFSC codes
  • Explain which methods treat a pattern as a regular expression by default in pandas, polars and DuckDB
  • Count and inspect extraction failures, then check extracted values against a reference list
On this page

People type values into text fields that were meant to hold more than one fact: a delivery address with the PIN code somewhere inside it, a contact note with a phone number and a bank code, a product name with a size. A regular expression (a regex) describes what such a value looks like, for example “six digits with no other digit beside them”, and pandas applies it to every row of a column. str.contains says whether each row has a match, str.extract copies the first match of each capture group into a column of its own, and str.extractall returns every match. Extraction is only half the work. Some rows will not match and some matches will be wrong, so you also normalise the text first, read and count the failures, check each value’s structure, and then check that the value exists in a reference list.

The examples use Indian formats: the six-digit PIN code, the ten-digit mobile number and the 11-character IFSC bank code. Each has a published definition, and the PIN codes have a public directory to check against. The same steps work for postal codes, phone numbers and identifiers anywhere.

Twelve delivery notes and one pattern

Customers of an online shop typed these delivery notes. Every note should contain a PIN code, and several of them hide a trap:

delivery_notes.csv CSV · delivery_notes.csv 
id,note
D01,"Flat 4B, Sai Residency, Kothrud, Pune 411038"
D02,"Near bus stand, Hubballi - 580020. Call before 6 pm"
D03,"Plot 12, Sector 21, Gurugram 122016 (alt PIN 122001)"
D04,"Old Market Road, Shillong 793 001"
D05,"C-14, Lajpat Nagar, New Delhi 110024"
D06,"Ward 6, Kozhikode 6730O4"
D07,"Banjara Hills, Hyderabad, near the clock tower"
D08,"House 12, Assi Ghat, Varanasi २२१००५"
D09,"Ph 9845012345, MG Road, Bengaluru 560001"
D10,"Anna Nagar, Chennai 600400"
D11,"Salt Lake Sector V, Kolkata, PIN 000000"
D12,"Main Road, Ranchi 834001-34"

str.extract takes a pattern with at least one capture group, a part in parentheses, and returns what the group matched in the first match of each row. The script tries two patterns:

The first match of each note, with and without word boundaries Python · extract_pins.py 
# The first match of a pattern, copied into a column of its own with str.extract.
import pandas as pd

notes = pd.read_csv("delivery_notes.csv", dtype="str")

# Six digits anywhere: also matches the first six digits of a phone number.
loose = notes["note"].str.extract(r"(\d{6})", expand=False)

# Six digits with no letter, digit or underscore touching them (\b is a word
# boundary). The named group (?P<pin>...) becomes the column name.
bounded = notes["note"].str.extract(r"(?P<pin>\b\d{6}\b)")

result = notes[["id"]].assign(loose=loose, pin=bounded["pin"])
print(result.to_string(index=False))
print()
print("rows:", len(notes))
print("with a PIN-like match:", notes["note"].str.contains(r"\b\d{6}\b").sum())
print("no match (NaN):", result["pin"].isna().sum())
print("dtype of the new column:", result["pin"].dtype)

Output

 id  loose    pin
D01 411038 411038
D02 580020 580020
D03 122016 122016
D04    NaN    NaN
D05 110024 110024
D06    NaN    NaN
D07    NaN    NaN
D08 २२१००५ २२१००५
D09 984501 560001
D10 600400 600400
D11 000000 000000
D12 834001 834001

rows: 12
with a PIN-like match: 8
no match (NaN): 3
dtype of the new column: str

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 extract_pins.py

  • The loose pattern (\d{6}) takes the first six digits it finds anywhere, so for D09 it returns 984501, the start of the phone number, instead of the PIN code 560001 that follows it. Nothing in the output warns you.
  • In the second pattern, \b is a word boundary: the six digits may not have a letter, a digit or an underscore directly before or after them. A phone number’s digits run on, so it no longer matches, and D09 gets its real PIN code.
  • (?P<pin>…) names the group, and the name becomes the column’s name. An unnamed group gives a column called 0.
  • A row without a match gets a missing value (NaN), never an error. That is why the count of missing values matters: three notes have no six-digit run at all.
  • The new column has pandas’ str dtype, so the codes stay text and keep any leading zeros.

The result for D08 deserves a second look. Its PIN code is written in Devanagari digits, २२१००५, and both patterns extracted it, because in Python’s re module \d means any decimal digit of any script. Yet str.contains with the same bounded pattern counted only 8 notes with a match, not 9: in pandas 3 with PyArrow installed it runs on another regex engine, in which \d means only 0 to 9, as the section on regex engines below shows. Later checks, other engines and most databases expect the ASCII digits 0 to 9, so from here on the patterns write [0-9] when they mean exactly those ten digits, and digits of other scripts are converted before matching.

Every match: str.extractall

D03 gives two PIN codes, 122016 and an alternative, 122001. str.extract returned the first one and said nothing about the second. str.extractall returns one row per match instead:

All matches, counted per note Python · all_matches.py 
# Every match, not only the first: str.extractall returns one row per match.
import pandas as pd

notes = pd.read_csv("delivery_notes.csv", dtype="str").set_index("id")

matches = notes["note"].str.extractall(r"(?P<pin>\b[0-9]{6}\b)")
print(matches.head(4))
print()
print("index levels:", matches.index.names)

# Candidates per note; notes without any are absent from extractall's result.
per_note = matches.groupby(level="id").size().reindex(notes.index, fill_value=0)
print("candidates per note:", per_note.value_counts().sort_index().to_dict())
print("notes with more than one:", per_note[per_note > 1].index.tolist())

# The first match of each note equals what str.extract returns.
first = matches.xs(0, level="match")["pin"]
extracted = notes["note"].str.extract(r"(?P<pin>\b[0-9]{6}\b)")["pin"].dropna()
print("first matches equal str.extract:", first.equals(extracted))

Output

              pin
id  match        
D01 0      411038
D02 0      580020
D03 0      122016
    1      122001

index levels: ['id', 'match']
candidates per note: {0: 4, 1: 7, 2: 1}
notes with more than one: ['D03']
first matches equal str.extract: True

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 all_matches.py

The result has a two-level index: the note’s own index, id, and match, which numbers the matches of each note from 0. A note without any match does not appear at all, which is why the script reindexes the counts against all twelve notes before counting them. Seven notes hold exactly one candidate, one holds two, and four hold none: D08 is among them now that the pattern accepts only 0 to 9. The first match of each note equals what str.extract returned. A note with two candidates is a question for a person, or for a rule you can write down, such as “take the first PIN code unless the note says alt”; it is not something to settle silently.

Read every row that failed

A pattern that finds a value in most rows looks finished, and that is exactly when people stop. The rows it missed show where its assumptions break, so list them before deciding anything:

Four misses, two safe repairs Python · failures.py 
# Every row the pattern missed, read one by one, and the two repairs that are safe.
import unicodedata

import pandas as pd

notes = pd.read_csv("delivery_notes.csv", dtype="str")
PIN = r"\b([0-9]{6})\b"

first_try = notes["note"].str.extract(PIN, expand=False)
print("missed by the pattern:")
print(notes.loc[first_try.isna(), ["id", "note"]].to_string(index=False))


def ascii_digits(text: str) -> str:
    """Write digits of any script (Devanagari २, Arabic-Indic ٢ ...) as 0-9."""
    return "".join(
        str(unicodedata.decimal(ch)) if ch.isdecimal() and not ch.isascii() else ch
        for ch in text
    )


repaired_text = (
    notes["note"]
    .map(ascii_digits)  # repair 1: other scripts' digits
    .str.replace(r"\b([0-9]{3}) ([0-9]{3})\b", r"\1\2", regex=True)  # repair 2: "793 001"
)
second_try = repaired_text.str.extract(PIN, expand=False)

changed = first_try.isna() & second_try.notna()
found = notes.loc[changed, "id"].tolist()
print()
print("found after the repairs:", found, second_try[changed].tolist())
print("still missing:", notes.loc[second_try.isna(), "id"].tolist())
print(f"rows: {len(notes)}, found first time: {first_try.notna().sum()}, "
      f"after repairs: {second_try.notna().sum()}, missing: {second_try.isna().sum()}")

Output

missed by the pattern:
 id                                           note
D04              Old Market Road, Shillong 793 001
D06                       Ward 6, Kozhikode 6730O4
D07 Banjara Hills, Hyderabad, near the clock tower
D08           House 12, Assi Ghat, Varanasi २२१००५

found after the repairs: ['D04', 'D08'] ['793001', '221005']
still missing: ['D06', 'D07']
rows: 12, found first time: 8, after repairs: 10, missing: 2

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 failures.py

Each miss has its own cause, and each cause gets its own decision:

  • D04, 793 001. The code is written as three digits, a space and three more, a common way to write PIN codes. Joining two groups of three digits is safe here because the joined value is still checked later, so this repair is automated.
  • D08, २२१००५. The digits are right; only the script differs. Python’s unicodedata.decimal gives the value of any decimal digit, so the repair writes them as 221005.
  • D06, 6730O4. A capital letter O stands where a zero belongs. The fix looks obvious, but turning letters into digits is a guess, and a guessed PIN code sends a parcel to the wrong post office. This note goes back to the customer.
  • D07. There is no PIN code at all, and no pattern can recover one.

After the two repairs, ten of the twelve notes have a candidate. Report all four numbers the script prints, so that anyone reading the result knows how much of it came from the text as typed and how much from a repair.

Structure first, then existence

Six digits are not yet a PIN code. A value can have the right shape and still be impossible, such as the placeholder 000000, or be possible and still not exist, such as a mistyped 600400. So two checks follow the extraction:

Extracting a PIN code in four steps: normalise the text, extract the match, check its structure, check it exists.Typed text"... Shillong 793 001"1. Normalisedigits of any script to 0-9,"793 001" to "793001"2. Extractstr.extract(r"\b([0-9]{6})\b")3. Check the structurestr.fullmatch(r"[1-9][0-9]{5}")4. Check that it existsisin(directory["pin"])A status for every rowok · no PIN found · not a valid PIN· not in the directory

From typed text to a checked PIN code

Text description of the diagram

A vertical flow from a typed delivery note to a status for every row, in four steps.

  1. Normalise the text: digits written in other scripts become the digits 0 to 9, and a PIN code typed with a space after its third digit, such as 793 001, is joined into 793001.
  2. Extract the first match of the pattern \b([0-9]{6})\b with str.extract: six digits with no digit, letter or underscore next to them.
  3. Check the structure with str.fullmatch and the pattern [1-9][0-9]{5}: six digits whose first digit is not 0.
  4. Check that the code exists with isin against the directory's list of PIN codes.

Every row ends with one status: ok, no PIN found, not a valid PIN, or not in the directory. The rows that are not ok are counted and listed, never dropped.

What makes a PIN code

India’s postal code, the Postal Index Number (PIN), has six digits and is written after the name of the town. Read from the left, the digits narrow the address down, from a region to a sorting and forwarding centre and then to the delivery post office, as the Universal Postal Union’s addressing guide for India shows. The Department of Posts publishes every PIN code with its post offices in the All India Pincode Directory on the Open Government Data Platform India, under the Government Open Data License – India. The file pin_directory.parquet holds one row per PIN code of the copy that MySmartCoPilot’s PIN Code Finder uses, with its districts and states.

Extract, check the structure, check the directory Python · check_pins.py 
# Extract, then check the structure, then check that the PIN code exists.
# pin_directory.parquet holds one row per PIN code of the Department of Posts'
# All India Pincode Directory (Open Government Data Platform India, under the
# Government Open Data License - India), with its districts and states.
import unicodedata

import numpy as np
import pandas as pd

notes = pd.read_csv("delivery_notes.csv", dtype="str")
directory = pd.read_parquet("pin_directory.parquet")
first_digits = "".join(sorted(directory["pin"].str[0].unique()))
print(f"directory: {len(directory):,} PIN codes, first digits {first_digits}")


def ascii_digits(text: str) -> str:
    return "".join(
        str(unicodedata.decimal(ch)) if ch.isdecimal() and not ch.isascii() else ch
        for ch in text
    )


text = notes["note"].map(ascii_digits)
text = text.str.replace(r"\b([0-9]{3}) ([0-9]{3})\b", r"\1\2", regex=True)
pin = text.str.extract(r"\b([0-9]{6})\b", expand=False)

well_formed = pin.str.fullmatch(r"[1-9][0-9]{5}")  # missing PINs give False
known = pin.isin(directory["pin"])

checked = notes[["id"]].assign(
    pin=pin,
    status=np.select(
        [pin.isna(), ~well_formed, ~known],
        ["no PIN found", "not a valid PIN", "not in the directory"],
        default="ok",
    ),
).merge(directory[["pin", "district"]], on="pin", how="left")

print(checked.to_string(index=False, na_rep="-"))
print()
print(checked["status"].value_counts().to_string())

Output

directory: 19,586 PIN codes, first digits 123456789
 id    pin               status           district
D01 411038                   ok               Pune
D02 580020                   ok            Dharwad
D03 122016                   ok           Gurugram
D04 793001                   ok   East Khasi Hills
D05 110024                   ok South / South East
D06      -         no PIN found                  -
D07      -         no PIN found                  -
D08 221005                   ok           Varanasi
D09 560001                   ok    Bengaluru Urban
D10 600400 not in the directory                  -
D11 000000      not a valid PIN                  -
D12 834001                   ok             Ranchi

status
ok                      8
no PIN found            2
not in the directory    1
not a valid PIN         1

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 check_pins.py

  • The structure check uses str.fullmatch, which must match the whole value rather than a part of it. [1-9][0-9]{5} rejects a leading zero, and the directory agrees: every one of its 19,586 PIN codes starts with a digit from 1 to 9. D11’s placeholder 000000 fails here.
  • The existence check is a plain isin against the directory. D10’s 600400 has the right shape but is not a PIN code. The customer probably meant 600040, the code of Anna Nagar in Chennai, but only the customer can confirm it.
  • np.select turns the checks into one readable status per row, and the merge adds the district, so a person can compare it with the town in the note. D02 says Hubballi, which lies in Dharwad district.
  • The final count, 8 of 12 notes ok, is the honest headline, and every other row carries a reason someone can act on.
PIN Code Finder (India Post) Look up a PIN code's post offices, district and state, or find the PIN code of an area.

Pattern or text? Each method’s default

A regex treats some characters as instructions: . matches any character, + repeats, ( opens a group. That makes it easy to search for a pattern when you meant a piece of text. Product codes with a dot in them show the trap, in all three engines this track uses:

Searching four product codes for the text 2.5 Python · literal_or_regex.py 
# Searching product codes for the text "2.5": which functions read the dot as
# "any character"?
import duckdb
import pandas as pd
import polars as pl

codes = ["OIL-2.5L", "OIL-235L", "OIL-2X5L", "GHEE-2-5KG"]
s = pd.Series(codes)
lf = pl.DataFrame({"code": codes})
code = pl.col("code")
con = duckdb.connect()
duck = con.sql(
    "SELECT contains(code, '2.5') AS c, regexp_matches(code, '2.5') AS r FROM lf"
).pl()

rows = {
    "pandas contains": s.str.contains("2.5"),
    "pandas regex=False": s.str.contains("2.5", regex=False),
    "polars contains": lf.select(code.str.contains("2.5")).to_series(),
    "polars literal=True": lf.select(code.str.contains("2.5", literal=True)).to_series(),
    "duckdb contains": duck["c"],
    "duckdb regexp_matches": duck["r"],
}
print(" " * 22 + "".join(f"{c:>11}" for c in codes))
for name, hits in rows.items():
    print(f"{name:22}" + "".join(f"{'match' if h else '-':>11}" for h in hits))


def one(sql):
    return con.sql(sql).fetchone()[0]


replaced = {
    "pandas str.replace": s.str.replace(".", "_")[0],
    "polars str.replace": lf.select(code.str.replace(".", "_")).item(0, 0),
    "duckdb replace": one("SELECT replace('OIL-2.5L', '.', '_')"),
    "duckdb regexp_replace": one("SELECT regexp_replace('OIL-2.5L', '.', '_')"),
}
print()
print("Replacing '.' with '_' in OIL-2.5L:")
for name, value in replaced.items():
    print(f"{name:22}{value}")

Output

                         OIL-2.5L   OIL-235L   OIL-2X5L GHEE-2-5KG
pandas contains             match      match      match      match
pandas regex=False          match          -          -          -
polars contains             match      match      match      match
polars literal=True         match          -          -          -
duckdb contains             match          -          -          -
duckdb regexp_matches       match      match      match      match

Replacing '.' with '_' in OIL-2.5L:
pandas str.replace    OIL-2_5L
polars str.replace    _IL-2.5L
duckdb replace        OIL-2_5L
duckdb regexp_replace _IL-2.5L

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 literal_or_regex.py

pandas’ str.contains reads "2.5" as “a 2, any one character, a 5”, so it also matches 235, 2X5 and 2-5; regex=False searches for the text itself. polars’ str.contains behaves the same way, and literal=True switches the pattern off. DuckDB went the other way: its contains() searches for text, and regexp_matches() is the pattern version. The replace functions disagree as well. pandas’ str.replace has read its pattern as plain text by default since pandas 2.0, while polars’ str.replace reads a pattern and replaces only the first match, which here is the first character, so OIL-2.5L became _IL-2.5L.

Pattern or text by default (pandas 3.0.6, polars 1.44.2, DuckDB 1.5.6)
Criterion pandaspolarsDuckDB
Does it contain …? str.contains: a pattern (regex=False for text)str.contains: a pattern (literal=True for text)contains(): text; regexp_matches(): a pattern
Replace str.replace: text (regex=True for a pattern), every matchstr.replace: a pattern, first match only; str.replace_all: every matchreplace(): text, every match; regexp_replace(): a pattern, first match unless the g option
Extract str.extract and str.extractall: always a patternstr.extract and str.extract_all: a patternregexp_extract(): a pattern, an empty string when nothing matches
Count matches str.count: a patternstr.count_matches: a pattern (literal=True for text)len(regexp_extract_all()): a pattern
Starts with … str.startswith: textstr.starts_with: textstarts_with(): text

When the text you search for comes from a user or a file, pass the text switch, or escape the string with re.escape so that every special character stands for itself.

One pattern, four regex engines

pandas, polars and DuckDB do not share a regex engine, and the differences reach patterns that look harmless. pandas runs str.extract, str.extractall and str.findall with Python’s re. With PyArrow installed, as in this track’s setup, its str dtype passes str.contains, str.match, str.fullmatch, str.count and regex str.replace to PyArrow’s compute functions, which are built on the RE2 library. It falls back to re for calls that PyArrow cannot handle, for example when the call passes flags or the pattern holds a lookaround or a backreference, as pandas’ own source code shows. polars uses Rust’s regex crate, and DuckDB uses RE2.

Unicode digits, a lookbehind and Python's ASCII flag, in four engines Python · engines.py 
# One pattern, four regex engines: Python's re, RE2 inside PyArrow and DuckDB,
# and Rust's regex crate in polars.
import duckdb
import pandas as pd
import polars as pl

notes = ["Pune 411038", "Varanasi २२१००५", "Hyderabad"]
s = pd.Series(notes)  # pandas 3 "str" dtype, stored by PyArrow
df = pl.DataFrame({"note": notes})
note = pl.col("note")
con = duckdb.connect()


def show(label, values):
    print(f"{label:28}{[None if pd.isna(v) else v for v in values]}")


def duck(pattern):
    sql = f"SELECT regexp_extract(note, '{pattern}', 1) FROM df"
    return con.sql(sql).pl().to_series()


print(r"Does \d match the Devanagari digits २२१००५?")
show("pandas str.contains", s.str.contains(r"\d{6}"))
show("pandas str.extract", s.str.extract(r"(\d{6})", expand=False))
show("polars str.extract", df.select(note.str.extract(r"(\d{6})", 1)).to_series())
show("duckdb regexp_extract", duck(r"(\d{6})"))
show("[0-9]{6}, pandas extract", s.str.extract(r"([0-9]{6})", expand=False))

behind = r"((?<=Pune )[0-9]{6})"
print()
print("A lookbehind, (?<=Pune )[0-9]{6}:")
show("pandas str.extract", s.str.extract(behind, expand=False))
for name, run in [
    ("polars str.extract", lambda: df.select(note.str.extract(behind, 1))),
    ("duckdb regexp_extract", lambda: duck(behind)),
]:
    try:
        run()
    except Exception as error:
        print(f"{name:28}{type(error).__name__}")

print()
print("Python's ASCII flag (?a) in pandas str.contains:")
try:
    s.str.contains(r"(?a)\d{6}")
except Exception as error:
    print(f"{'':28}{type(error).__name__}: {str(error).split(': ', 1)[1]}")

Output

Does \d match the Devanagari digits २२१००५?
pandas str.contains         [True, False, False]
pandas str.extract          ['411038', '२२१००५', None]
polars str.extract          ['411038', '२२१००५', None]
duckdb regexp_extract       ['411038', '', '']
[0-9]{6}, pandas extract    ['411038', None, None]

A lookbehind, (?<=Pune )[0-9]{6}:
pandas str.extract          ['411038', None, None]
polars str.extract          ComputeError
duckdb regexp_extract       InvalidInputException

Python's ASCII flag (?a) in pandas str.contains:
                            ArrowInvalid: invalid perl operator: (?a

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 engines.py

  • \d means any decimal digit in Python’s re and in Rust’s regex crate, but only 0 to 9 in RE2. So on the same pandas column, str.contains(r"\d{6}") said no to Varanasi’s Devanagari PIN code, while str.extract with the same pattern returned it. [0-9] means the same ten digits in every engine.
  • RE2 and Rust’s regex crate leave out lookarounds and backreferences, features for which no fast implementation is known, as the regex crate’s documentation explains. polars raised ComputeError and DuckDB InvalidInputException for the lookbehind that pandas accepted. Put a group around the part you want instead: Pune ([0-9]{6}).
  • DuckDB’s regexp_extract returns an empty string where pandas and polars return a missing value, so a count of missing values in DuckDB stays at zero unless you write NULLIF(regexp_extract(...), '').
  • Python’s inline flag (?a), which makes \d mean 0 to 9 in re, is not RE2 syntax. pandas passed the pattern to PyArrow, which rejected it with ArrowInvalid.

Test a pattern on examples before you trust it

A pattern is a small program, and it deserves tests. Write down the values it must accept and the values it must reject, near misses above all, and run the pattern over them before it meets a million rows. The script below tests a pattern for each of three formats, plus a naive first attempt at the mobile number:

The mobile number and IFSC rules used here

Within India a mobile number is dialled as ten digits; from a fixed line a 0 goes in front, and callers from abroad use India’s country code, +91, as TRAI’s consultation paper on the National Numbering Plan and the ITU’s list of country codes set out. Mobile numbers in use start with 6, 7, 8 or 9. The same TRAI paper describes a plan, accepted by the Department of Telecommunications, to give fixed-line area codes that start with 6, 3, 4 and 2 to mobile numbers once the current ranges run out, so a check on the first digit tests today’s numbering, not a permanent rule. An IFSC (Indian Financial System Code) identifies a bank branch for NEFT transfers: 11 characters, of which the first four letters stand for the bank, the fifth is always 0 and the last six stand for the branch, according to the Reserve Bank of India’s FAQ on NEFT. That gives the pattern [A-Z]{4}0[A-Z0-9]{6}.

Six examples per pattern, near misses included Python · pattern_tests.py 
# A pattern is tested on values it must accept and values it must reject
# before it meets real data.
import re

PIN = re.compile(r"[1-9][0-9]{5}")
MOBILE_NAIVE = re.compile(r"[0-9]{10}")
MOBILE = re.compile(r"(?:\+91|91|0)?([6-9][0-9]{9})")
IFSC = re.compile(r"[A-Z]{4}0[A-Z0-9]{6}")


def tidy_phone(text: str) -> str:
    return re.sub(r"[\s-]", "", text)  # spaces and hyphens are only for reading


PIN_CASES = {"411038": True, "900099": True, "011038": False, "41103": False,
             "4110380": False, "४११०३८": False}
MOBILE_CASES = {"9845012345": True, "+91 98450 12345": True, "098450-12345": True,
                "5123456789": False, "984501234": False, "044 2626 1234": False}
IFSC_CASES = {"SBIN0000691": True, "hdfc0001234": True, "ICIC0ABC123": True,
              "HDFCO001234": False, "SBIN000691": False, "SBIN1000691": False}
CASES = {
    "PIN code": (PIN.fullmatch, str, PIN_CASES),
    "mobile, naive": (MOBILE_NAIVE.fullmatch, tidy_phone, MOBILE_CASES),
    "mobile": (MOBILE.fullmatch, tidy_phone, MOBILE_CASES),
    "IFSC": (IFSC.fullmatch, str.upper, IFSC_CASES),
}

for name, (test, prepare, cases) in CASES.items():
    wrong = [text for text, expected in cases.items()
             if bool(test(prepare(text))) != expected]
    verdict = "all agree" if not wrong else f"wrong on {', '.join(wrong)}"
    print(f"{name:14} {len(cases)} examples: {verdict}")

print()
print("normalised mobile:", MOBILE.fullmatch(tidy_phone("+91 98450-12345")).group(1))

Output

PIN code       6 examples: all agree
mobile, naive  6 examples: wrong on +91 98450 12345, 098450-12345, 5123456789
mobile         6 examples: all agree
IFSC           6 examples: all agree

normalised mobile: 9845012345

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 pattern_tests.py

The naive mobile pattern, ten digits and nothing else, got half of its examples wrong: it rejected two real numbers written with +91 or a leading 0, and it accepted 5123456789. The final pattern removes spaces and hyphens first, allows the prefixes and captures the ten-digit number, which is the form worth storing. Keep each test table next to its pattern, so that the next change to the pattern meets the same near misses.

Regex Tester Try a pattern on your own examples with live highlighting. It runs JavaScript's regex engine, so check \d and lookarounds in your target engine as well. IFSC Code Finder & Validator Check whether an IFSC with the right shape belongs to a real bank branch.

Your turn: three fields from a contact column

Customers who paid on delivery typed their contact details for a refund into one text column. Write extract_contacts(df) so that it returns a PIN code, a mobile number and an IFSC for each of the ten contacts, with a missing value wherever a field is absent or not valid. The docstring of the starting point states the rules:

exercise/contacts.csv CSV · exercise/contacts.csv 
id,contact
C01,"Asha Rao, 98450 12345, Indiranagar, Bengaluru 560038, refund to HDFC0001234"
C02,"Imran S. | +91-91234-56789 | Andheri East, Mumbai 400053 | IFSC icic0006543"
C03,"R. Iyer, landline 044 2626 1234, Anna Nagar, Chennai 600040, SBIN0001234"
C04,"Neha, 9876543210, Gomti Nagar, Lucknow 226010, IFSC HDFCO002345"
C05,"Karan Mehta 0 98765 43210, Sector 62, Noida 201309, UTIB0000123"
C06,"Fatima, 5123456789, Salt Lake, Kolkata, PIN 000000, PUNB0123400"
C07,"Joseph, +91 99887 76655, Ernakulam 682 011, KKBK0000958"
C08,"Priya S, 98451-23456, Jayanagar, Bengaluru 560011, ifsc canr0001234"
C09,"Sunil, 99887 76650 or 99887 76651, Jaipur 302017, no IFSC yet"
C10,"Meera, +91 87654 32109, Coimbatore 641018, HDFC0001234 (old: HDFC0000999)"
exercise/extract_contacts.py (your starting point) Python · exercise/extract_contacts.py 
# Your starting point: fill in pin, mobile and ifsc for every contact.
import pandas as pd


def extract_contacts(df: pd.DataFrame) -> pd.DataFrame:
    """Return the columns id, pin, mobile and ifsc, one row per contact.

    A field that is absent or not valid is a missing value.
    pin     six digits with no digit next to them, the first 1-9;
            "682 011" counts as 682011
    mobile  ten digits starting with 6, 7, 8 or 9, with or without +91, 91
            or 0 in front, spaces and hyphens allowed; keep only the ten
            digits, from the first valid number of a contact
    ifsc    four letters, the digit 0, then six letters or digits, in any
            case; return it in capitals
    """
    out = df[["id"]].copy()
    # Write your code here.
    return out

The sample checks compare your three columns with the expected values, row by row. With the starting point, one of the six checks passes:

exercise/check_contacts.py, run on the starting point Python · exercise/check_contacts.py 
# Sample checks for extract_contacts.py. Run `python check_contacts.py` in the
# folder that holds all three files.
import pandas as pd

from extract_contacts import extract_contacts

contacts = pd.read_csv("contacts.csv", dtype="str")
before = contacts.copy()
EXPECTED = {
    "pin": ["560038", "400053", "600040", "226010", "201309",
            None, "682011", "560011", "302017", "641018"],
    "mobile": ["9845012345", "9123456789", None, "9876543210", "9876543210",
               None, "9988776655", "9845123456", "9988776650", "8765432109"],
    "ifsc": ["HDFC0001234", "ICIC0006543", "SBIN0001234", None, "UTIB0000123",
             "PUNB0123400", "KKBK0000958", "CANR0001234", None, "HDFC0001234"],
}

result = extract_contacts(contacts)
checks = []


def check(name, test):
    try:
        problem = test()
    except Exception as error:  # a missing column or a wrong type fails the check
        problem = f"{type(error).__name__}: {error}"
    checks.append(problem is None)
    label = "ok  " if problem is None else "FAIL"
    print(label, name + ("" if problem is None else f": {problem}"))


def columns():
    want = ["id", "pin", "mobile", "ifsc"]
    if list(result.columns) != want or len(result) != len(contacts):
        return (f"expected the columns {want} and {len(contacts)} rows, "
                f"got {list(result.columns)} and {len(result)} rows")
    return None


def values(column):
    def test():
        got = [None if pd.isna(v) else v for v in result[column]]
        pairs = zip(contacts["id"], EXPECTED[column], got)
        wrong = [f"{i} expected {e}, got {g}" for i, e, g in pairs if e != g]
        if not wrong:
            return None
        more = f" (and {len(wrong) - 2} more)" if len(wrong) > 2 else ""
        return "; ".join(wrong[:2]) + more
    return test


def real_missing():
    empty = [c for c in ["pin", "mobile", "ifsc"] if (result[c] == "").any()]
    return None if not empty else f"empty strings in {empty}: use a missing value instead"


def unchanged():
    if contacts.equals(before):
        return None
    return "extract_contacts changed the DataFrame it was given"


check("columns and rows", columns)
for column in ["pin", "mobile", "ifsc"]:
    check(f"{column} values", values(column))
check("missing values are missing", real_missing)
check("input left unchanged", unchanged)
print(f"{sum(checks)} of {len(checks)} checks pass")

Output

FAIL columns and rows: expected the columns ['id', 'pin', 'mobile', 'ifsc'] and 10 rows, got ['id'] and 10 rows
FAIL pin values: KeyError: 'pin'
FAIL mobile values: KeyError: 'mobile'
FAIL ifsc values: KeyError: 'ifsc'
FAIL missing values are missing: KeyError: 'pin'
ok   input left unchanged
1 of 6 checks pass

Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 check_contacts.py

Put the three files in a folder of their own and edit only extract_contacts.py. You are done when python check_contacts.py reports 6 of 6 checks pass. First hint: a mobile number may contain spaces and hyphens, so find the candidates with a pattern such as \+?[0-9][0-9 -]{8,}[0-9] and str.extractall, tidy each candidate, and keep the first one that passes fullmatch. Upper-case the text before you look for the IFSC.

Check yourself

6 questions about this lesson. Every answer and why it is right is on the page, behind “Show the answer”. Your score stays in this browser.

  1. Question 1 of 6 A Series holds the codes OIL-2.5L, OIL-235L, OIL-2X5L and GHEE-2-5KG. With pandas 3 defaults, which of them does s.str.contains("2.5") report as a match?

    Choose one answer.

    Show the answer

    Answer: All four

    str.contains reads its argument as a regular expression by default, and a dot matches any one character, so 235, 2X5 and 2-5 all match. regex=False, or the escaped pattern r"2\.5", matches only OIL-2.5L.

  2. Question 2 of 6 Which pattern makes str.extract return 560001, not 984501, for the note "Ph 9845012345, MG Road, Bengaluru 560001"?

    Choose one answer.

    Show the answer

    Answer: (\b[0-9]{6}\b)

    Without word boundaries the first six digits of the phone number match first. \b requires that no letter, digit or underscore touches the six digits, so the ten-digit phone number no longer matches and the PIN code does.

  3. Question 3 of 6 A note holds two PIN codes. What does str.extractall return for it?

    Choose one answer.

    Show the answer

    Answer: Two rows under the note's index, with the match level numbering them 0 and 1

    extractall returns one row per match, indexed by the original index plus a level called match. Notes without any match are left out of its result, which is why the lesson reindexes the counts against every note.

  4. Question 4 of 6 In pandas 3 with PyArrow installed, a column holds a PIN code written in Devanagari digits. Which calls with the pattern \d{6} find no match in it?

    Choose every answer that is right.

    Show the answer

    Answer:

    • str.count
    • str.contains

    With the PyArrow-backed str dtype, str.contains, str.match, str.fullmatch, str.count and regex str.replace run on RE2, where \d means only 0 to 9. str.extract, str.extractall and str.findall use Python's re, where \d matches the digits of every script. Writing [0-9] gives the same answer everywhere.

  5. Question 5 of 6 DuckDB's regexp_extract finds no match in a row. What does it return?

    Choose one answer.

    Show the answer

    Answer: An empty string

    DuckDB returns an empty string, while pandas and polars return a missing value. Wrap the call in NULLIF(…, '') so that missing values can be counted.

  6. Question 6 of 6 In the run of check_pins.py, how many of the 12 delivery notes ended with the status ok?

    Type a number.

    Show the answer

    Answer: 8

    Eight notes passed all checks, two repaired ones (D04 and D08) among them. Two had no PIN code, one held the placeholder 000000, and one held 600400, which is not in the directory.

Interview questions

Fresher · What is the difference between str.extract and str.extractall in pandas?

str.extract applies a pattern with at least one capture group to each row and keeps only the first match: one result row per input row, one column per group, columns named after (?P<name>…) groups, and a missing value where a row does not match. str.extractall keeps every match: one result row per match, with the original index plus a level called match that numbers the matches of each row from 0, and rows without a match are left out. Use str.extract when a row should hold one value, such as the PIN code of an address, and str.extractall when it may hold several, such as every phone number in a note. Then count the matches per row, so that rows with two candidates are reviewed instead of quietly reduced to the first.

Fresher · Why does df[“code”].str.contains(“2.5”) match OIL-235L, and how do you search for the text itself?

str.contains treats its argument as a regular expression by default (regex=True), and in a regular expression a dot matches any single character, so 2.5 also matches 235, 2X5 and 2-5. To search for the text itself, pass regex=False, or escape the dot as r"2\.5"; for a whole string that comes from a user, re.escape escapes every special character. polars has the same default, which literal=True switches off, while DuckDB’s contains() searches for plain text and regexp_matches() is its pattern version. pandas’ str.replace went the other way and reads plain text by default, so check the default of each method rather than assuming one rule for all of them.

Key takeaways

  • str.extract returns the first match of each capture group as columns, named after (?P<name>…) groups, with a missing value where a row does not match; str.extractall returns every match, with a match index level.
  • Put word boundaries around codes: \b[0-9]{6}\b keeps a PIN-code pattern out of phone numbers.
  • Write [0-9] when you mean the digits 0 to 9. \d also matches other scripts’ digits in Python’s re and in polars, but not in RE2, which pandas 3 uses for str.contains when PyArrow is installed.
  • List and classify the rows that fail; automate only the repairs that need no guess, and report how many rows each repair changed.
  • Check the structure with str.fullmatch, then the existence against a reference list, and give every row a status.
  • str.contains reads a pattern by default in pandas and polars, DuckDB’s contains() reads text, and pandas’ str.replace reads text.
  • Test every pattern on values it must accept and near misses it must reject.

References

Related tools

Regex Tester Live matches, capture groups and replace preview — safe from runaway patterns. PIN Code Finder (India Post) Every Indian PIN code and post office — search by number, place or district. IFSC Code Finder & Validator Find or check an IFSC — 1.8 lakh bank branches, searched on your device.

Report a problem with this lesson