Clean prices, dates, units and Indian numbers

Thirty price strings from the practice shop’s board and product pages write rupees with three different marks, two groupings of digits, words, units and ranges, from ₹25.00 and Rs. 2,450/- to INR 12,34,567.50 and ₹1.2 lakh, and the shortcut many scrapers start with, keep the digits and call float(), got 14 of them wrong without raising a single error. Dates are no safer: read month first, 132 of the board’s 365 dates in a year land on the wrong day, again without an error.

The cleaner built here turns price text into whole paise with exact arithmetic, accepts the Indian and Western ways of grouping digits and refuses every other, reads the board’s DD-MM-YYYY dates with an explicit format, keeps the +05:30 offset of Indian Standard Time on every time, and brings pack and quintal prices to one unit.

A price becomes whole paise, never a float

Money needs exact arithmetic, and binary floating point does not have it. The Python documentation’s own example is 1.1 + 2.2, which binary floating point shows as 3.3000000000000003, while a Decimal holds both numbers exactly and adds them to exactly 3.3. Prices fail the same way: 19.99 has no exact binary value, so float('19.99') * 100 is 1998.9999999999998, and int() cuts it to 1998. Store every price as a whole number of paisepaise: The hundredth parts of a rupee: 100 paise make one rupee. A price stored as a whole number of paise stays exact in every language and database, where a float would lose a paisa to rounding., and build that number from the text with Decimal, which keeps the digits as written.

Getting from the text to the number takes a few checks, and each can refuse the text with a reason instead of guessing: plain spaces and digits first, then a shape for the whole text (an optional ₹, Rs. or INR, an amount or a range, a word, a unit), then the grouping of the digits and at most two decimals. Follow four of the board’s strings through them:

The program runs the same cleaner on all 30 strings, then runs the shortcut on each one and lists every answer that differs:

"""The 30 price strings of the practice shop's board and product pages, cleaned into whole paise."""
import re
from collections import Counter
from pathlib import Path

from money import parse_price

texts = Path("prices.txt").read_text(encoding="utf-8").splitlines()
kinds = Counter()
for text in texts:
    result = parse_price(text)
    kinds["refused" if "refused" in result else "range" if "low" in result else "price"] += 1
    print(f"{text!r:24} {result}")
print(f"\n{len(texts)} strings: {kinds['price']} prices, {kinds['range']} range, {kinds['refused']} refused")


def float_paise(text):
    """The shortcut: keep digits and dots, read a float, times 100."""
    return int(float(re.sub(r"[^\d.]", "", text)) * 100)


answered, wrong = 0, []
for text in texts:
    result = parse_price(text)
    try:
        quick = float_paise(text)
    except ValueError:
        continue  # no digits at all: here, at least, the shortcut fails loudly
    answered += 1
    if result.get("paise") != quick:
        right = result.get("paise") or ("a range" if "low" in result else "refused")
        wrong.append((text, quick, right))
print(f"The shortcut gave a number for {answered} strings and was wrong for {len(wrong)}:")
for text, quick, right in wrong:
    print(f"  {text!r:24} shortcut {quick:>13}   cleaner {right}")

'₹25.00'                 {'paise': 2500}
'₹1,350.00'              {'paise': 135000}
'₹1,98,000'              {'paise': 19800000}
'Rs. 2,450/-'            {'paise': 245000}
'Rs 900'                 {'paise': 90000}
'INR 12,34,567.50'       {'paise': 123456750}
'₹\xa019.99'             {'paise': 1999}
'₹19.9'                  {'paise': 1990}
'₹2,10,000.00/qtl'       {'paise': 21000000, 'unit': 'qtl'}
'₹144.00 /kg'            {'paise': 14400, 'unit': 'kg'}
'₹1.2 lakh'              {'paise': 12000000}
'₹2.5 crore'             {'paise': 2500000000}
'₹1,200–1,500'           {'low': 120000, 'high': 150000}
'₹99'                    {'paise': 9900}
'Not traded today'       {'refused': 'no amount'}
'Price on request'       {'refused': 'no amount'}
'₹१,२५०'                 {'paise': 125000}
'₹1,23,4567'             {'refused': '1,23,4567 is grouped neither the Indian nor the Western way'}
'₹12,345,678'            {'paise': 1234567800}
'₹1.234,50'              {'refused': 'a decimal comma (1.234,50 or 1 350,00)'}
'₹0.50'                  {'paise': 50}
'₹7,000.005'             {'refused': '3 decimal places'}
'-₹50'                   {'refused': 'a negative amount'}
'USD 12.00'              {'refused': 'not rupees'}
'Rs.45'                  {'paise': 4500}
'₹5,00,000.00'           {'paise': 50000000}
'₹1 350,00'              {'refused': 'a decimal comma (1.234,50 or 1 350,00)'}
'₹1,00,00,000'           {'paise': 1000000000}
'₹4,500 per quintal'     {'paise': 450000, 'unit': 'qtl'}
'₹19.99'                 {'paise': 1999}

30 strings: 21 prices, 1 range, 8 refused
The shortcut gave a number for 28 strings and was wrong for 14:
  'Rs. 2,450/-'            shortcut            24   cleaner 245000
  '₹\xa019.99'             shortcut          1998   cleaner 1999
  '₹19.9'                  shortcut          1989   cleaner 1990
  '₹1.2 lakh'              shortcut           120   cleaner 12000000
  '₹2.5 crore'             shortcut           250   cleaner 2500000000
  '₹1,200–1,500'           shortcut    1200150000   cleaner a range
  '₹1,23,4567'             shortcut     123456700   cleaner refused
  '₹1.234,50'              shortcut           123   cleaner refused
  '₹7,000.005'             shortcut        700000   cleaner refused
  '-₹50'                   shortcut          5000   cleaner refused
  'USD 12.00'              shortcut          1200   cleaner refused
  'Rs.45'                  shortcut            45   cleaner 4500
  '₹1 350,00'              shortcut      13500000   cleaner refused
  '₹19.99'                 shortcut          1998   cleaner 1999

"""Turn the price text a page shows into whole paise (100 paise make 1 rupee), or say why it cannot."""
import re
import unicodedata
from decimal import Decimal

NUM = r"[0-9][0-9,]*(?:\.[0-9]+)?"
MARK = r"(?:₹|Rs\.?|INR)?\s*"
# What a price may look like: a rupee mark, an amount or a range, a word, a unit, and "/-" (no paise).
PRICE = re.compile(
    rf"{MARK}(?P<low>{NUM})(?:\s*(?:-|–|to)\s*{MARK}(?P<high>{NUM}))?"
    r"(?:\s*(?P<word>lakhs?|crores?))?(?:\s*(?:/|per\s)\s*(?P<unit>kg|g|qtl|quintal|tonne))?(?:/-)?",
    re.IGNORECASE | re.ASCII,
)
# Commas the Indian way (12,34,567), the Western way (1,234,567), or none at all.
GROUPED = re.compile(r"[0-9]{1,2}(?:,[0-9]{2})*,[0-9]{3}|[0-9]{1,3}(?:,[0-9]{3})+|[0-9]+")
WORDS = {"lakh": 100_000, "crore": 10_000_000}


def plain_spaces(text):
    """Every space character (no-break, narrow, thin ...) as one plain space."""
    text = "".join(" " if unicodedata.category(ch) == "Zs" else ch for ch in text)
    return " ".join(text.split())


def ascii_digits(text):
    """Digits of any script as 0-9, as long as the text uses one script: १,२५० is 1,250."""
    scripts, out = set(), []
    for ch in text:
        if unicodedata.category(ch) == "Nd":
            scripts.add(unicodedata.name(ch).split("DIGIT")[0].strip() or "LATIN")
            ch = str(unicodedata.decimal(ch))
        out.append(ch)
    if len(scripts) > 1:
        raise ValueError("digits from two scripts: " + " and ".join(sorted(scripts)))
    return "".join(out)


def amount(token, word):
    """One number, checked and scaled, in whole paise."""
    whole, _, decimals = token.partition(".")
    if not GROUPED.fullmatch(whole):
        raise ValueError(f"{whole} is grouped neither the Indian nor the Western way")
    if len(decimals) > 2 and not word:
        raise ValueError(f"{len(decimals)} decimal places")
    rupees = Decimal(whole.replace(",", "") + "." + (decimals or "0")) * WORDS.get(word, 1)
    paise = rupees * 100
    if paise != paise.to_integral_value():
        raise ValueError("not a whole number of paise")
    return int(paise)


def parse_price(text):
    """{'paise': n}, {'low': n, 'high': n} or {'refused': why}, with 'unit' when the text names one."""
    try:
        t = ascii_digits(plain_spaces(text))
    except ValueError as error:
        return {"refused": str(error)}
    if not re.search(r"[0-9]", t):
        return {"refused": "no amount"}
    if re.search(r"USD|EUR|GBP|[$€£]", t):
        return {"refused": "not rupees"}
    if t.startswith(("-", "−")):
        return {"refused": "a negative amount"}
    if re.search(r"[0-9],[0-9]{1,2}(?![0-9,])|\.[0-9]+,", t):
        return {"refused": "a decimal comma (1.234,50 or 1 350,00)"}
    m = PRICE.fullmatch(t)
    if not m:
        return {"refused": f"unexpected text in {t!r}"}
    word = (m["word"] or "").lower().removesuffix("s")
    try:
        low = amount(m["low"], word)
        high = amount(m["high"], word) if m["high"] else None
    except ValueError as error:
        return {"refused": str(error)}
    if high is not None and high < low:
        return {"refused": "a range that ends below its start"}
    result = {"paise": low} if high is None else {"low": low, "high": high}
    if m["unit"]:
        result["unit"] = "qtl" if m["unit"].lower() == "quintal" else m["unit"].lower()
    return result

₹25.00
₹1,350.00
₹1,98,000
Rs. 2,450/-
Rs 900
INR 12,34,567.50
₹ 19.99
₹19.9
₹2,10,000.00/qtl
₹144.00 /kg
₹1.2 lakh
₹2.5 crore
₹1,200–1,500
₹99
Not traded today
Price on request
₹१,२५०
₹1,23,4567
₹12,345,678
₹1.234,50
₹0.50
₹7,000.005
-₹50
USD 12.00
Rs.45
₹5,00,000.00
₹1 350,00
₹1,00,00,000
₹4,500 per quintal
₹19.99

The cleaner turned 21 strings into prices and one, ₹1,200–1,500, into a range from 120000 to 150000 paise, and refused 8, each with its reason. The shortcut gave a number for 28 strings, and 14 of those numbers were wrong. Three are the rounding above, ₹19.99 twice and ₹19.9. Two took the dot of Rs. for a decimal point, so Rs.45 became 45 paise. Two dropped the words lakh and crore and came out 100,000 and 10,000,000 times too small. The other seven accepted text that is not one price: a range, a negative discount, dollars, three decimals, two decimal commas and a grouping nobody writes.

Indian grouping puts pairs of digits before the last three

The board writes cardamom’s modal price as 2,10,000, and a product page writes a bulk order as 12,345,678. Both are correct. The Unicode standard for locale data describes the difference as two groupingdigit grouping: How a locale separates the digits of a large number. English (United States) writes 1,234,567, in threes; English (India) writes 12,34,567, with three digits at the right and pairs before them. sizes: most locales group every three digits, while a pattern such as #,##,##0, its example for Hindi, puts one group of three at the right and groups of two before it, so that 123456789 is written 12,34,56,789. English as written in India uses the same grouping, as the programs below print.

A cleaner that knows both shapes can accept them and refuse the rest. 1,23,4567, 12,3456 and 1,2,3 follow neither, and a cleaner that simply deletes commas, as the shortcut did, reads them as numbers without a word. In Python, formatting the Indian way takes a few lines of your own, and the cleaner’s pattern decides which groupings to accept:

"""Digit grouping both ways: format() groups in thousands only, so Indian grouping takes a few lines of its own."""
from money import GROUPED


def indian(n):
    """12345678 as 1,23,45,678: the last three digits, then pairs."""
    digits = str(n)
    head, tail = digits[:-3], digits[-3:]
    pairs = [head[max(0, i - 2):i] for i in range(len(head), 0, -2)]
    return ",".join(pairs[::-1] + [tail])


for n in (198000, 1234567, 123456789):
    print(f"{n:<10} format(n, ',') {format(n, ','):<12} indian(n) {indian(n)}")
for text in ("1,98,000", "198,000", "12,34,567", "1,234,567", "1,23,4567", "12,3456", "1,2,3"):
    print(f"{text:10} {'a valid grouping' if GROUPED.fullmatch(text) else 'refused'}")

198000     format(n, ',') 198,000      indian(n) 1,98,000
1234567    format(n, ',') 1,234,567    indian(n) 12,34,567
123456789  format(n, ',') 123,456,789  indian(n) 12,34,56,789
1,98,000   a valid grouping
198,000    a valid grouping
12,34,567  a valid grouping
1,234,567  a valid grouping
1,23,4567  refused
12,3456    refused
1,2,3      refused

Python’s format(n, ',') puts a comma every three digits, so the program writes its own indian(), and the cleaner’s GROUPED pattern accepted the four grouped numbers and refused the three others. JavaScript’s Intl.NumberFormat knows the locale’s grouping, and its number readers know none:

// How Node's Intl writes the board's numbers in two locales, and what JavaScript's own number parsers make of them.
const format = (locale, options, n) => new Intl.NumberFormat(locale, options).format(n);
const inr = { style: 'currency', currency: 'INR' };

for (const n of [198000, 1234567, 123456789]) {
  console.log(`${String(n).padEnd(10)} en-IN ${format('en-IN', {}, n).padEnd(12)} en-US ${format('en-US', {}, n)}`);
}
console.log('Rupees, en-IN:', format('en-IN', inr, 123456.5), '| Devanagari digits:', format('hi-IN-u-nu-deva', {}, 125000));
console.log('Compact, en-IN:', [198000, 1234567, 123456789].map((n) => format('en-IN', { notation: 'compact' }, n)).join(' '),
  '| long:', format('en-IN', { notation: 'compact', compactDisplay: 'long' }, 1234567));

// Reading numbers back: neither function knows digit grouping, and parseFloat stops quietly at the first comma.
for (const text of ['1,23,456', '123456', '१२५०', ' 42 ', '']) {
  console.log(JSON.stringify(text).padEnd(11), 'Number:', String(Number(text)).padEnd(7), 'parseFloat:', parseFloat(text));
}

198000     en-IN 1,98,000     en-US 198,000
1234567    en-IN 12,34,567    en-US 1,234,567
123456789  en-IN 12,34,56,789 en-US 123,456,789
Rupees, en-IN: ₹1,23,456.50 | Devanagari digits: १,२५,०००
Compact, en-IN: 2L 12L 12Cr | long: 12 lakh
"1,23,456"  Number: NaN     parseFloat: 1
"123456"    Number: 123456  parseFloat: 123456
"१२५०"      Number: NaN     parseFloat: NaN
" 42 "      Number: 42      parseFloat: 42
""          Number: 0       parseFloat: NaN

Reading back is where JavaScript bites: Number('1,23,456') is NaN, which at least fails, but parseFloat('1,23,456') stops quietly at the first comma and returns 1, a plausible price that nothing downstream will question. Number('१२५०') is NaN too, and so is every other number written in Devanagari digits.

A date needs its format written down

The board’s update line reads 05-10-2026. The board writes day, month and year, so that is day 5 of month 10, but nothing in the digits says so: read month first, the same text is day 10 of month 5, and no error tells you. Libraries that guess fall into exactly this. The dateparser library’s DATE_ORDER setting defaults to MDY, month first, and its documentation notes that English has no date order of its own; JavaScript’s Date reads strings outside its one defined format in a way MDN calls implementation-defined. Step through the two readings:

The fix is to write the format down, one per source, from the source’s own pages, and let the parser check the text against it: datetime.strptime reads exactly the format it is given and raises ValueError when the text does not match or the day does not exist. The program reads the shop’s five sources, each with its format, and ends with what a month-first reading does to a whole year of board dates:

"""The practice shop's dates in five formats, each read with the format its source uses, stored as ISO 8601."""
from datetime import date, datetime, timedelta, timezone
from email.utils import parsedate_to_datetime

IST = timezone(timedelta(hours=5, minutes=30), "IST")
FORMATS = {  # how each source writes its dates, from the source's own pages
    "board": "%d-%m-%Y",
    "board time": "%d-%m-%Y %H:%M IST",
    "news": "%d %b %Y",
    "bulletin": "%d/%m/%y",
}


def read(source, text):
    """A date, or an aware datetime, from one source's text."""
    if source == "api":
        return datetime.fromisoformat(text)  # ISO 8601 with its offset
    if source == "feed":
        return parsedate_to_datetime(text)  # the date format of email and RSS
    value = datetime.strptime(text, FORMATS[source])
    return value.replace(tzinfo=IST) if source == "board time" else value.date()


samples = [("board", "05-10-2026"), ("board time", "05-10-2026 14:30 IST"), ("news", "5 Oct 2026"),
           ("bulletin", "05/10/26"), ("api", "2026-10-05T09:00:00Z"), ("feed", "Mon, 05 Oct 2026 09:00:00 GMT")]
for source, text in samples:
    value = read(source, text)
    utc = f"  UTC {value.astimezone(timezone.utc).isoformat()}" if isinstance(value, datetime) else ""
    print(f"{source:10} {text!r:33} {value.isoformat()}{utc}")

for text in ("31-04-2026", "2026-10-05"):
    try:
        read("board", text)
    except ValueError as error:
        print(f"board      {text!r:33} ValueError: {error}")
print("bulletin   '05/10/69'", "->", read("bulletin", "05/10/69"), "(two-digit years 69-99 are 1969-1999)")

# What a month-first reader does to the board's DD-MM-YYYY dates, over every day of one year.
wrong = same = refused = 0
for n in range(365):
    day = date(2026, 1, 1) + timedelta(days=n)
    try:
        month_first = datetime.strptime(day.strftime("%d-%m-%Y"), "%m-%d-%Y").date()
    except ValueError:
        refused += 1
        continue
    wrong, same = (wrong, same + 1) if month_first == day else (wrong + 1, same)
print(f"Month first, the 365 board dates of 2026: {wrong} wrong without an error, {same} right by luck, {refused} refused")

board      '05-10-2026'                      2026-10-05
board time '05-10-2026 14:30 IST'            2026-10-05T14:30:00+05:30  UTC 2026-10-05T09:00:00+00:00
news       '5 Oct 2026'                      2026-10-05
bulletin   '05/10/26'                        2026-10-05
api        '2026-10-05T09:00:00Z'            2026-10-05T09:00:00+00:00  UTC 2026-10-05T09:00:00+00:00
feed       'Mon, 05 Oct 2026 09:00:00 GMT'   2026-10-05T09:00:00+00:00  UTC 2026-10-05T09:00:00+00:00
board      '31-04-2026'                      ValueError: day 31 must be in range 1..30 for month 4 in year 2026
board      '2026-10-05'                      ValueError: time data '2026-10-05' does not match format '%d-%m-%Y'
bulletin   '05/10/69' -> 1969-10-05 (two-digit years 69-99 are 1969-1999)
Month first, the 365 board dates of 2026: 132 wrong without an error, 12 right by luck, 221 refused

Every source came out as the same ISO 8601ISO 8601: The international standard for writing dates and times with the largest unit first, as YYYY-MM-DD, or YYYY-MM-DDThh:mm:ss followed by a UTC offset such as +05:30. RFC 3339 is its profile for internet protocols. date, 2026-10-05, and the board’s time kept its offset. 31-04-2026 was refused, because day 31 of month 4 does not exist, and so was an ISO date in the board’s column, because it is not the board’s format. Two-digit years carry a trap of their own: by the POSIX rule that the time documentation states, %y maps 69 to 99 onto 1969 to 1999, so a bulletin dated 05/10/69 lands in 1969. Month first, 132 of the year’s board dates were stored on the wrong day without an error, 12 such as 05-05-2026 were right by luck, and the 221 with a day of 13 or more were refused, which is the only way such a bug announces itself.

A time without its offset is only half a time

The board’s line updated 05-10-2026 14:30 IST names a moment only together with its zone. Indian Standard Time is UTC plus 5 hours 30 minutes: the tz database’s entry for Asia/Kolkata has kept that one offsetUTC offset: How far a local time is ahead of or behind UTC, written +hh:mm or -hh:mm after the time. Indian Standard Time is +05:30 all year, so 14:30 in India is 09:00 UTC., without daylight saving, since 1945. A Python datetime with no offset is what its documentation calls a naivenaive datetime: A Python date and time without a UTC offset. It is a clock reading rather than a moment: each computer that turns it into a timestamp assumes its own time zone. object, without enough information to place itself among other times, and its timestamp() assumes the computer’s local time. Follow the update time from the board to a database on a server set to UTC:

Kept with its offset, the time is written 2026-10-05T14:30:00+05:30, the form of RFC 3339, the profile of ISO 8601 for internet protocols, and the dates program printed it that way and as 09:00:00+00:00 in UTC. Store one of the two: both name the same moment on any computer. JavaScript makes the bare time easy to get wrong. MDN states that an ISO date without a time is read as UTC and an ISO date and time without an offset as the computer’s local time, and leaves every other string to the engine:

// What JavaScript's Date makes of the board's dates, on a collector whose clock is set to India's time zone.
process.env.TZ = 'Asia/Kolkata';

const show = (text) => {
  const d = new Date(text);
  console.log(JSON.stringify(text).padEnd(28), Number.isNaN(d.getTime()) ? 'Invalid Date' : d.toISOString());
};
show('05-10-2026'); // not a format JavaScript defines: this engine reads it month first
show('13-10-2026'); // month 13 does not exist, so the same reading fails
show('2026-10-05'); // date only, ISO: midnight UTC
show('2026-10-05T14:30'); // date and time, no offset: the computer's local time
show('2026-10-05T14:30:00+05:30'); // the offset written down: one moment everywhere

// Reading the board's format yourself leaves nothing to the engine.
const [day, month, year] = '05-10-2026'.split('-').map(Number);
const noon = new Date(Date.UTC(year, month - 1, day, 14, 30) - 330 * 60_000); // 14:30 IST is 09:00 UTC
console.log('Read day first:'.padEnd(28), noon.toISOString());
console.log('Shown for India:'.padEnd(28), new Intl.DateTimeFormat('en-IN', { dateStyle: 'medium', timeStyle: 'short', timeZone: 'Asia/Kolkata' }).format(noon));

"05-10-2026"                 2026-05-09T18:30:00.000Z
"13-10-2026"                 Invalid Date
"2026-10-05"                 2026-10-05T00:00:00.000Z
"2026-10-05T14:30"           2026-10-05T09:00:00.000Z
"2026-10-05T14:30:00+05:30"  2026-10-05T09:00:00.000Z
Read day first:              2026-10-05T09:00:00.000Z
Shown for India:             5 Oct 2026, 2:30 pm

On a computer set to India’s zone, 05-10-2026 became 2026-05-09T18:30:00.000Z: read month first, as local midnight of day 10 of month 5, which in UTC is still the evening before. 13-10-2026 was an Invalid Date. The ISO date and time without an offset was taken as India’s local time, 09:00 UTC, and on a server in UTC it would be taken as 14:30 UTC. Splitting the board’s format yourself and writing the offset leaves nothing to the engine.

Units and digits come from tables you write down

The board quotes cardamom at ₹2,10,000.00 a quintal and the shop sells a 100 g pack at ₹245.00. Neither number means anything next to the other until both are in one unit. Write the units down as a table, with each source’s own definition (the board’s legend says that a quintal is 100 kg), convert with exact arithmetic, and refuse any unit the table does not have instead of assuming kilograms:

"""Prices per pack and per quintal brought to one unit, and digits from other scripts read on purpose."""
import re
from decimal import Decimal, ROUND_HALF_UP

from money import ascii_digits, parse_price

# Kilograms in each unit. The board's legend says "1 quintal = 100 kg"; a unit missing here is refused.
KG = {"g": Decimal("0.001"), "kg": Decimal(1), "qtl": Decimal(100), "tonne": Decimal(1000)}


def paise_per_kg(paise, quantity, unit):
    """Whole paise per kilogram, rounded half up, or None for a unit the table does not know."""
    if unit not in KG or quantity <= 0:
        return None
    return int((Decimal(paise) / (Decimal(quantity) * KG[unit])).quantize(Decimal(1), ROUND_HALF_UP))


rows = [  # (what, price text as shown, quantity, unit)
    ("Cardamom, board modal", "₹2,10,000.00", 1, "qtl"),
    ("Cardamom, shop 100 g pack", "₹245.00", 100, "g"),
    ("Onion, board modal", "₹2,150", 1, "qtl"),
    ("Onion, shop 1 kg bag", "₹28.00", 1, "kg"),
    ("Jaggery, shop 5 kg", "₹410.00", 5, "kg"),
    ("Mustard oil, 1 tin", "₹1,650.00", 1, "tin"),
]
per_kg = {}
for what, text, quantity, unit in rows:
    value = paise_per_kg(parse_price(text)["paise"], quantity, unit)
    per_kg[what] = value
    shown = f"refused: no unit {unit!r} in the table" if value is None else f"₹{Decimal(value) / 100:,.2f} per kg"
    print(f"{what:27} {text:>13} for {quantity} {unit:5} -> {shown}")
for item in ("Cardamom", "Onion"):
    shop = next(v for k, v in per_kg.items() if k.startswith(item + ", shop"))
    board = per_kg[f"{item}, board modal"]
    print(f"{item}: the shop charges {(shop - board) / board:.1%} above the board's modal price")

print("\nDigits of other scripts:")
print("int('१२५०') =", int("१२५०"), "| re.findall(r'\\d+', '₹१,२५०') =", re.findall(r"\d+", "₹१,२५०"))
print("with re.ASCII:", re.findall(r"\d+", "₹१,२५०", re.ASCII), "| int('1२5') =", int("1२5"), "(two scripts, no error)")
for text in ("₹१,२५०", "₹1२5"):
    try:
        print(f"ascii_digits({text!r}) = {ascii_digits(text)!r}")
    except ValueError as error:
        print(f"ascii_digits({text!r}) refused: {error}")

Cardamom, board modal        ₹2,10,000.00 for 1 qtl   -> ₹2,100.00 per kg
Cardamom, shop 100 g pack         ₹245.00 for 100 g     -> ₹2,450.00 per kg
Onion, board modal                 ₹2,150 for 1 qtl   -> ₹21.50 per kg
Onion, shop 1 kg bag               ₹28.00 for 1 kg    -> ₹28.00 per kg
Jaggery, shop 5 kg                ₹410.00 for 5 kg    -> ₹82.00 per kg
Mustard oil, 1 tin              ₹1,650.00 for 1 tin   -> refused: no unit 'tin' in the table
Cardamom: the shop charges 16.7% above the board's modal price
Onion: the shop charges 30.2% above the board's modal price

Digits of other scripts:
int('१२५०') = 1250 | re.findall(r'\d+', '₹१,२५०') = ['१', '२५०']
with re.ASCII: [] | int('1२5') = 125 (two scripts, no error)
ascii_digits('₹१,२५०') = '₹1,250'
ascii_digits('₹1२5') refused: digits from two scripts: DEVANAGARI and LATIN

Per kilogram, the board’s cardamom costs ₹2,100.00 and the pack ₹2,450.00, so the shop charges 16.7 % more, and its 1 kg bag of onions costs 30.2 % more than the board’s modal ₹21.50. The tin was refused: until the source says what a tin holds, any number would be invented.

Digits need the same care. Python’s int() accepts every Unicode decimal digit, so int('१२५०') is 1250, and the re module’s \d matches all of them as well. That sounds helpful until two scripts meet in one string: int('1२5') returns 125 without complaint. The cleaner’s ascii_digits() converts digits on purpose, one script per number, and refused the mix. Spaces get the same treatment: the no-break space U+00A0 and the narrow no-break space U+202F become plain spaces before any pattern runs, and the encodings lesson covers normalisingUnicode normalization: Turning text into one standard form so that canonically equivalent strings compare equal. NFC composes characters such as e plus a combining accent into é wherever Unicode allows it. More in “Encodings: getting Hindi text and ₹ right” the rest of the text.

Practise: build the board’s cleaner

Three small functions make the cleaner of this lesson: price text into paise, the board’s dates and times into ISO 8601, and any price into paise per kilogram. Each step has sample tests built from the board’s own strings:

def to_paise(text):
    """Whole paise for a price text such as '₹1,98,000' or 'Rs. 2,450/-', or None."""
    # Replace this: float() turns 19.99 rupees into 1998 paise, and it refuses every grouped number.
    try:
        return int(float(text.replace("₹", "")) * 100)
    except ValueError:
        return None

from prices import to_paise


def test_amounts_and_decimals():
    """whole rupees, one decimal and two, in exact arithmetic"""
    assert to_paise("₹25.00") == 2500
    assert to_paise("₹99") == 9900
    assert to_paise("₹19.9") == 1990
    assert to_paise("₹19.99") == 1999, "exact decimals: 19.99 rupees are 1999 paise, never 1998"


def test_indian_and_western_grouping():
    """pairs of digits before the last three, or groups of three"""
    assert to_paise("₹1,98,000") == 19800000
    assert to_paise("INR 12,34,567.50") == 123456750
    assert to_paise("₹12,345,678") == 1234567800


def test_marks_spaces_and_no_paise():
    """₹, Rs., Rs and INR in any case, no-break spaces and the /- ending"""
    assert to_paise("Rs. 2,450/-") == 245000
    assert to_paise("rs 900") == 90000
    assert to_paise("Rs.45") == 4500, "the dot of Rs. is not a decimal point"
    assert to_paise("₹\u00a019.99") == 1999, "a no-break space after the sign"


def test_not_one_price():
    """no amount, a wrong grouping, three decimals and a range"""
    assert to_paise("Not traded today") is None
    assert to_paise("₹1,23,4567") is None, "grouped neither the Indian nor the Western way"
    assert to_paise("₹7,000.005") is None, "three decimal places"
    assert to_paise("₹1,200–1,500") is None, "a range is not one price"

from datetime import datetime, timedelta, timezone

IST = timezone(timedelta(hours=5, minutes=30))


def to_iso(text):
    """'05-10-2026' as '2026-10-05', '05-10-2026 14:30 IST' as '2026-10-05T14:30:00+05:30', anything else as None."""
    # Replace this: it reads the date month first, and it knows nothing of times.
    try:
        return datetime.strptime(text, "%m-%d-%Y").date().isoformat()
    except ValueError:
        return None

from dates import to_iso


def test_dates_are_read_day_first():
    """DD-MM-YYYY, including the first twelve days of a month"""
    assert to_iso("05-10-2026") == "2026-10-05", "day 05 of month 10, not day 10 of month 05"
    assert to_iso("13-10-2026") == "2026-10-13"
    assert to_iso("01-12-2026") == "2026-12-01"


def test_times_keep_the_offset():
    """DD-MM-YYYY HH:MM IST, with +05:30"""
    assert to_iso("05-10-2026 14:30 IST") == "2026-10-05T14:30:00+05:30"
    assert to_iso("31-12-2026 23:59 IST") == "2026-12-31T23:59:00+05:30"


def test_anything_else_is_none():
    """a day that does not exist, and other formats"""
    assert to_iso("31-04-2026") is None, "month 04 has 30 days"
    assert to_iso("2026-10-05") is None, "ISO is not the board's format"
    assert to_iso("05/10/2026") is None
    assert to_iso("") is None

from decimal import Decimal

KG = {"g": Decimal("0.001"), "kg": Decimal(1), "qtl": Decimal(100), "tonne": Decimal(1000)}


def paise_per_kg(paise, quantity, unit):
    """Whole paise per kilogram, rounded half up, or None for an unknown unit or a quantity of 0 or less."""
    # Replace this: it treats every unit as a kilogram.
    return paise // quantity

from perkg import paise_per_kg


def test_kilograms_and_grams():
    """packs sold by the kilogram and by the gram"""
    assert paise_per_kg(2800, 1, "kg") == 2800
    assert paise_per_kg(41000, 5, "kg") == 8200
    assert paise_per_kg(24500, 100, "g") == 245000, "a 100 g pack at 245 rupees is 2,450 rupees a kilogram"


def test_quintals_and_tonnes():
    """the board's units"""
    assert paise_per_kg(21000000, 1, "qtl") == 210000
    assert paise_per_kg(215000, 1, "qtl") == 2150
    assert paise_per_kg(1500000, 1, "tonne") == 1500


def test_rounding_half_up():
    """halves round up, never to the even neighbour"""
    assert paise_per_kg(1999, 2, "kg") == 1000
    assert paise_per_kg(1997, 2, "kg") == 999, "998.5 rounds up to 999"
    assert paise_per_kg(100, 3, "kg") == 33


def test_unknown_units_and_quantities():
    """no guess for a unit the table does not have"""
    assert paise_per_kg(165000, 1, "tin") is None
    assert paise_per_kg(2800, 0, "kg") is None

Then test yourself on paise, grouping, dates, offsets and units, weakest idea first:

Interview questions

Warm-up (fresher): Why should a scraper store prices as whole paise, or as Decimal, and never as float?

Because binary floating point cannot hold most decimal fractions exactly. 19.99 is stored as the nearest binary fraction, so 19.99 * 100 is 1998.9999999999998 and int() gives 1998: a paisa lost on one price, and sums of thousands of prices drift further. Whole paise are integers, so adding, comparing and storing them is exact in every language and database, and Decimal keeps the digits as written when a value needs fractions of a paisa, such as a price per gram. The conversion matters as much as the type: build the paise from the text with Decimal, never by way of float(), and keep the currency and the unit beside the number, because 245 rupees per 100 g and per kilogram are different prices.

Two harder questions start from this lesson: totals that drift by paise while a few prices are 100 times too low, and a dashboard whose activity piles up on the first twelve days of some months. The module’s bank of interview questions gives each one’s first step and follow-up questions:

Key takeaways

• Keep every price as a whole number of paise built from the text with Decimal, and refuse text that is not exactly one price, with the reason.
• Accept the two groupings real pages use, Indian (12,34,567) and Western (1,234,567), refuse every other, and treat words such as lakh and crore as part of the amount.
• Read each source’s dates with that source’s own format: month first, DD-MM-YYYY text puts the first twelve days of every month on the wrong day without an error.
• Store every time with its UTC offset or in UTC, because a naive time is a clock reading that each computer places differently.
• Convert units with a table written from each source’s own definitions, refuse units the table lacks, and convert digits of other scripts on purpose, one script per number.