Web Scraping & Browser Automation (Python and JavaScript) Module 2 – HTTP, sessions, pagination and APIs
Encodings: getting Hindi text and ₹ right
Find a page's encoding from its BOM, header or meta tag, repair mojibake from client defaults, write UTF-8 files and normalise text before comparing.
What you will learn
- Find a page's character encoding from its byte-order mark, its Content-Type header or its meta tag, in that order
- Repair mojibake caused by a client's wrong default encoding, and decode legacy pages correctly
- Write text files and CSVs with an explicit UTF-8 encoding that spreadsheet apps read correctly
- Normalise text to NFC and clean invisible characters before comparing or storing it
Before you start
What you’ll learn · Before you start
What you will learn
- Find a page's character encoding from its byte-order mark, its Content-Type header or its meta tag, in that order
- Repair mojibake caused by a client's wrong default encoding, and decode legacy pages correctly
- Write text files and CSVs with an explicit UTF-8 encoding that spreadsheet apps read correctly
- Normalise text to NFC and clean invisible characters before comparing or storing it
Before you start
On this page
The 20 characters of a Hindi product title travel as 43 bytes, and read with the wrong table they arrive as
बासमती, mojibake: Text decoded with another encoding than the one it was written in, so that characters turn into wrong ones, such as é for é. No error is raised, which is why it spreads unnoticed., with no error to warn you. A collector that stores that text has lost the
names of its products until someone notices.
Here you see how text becomes bytes and back, find a page’s encoding in the order the HTML standard uses, see which client defaults produce mojibake and how to repair it, write CSV files that spreadsheet apps open correctly, and normalise text so that two names that look the same also compare the same.
Text travels as bytes, and an encoding turns them back into characters
Every character has a number, its code point: A is U+0041, é U+00E9, the Devanagari letter ब U+092C and the rupee
sign ₹ U+20B9. A character encoding: A table that turns characters into bytes and back, such as UTF-8 or windows-1252. Bytes carry no note of their encoding, so it must be declared or guessed. is the table that turns those numbers into
bytes and back. UTF-8, which the Encoding Standard calls “the most appropriate encoding for interchange of Unicode”, uses
1 byte for ASCII, 2 for many Latin and other letters, 3 for Devanagari, the rupee sign and every other character up
to U+FFFF, and 4 for those beyond it, such as emoji. Step through one title on its way out and back:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Characters to bytes. The title has 20 characters. UTF-8 writes the Latin letters and digits as 1 byte each, the middle dot as 2 and each Devanagari letter and the rupee sign as 3: 43 bytes in all.
- Only bytes travel. The network and the file carry only the bytes. Nothing in e0 a4 ac says which table turns it back into a character: the encoding has to be declared, or guessed.
- The right table. Decoded with UTF-8, the table it was written with, e0 a4 ac becomes one character, BA, and the title comes back exactly: 20 characters.
- The wrong table. windows-1252 gives almost every byte a character of its own, so e0 a4 ac becomes three: à, ¤ and ¬. Nothing fails, and 43 characters of mojibake replace the title.
- Undoing it. No byte was lost: encoding the mojibake back with windows-1252 restores the 43 bytes, and reading those as UTF-8 restores the title. That repair works only when every byte survived.
The program below prints the code point and UTF-8 bytes of six characters, then what those bytes become when they are read as windows-1252, the table of many old Western pages, which gives almost every byte a character of its own:
"""Characters, their code points and their UTF-8 bytes, and what the bytes become when read with the wrong table."""
import unicodedata
print(f"{'char':4} {'code point':10} {'UTF-8 bytes':12} {'read as windows-1252':20} name")
for char in ["A", "é", "–", "₹", "ब", "च"]:
data = char.encode("utf-8")
print(f"{char:4} U+{ord(char):04X} {data.hex(' '):12} {data.decode('cp1252'):20} {unicodedata.name(char)}")
title = "बासमती चावल · ₹25.00"
data = title.encode("utf-8")
print()
print(len(title), "characters became", len(data), "bytes")
wrong = data.decode("cp1252")
print("read as windows-1252:", wrong)
print("encoded back the same wrong way and read as UTF-8:", wrong.encode("cp1252").decode("utf-8"))Output
char code point UTF-8 bytes read as windows-1252 name A U+0041 41 A LATIN CAPITAL LETTER A é U+00E9 c3 a9 é LATIN SMALL LETTER E WITH ACUTE – U+2013 e2 80 93 – EN DASH ₹ U+20B9 e2 82 b9 ₹ INDIAN RUPEE SIGN ब U+092C e0 a4 ac ब DEVANAGARI LETTER BA च U+091A e0 a4 9a च DEVANAGARI LETTER CA 20 characters became 43 bytes read as windows-1252: बासमती चावल · ₹25.00 encoded back the same wrong way and read as UTF-8: बासमती चावल · ₹25.00
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 bytes_and_text.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The ASCII letter came through unchanged, which is why mojibake hides in English text and shows in everything else.
Each 3-byte character became three characters, so the 20-character title became 43 characters of noise, beginning
with ब. The last line is the repair: encode the mojibake back with the table that was wrongly used, and decode the
bytes with the right one. It works only when no byte was lost or replaced on the way.
Repairing text that has already been replaced
The repair undoes a wrong decode, not a lossy one. Once a decoder has replaced bytes with ? or � (U+FFFD, the
replacement character), the original bytes are gone and no table brings them back. Fix the decode at the source,
where you still have the bytes, instead of repairing stored text.
Three places declare the encoding, and a BOM beats them all
A page can say what its encoding is in three places. A byte-order mark: The character U+FEFF written at the very start of a text to identify its encoding: the bytes EF BB BF mean UTF-8, FE FF and FF FE the two byte orders of UTF-16., the bytes
EF BB BF at the very start, says UTF-8. The charset parameter of the Content-Type header, as in text/html; charset=utf-8, is the HTTP way; RFC
9110 compares charset names in any case. And the page itself can carry <meta charset="utf-8"> near the top. The HTML
standard’s sniffing algorithm checks them in a fixed order: the byte-order mark first, then the header, then a prescan
of the start of the page, which it encourages user agents to keep to the first 1024 bytes, and only then a default.
The Encoding Standard explains why the BOM goes first: so that pages already published keep working, it treats the
mark as “more authoritative than anything else”. Follow three of the shop’s pages through the checks:
The drawing shows the whole scene; its 4 steps are listed below.
Every step as text
- A BOM against the header. /encoding/bom says windows-1252 in its header, but its body starts with EF BB BF, the UTF-8 byte-order mark. The mark wins: the Encoding Standard treats it as more authoritative than anything else.
- An old page, honestly labelled. /encoding/legacy has no BOM, and its header says windows-1252. That is true of its bytes, so its curly quotes and en dash come out right.
- The header is silent. /encoding/meta-only sends text/html with no charset. The page itself declares meta charset utf-8 in its first bytes, so the meta tag decides.
- Nothing declared. A page with no BOM, no charset and no meta tag gets a guess. Bytes that match UTF-8's pattern are very likely UTF-8, as the HTML standard notes; anything else is read as windows-1252, the usual default for old Western pages.
The shop serves the same product page five ways: with a correct header, with only a meta tag, with a BOM that contradicts its header, as an old windows-1252 page, and with nothing declared at all. The program decodes each in the standard’s order and prints the rule that decided:
text/where_declared.py
"""Five copies of one page, each declaring its encoding differently, decoded in the order the HTML standard uses."""
import codecs
import re
import nimbu_lab as lab
shop = lab.opener()
BOMS = [(codecs.BOM_UTF8, "utf-8"), (codecs.BOM_UTF16_BE, "utf-16-be"), (codecs.BOM_UTF16_LE, "utf-16-le")]
def decode_body(content_type, body):
"""The text of an HTML body and the rule that chose its encoding: BOM, header, meta tag, then a guess."""
for bom, name in BOMS:
if body.startswith(bom):
return body[len(bom):].decode(name), f"BOM ({name})"
declared = re.search(r"charset=([\w-]+)", content_type or "", re.I)
if declared:
return body.decode(declared.group(1)), f"header ({declared.group(1)})"
meta = re.search(rb"<meta[^>]*charset=[\"']?([\w-]+)", body[:1024], re.I)
if meta:
return body.decode(meta.group(1).decode()), f"meta tag ({meta.group(1).decode()})"
try:
return body.decode("utf-8"), "guess: valid UTF-8"
except UnicodeDecodeError:
return body.decode("cp1252"), "guess: windows-1252"
def title(html):
return re.search(r"<title>(.*?)</title>", html).group(1)
for path in ["/encoding/declared", "/encoding/meta-only", "/encoding/bom", "/encoding/legacy", "/encoding/undeclared"]:
with shop.open(lab.SITE + path, timeout=10) as response:
kind, body = response.headers["Content-Type"], response.read()
html, rule = decode_body(kind, body)
print(f"{path:21} {kind:31} -> {rule}")
print(f"{'':21} {title(html)}")
if "charset" not in kind:
# The ISO-8859-1 rule some clients apply to text/* without a charset, ignoring the meta tag:
print(f"{'':21} as ISO-8859-1: {title(body.decode('iso-8859-1'))!r}")Output
/encoding/declared text/html; charset=utf-8 -> header (utf-8)
बासमती चावल 250 g · ₹25.00
/encoding/meta-only text/html -> meta tag (utf-8)
बासमती चावल 250 g · ₹25.00
as ISO-8859-1: 'बासमतà¥\x80 à¤\x9aावल 250 g · â\x82¹25.00'
/encoding/bom text/html; charset=windows-1252 -> BOM (utf-8)
बासमती चावल 250 g · ₹25.00
/encoding/legacy text/html; charset=windows-1252 -> header (windows-1252)
Kashmiri ‘Mongra’ saffron – 1 g · Rs. 310
/encoding/undeclared text/html -> guess: valid UTF-8
बासमती चावल 250 g · ₹25.00
as ISO-8859-1: 'बासमतà¥\x80 à¤\x9aावल 250 g · â\x82¹25.00'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 where_declared.py
nimbu_lab.py (the shop) · text/nimbu_lab.py
"""Nimbu Bazaar, the practice shop of this track. Every product, price and person in it is invented.
The shop is a WSGI application (PEP 3333): app() answers one request. opener() builds Python's own HTTP client,
urllib.request, with its usual handlers for redirects, errors and cookies, and one change: the last step hands each
request to app() instead of opening a socket. The client code is real, nothing leaves your program, and any host but
the shop's gets 421 Misdirected Request.
"""
import fnmatch
import http.client
import http.cookies
import io
import json
import socket
import urllib.request
import urllib.response
from urllib.parse import parse_qs, urlsplit
HOST = "bazaar.nimbu.test"
SITE = f"https://{HOST}"
KINDS = ["Basmati rice", "Toor dal", "Green tea", "Jaggery", "Turmeric", "Mustard oil", "Poha", "Rock salt",
"Cashews", "Cardamom", "Ragi flour", "Ghee", "Honey", "Peanuts", "Coffee", "Raisins", "Sesame seeds",
"Tamarind", "Cumin", "Black pepper", "Rolled oats", "Cloves", "Fennel", "Moong dal", "Chana dal",
"Wheat flour", "Semolina", "Coriander", "Red chilli", "Almonds", "Dates", "Coconut oil", "Brown rice",
"Millet", "Flax seeds", "Cinnamon", "Ginger powder", "Vermicelli", "Masoor dal", "Sago"]
SIZES = [250, 500, 1000, 2000, 5000, 10000] # grams
LOG = [] # what the shop saw: one entry per request
class Clock:
"""The shop's clock, in seconds. A slow page moves it on instead of waiting, so every example runs at once."""
def __init__(self):
self.seconds = 0.0
def now(self):
return self.seconds
def sleep(self, seconds):
self.seconds += seconds
clock = Clock()
def products():
"""The 240 products: 40 kinds in 6 pack sizes, each price in paise (100 paise = 1 rupee)."""
out = []
for k, kind in enumerate(KINDS):
per_kg = 60 + (k * 37) % 900 # rupees a kilogram
for grams in SIZES:
size = f"{grams // 1000} kg" if grams >= 1000 else f"{grams} g"
out.append({"sku": f"NB-{len(out) + 1:04d}", "name": f"{kind} {size}",
"paise": per_kg * grams // 10 + 1000})
return out
def rupees(paise):
return f"₹{paise // 100:,}.{paise % 100:02d}"
class Asked:
"""One request as the shop's pages see it."""
def __init__(self, environ):
self.method, self.path = environ["REQUEST_METHOD"], environ["PATH_INFO"]
self.query = {k: v[-1] for k, v in parse_qs(environ.get("QUERY_STRING", "")).items()}
self.headers = {k[5:].replace("_", "-").title(): v for k, v in environ.items() if k.startswith("HTTP_")}
size = int(environ.get("CONTENT_LENGTH") or 0)
self.form = {k: v[-1] for k, v in parse_qs(environ["wsgi.input"].read(size).decode()).items()}
self.cookies = {k: m.value for k, m in http.cookies.SimpleCookie(environ.get("HTTP_COOKIE", "")).items()}
self.wait = environ.get("lab.wait", clock.sleep) # the shop takes this long before its next bytes
PAGES = {} # path pattern ("/products/NB-*/" matches every product) -> function(asked) -> (status, headers, body)
def page(pattern):
def register(fn):
PAGES[pattern] = fn
return fn
return register
def find_page(path):
exact = PAGES.get(path)
return exact or next((fn for p, fn in PAGES.items() if "*" in p and fnmatch.fnmatchcase(path, p)), None)
def text(status, body, kind="text/plain; charset=utf-8", extra=()):
data = body.encode() if isinstance(body, str) else body
return status, [("Content-Type", kind), ("Content-Length", str(len(data))), *extra], data
def not_found(asked):
return text("404 Not Found", "no such page\n")
def app(environ, start_response):
"""The shop: answers one request, and writes it into LOG first."""
asked = Asked(environ)
if environ.get("HTTP_HOST") != HOST:
status, headers, body = text("421 Misdirected Request", "not a Nimbu Bazaar address\n")
else:
status, headers, body = (find_page(asked.path) or not_found)(asked)
target = asked.path + ("?" + environ["QUERY_STRING"] if environ.get("QUERY_STRING") else "")
LOG.append({"line": f"{asked.method} {target} HTTP/1.1", "headers": environ.get("lab.sent", []),
"status": int(status[:3])})
start_response(status, headers)
return [body] if isinstance(body, bytes) else body
class Stream(io.RawIOBase):
"""A response body read chunk by chunk, as from a socket: a slow chunk can time out in the middle of read()."""
def __init__(self, chunks):
self.chunks, self.rest = iter(chunks), b""
def readable(self):
return True
def readinto(self, buffer):
while not self.rest:
self.rest = next(self.chunks, None)
if self.rest is None:
self.rest = b""
return 0
n = min(len(buffer), len(self.rest))
buffer[:n], self.rest = self.rest[:n], self.rest[n:]
return n
class ShopTransport(urllib.request.AbstractHTTPHandler):
"""urllib's last step, with app() where the socket would be. Everything before it is urllib's own code."""
handler_order = 100 # runs before any handler that would open a real connection
http_request = https_request = urllib.request.AbstractHTTPHandler.do_request_
def https_open(self, req):
# The headers urllib's do_open() writes to the socket, and the one http.client adds by itself.
headers = dict(req.unredirected_hdrs)
headers.update({k: v for k, v in req.headers.items() if k not in headers})
headers["Connection"] = "close"
sent = [(k.title(), v) for k, v in headers.items()]
if "Accept-encoding" not in headers:
sent.insert(0, ("Accept-Encoding", "identity"))
timeout = socket.getdefaulttimeout() if req.timeout is socket._GLOBAL_DEFAULT_TIMEOUT else req.timeout
def wait(seconds):
"""A socket in timeout mode gives up when one wait for bytes lasts longer than the timeout."""
if timeout is not None and seconds > timeout:
clock.sleep(timeout)
raise TimeoutError("timed out")
clock.sleep(seconds)
parts, data = urlsplit(req.full_url), req.data or b""
environ = {"REQUEST_METHOD": req.get_method(), "PATH_INFO": parts.path or "/", "QUERY_STRING": parts.query,
"SERVER_NAME": parts.hostname or "", "SERVER_PORT": str(parts.port or 443),
"SERVER_PROTOCOL": "HTTP/1.1", "wsgi.url_scheme": parts.scheme, "wsgi.input": io.BytesIO(data),
"CONTENT_TYPE": headers.get("Content-type", ""), "CONTENT_LENGTH": str(len(data)),
"lab.wait": wait, "lab.sent": sent}
environ.update({"HTTP_" + k.upper().replace("-", "_"): v for k, v in sent
if k not in ("Content-Type", "Content-Length")})
answer = {}
body = app(environ, lambda status, hdrs, exc_info=None: answer.update(status=status, headers=hdrs))
raw = "".join(f"{k}: {v}\r\n" for k, v in answer["headers"]) + "\r\n"
message = http.client.parse_headers(io.BytesIO(raw.encode("iso-8859-1")))
response = urllib.response.addinfourl(io.BufferedReader(Stream(body)), message, req.full_url,
int(answer["status"][:3]))
response.reason = response.msg = answer["status"][4:] # as urllib's own responses carry them
return response
http_open = https_open
DEFAULTS = [urllib.request.HTTPDefaultErrorHandler, urllib.request.HTTPRedirectHandler,
urllib.request.HTTPErrorProcessor, urllib.request.UnknownHandler]
def opener(*handlers):
"""Like urllib.request.build_opener(*handlers), with the shop in place of the network: urllib's usual handlers,
except those a handler you pass replaces (a subclass of HTTPRedirectHandler, say)."""
director = urllib.request.OpenerDirector()
kept = [d for d in DEFAULTS if not any(isinstance(h, d) for h in handlers)]
for handler in [ShopTransport(), *(d() for d in kept), *handlers]:
director.add_handler(handler)
return director
# ---- The pages this lesson's examples ask for ----------------------------------------------------------------------
# One product page, five ways of declaring (or not declaring) its encoding. The bytes are what the shop sends.
TITLE = "बासमती चावल 250 g · ₹25.00"
UTF8_PAGE = f'<!doctype html>\n<html lang="hi"><head><meta charset="utf-8"><title>{TITLE}</title></head></html>\n'
NO_META_PAGE = f"<!doctype html>\n<html lang=\"hi\"><head><title>{TITLE}</title></head></html>\n"
LEGACY_PAGE = "<!doctype html>\n<title>Kashmiri ‘Mongra’ saffron – 1 g · Rs. 310</title>\n" # an old supplier's page
ENCODING_PAGES = {
"/encoding/declared": ("text/html; charset=utf-8", UTF8_PAGE.encode("utf-8")),
"/encoding/meta-only": ("text/html", UTF8_PAGE.encode("utf-8")),
"/encoding/bom": ("text/html; charset=windows-1252", b"\xef\xbb\xbf" + NO_META_PAGE.encode("utf-8")),
"/encoding/legacy": ("text/html; charset=windows-1252", LEGACY_PAGE.encode("cp1252")),
"/encoding/undeclared": ("text/html", NO_META_PAGE.encode("utf-8")),
}
@page("/encoding/*")
def encoding_page(asked):
if asked.path not in ENCODING_PAGES:
return not_found(asked)
kind, body = ENCODING_PAGES[asked.path]
return text("200 OK", body, kind)Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Every page came out right, the legacy one with its curly quotes and en dash. The BOM overruled a header that said
windows-1252; the meta tag decided where the header was silent; the undeclared page was valid UTF-8, and the HTML
standard notes that bytes matching UTF-8’s distinctive pattern are very likely UTF-8. For the two pages without a
charset, the program also printed what the ISO-8859-1 rule gives: the same
mojibake as before, with invisible control characters such as \x9a where windows-1252 would have shown symbols.
Key idea
Decode a page’s bytes in the standard’s order, the byte-order mark, then the header’s charset, then a meta tag in the first 1024 bytes, then a guess, and keep the bytes until you have.
Checkpoint
Check: Characters, bytes and the table between them · Where a page declares its encoding
Client defaults cause most mojibake
Most mojibake in collected data comes from a client that decided the encoding before your code could. Requests
documents its rule plainly: it guesses from the headers and, when the Content-Type contains text but no charset,
follows the old RFC 2616 default of ISO-8859-1, so response.text ignores a page’s <meta charset="utf-8">. Its
quickstart’s advice is to read response.content, find the encoding, and set response.encoding. HTTPX assumes
UTF-8 when nothing is declared and replaces any bytes that are not valid UTF-8; it guesses only when you give its
client a guessing function as default_encoding.
| Client | A text page with no charset becomes | What to do |
|---|---|---|
| urllib | bytes from read(); get_content_charset() returns None |
decode it yourself, in the standard’s order |
| Requests | response.text decoded as ISO-8859-1 |
use response.content, or set response.encoding first |
| HTTPX | response.text decoded as UTF-8, with � for invalid bytes |
set response.encoding from a declaration you trust |
Node’s fetch |
response.text() always decodes UTF-8 |
TextDecoder(charset) on arrayBuffer() |
The last row is the Fetch standard’s own definition: text() consumes the body with UTF-8 decode, whatever the
charset says. Against the JavaScript version of the shop, the old windows-1252 page lost every non-ASCII character to
�, and a TextDecoder with the declared charset got them all back. The run also showed the Encoding Standard’s
label table at work: a decoder asked for iso-8859-1 is a windows-1252 decoder.
node/text_vs_decoder.mjs
// response.text() always decodes UTF-8, whatever the charset says; TextDecoder decodes what the page declares.
import { SITE, shop } from './nimbu_lab.mjs';
const lab = { dispatcher: shop, signal: AbortSignal.timeout(10_000) }; // a real request leaves dispatcher out
const title = (html) => html.match(/<title>(.*?)<\/title>/)[1];
const legacy = await fetch(`${SITE}/encoding/legacy`, lab);
const viaText = await legacy.clone().text();
const bytes = new Uint8Array(await legacy.arrayBuffer());
const charset = /charset=([\w-]+)/i.exec(legacy.headers.get('content-type'))?.[1] ?? 'utf-8';
console.log('Content-Type:', legacy.headers.get('content-type'));
console.log('response.text():', title(viaText));
console.log(`with TextDecoder('${charset}'):`, title(new TextDecoder(charset).decode(bytes)));
console.log("TextDecoder('iso-8859-1') is", new TextDecoder('iso-8859-1').encoding, 'in the Encoding Standard');
const modern = await fetch(`${SITE}/encoding/meta-only`, lab);
console.log('meta-only page, text():', title(await modern.text()));Output
Content-Type: text/html; charset=windows-1252
response.text(): Kashmiri �Mongra� saffron � 1 g � Rs. 310
with TextDecoder('windows-1252'): Kashmiri ‘Mongra’ saffron – 1 g · Rs. 310
TextDecoder('iso-8859-1') is windows-1252 in the Encoding Standard
meta-only page, text(): बासमती चावल 250 g · ₹25.00
Recorded with Node.js 24.21.0 on macOS 26 arm64. To run it yourself: mise exec node@24.21.0 -- node text_vs_decoder.mjs
nimbu_lab.mjs (the shop for fetch) · node/nimbu_lab.mjs
// Two of Nimbu Bazaar's product pages for Node's fetch: one in UTF-8, one in windows-1252, as the shop sends them.
//
// `shop` is a dispatcher, the object Node's fetch hands each request to (its `dispatcher` option, which Node's
// documentation describes). It answers with the bytes below instead of opening a socket. Nothing leaves the program.
export const SITE = 'https://bazaar.nimbu.test';
// windows-1252 bytes for the few characters of the old page that are not ASCII.
const CP1252 = { '‘': 0x91, '’': 0x92, '–': 0x96, '·': 0xb7 };
const cp1252 = (s) => Buffer.from([...s].map((c) => CP1252[c] ?? c.charCodeAt(0)));
const PAGES = {
'/encoding/meta-only': ['text/html', Buffer.from('<!doctype html>\n<html lang="hi"><head><meta charset="utf-8"><title>बासमती चावल 250 g · ₹25.00</title></head></html>\n')],
'/encoding/legacy': ['text/html; charset=windows-1252', cp1252('<!doctype html>\n<title>Kashmiri ‘Mongra’ saffron – 1 g · Rs. 310</title>\n')],
};
export const shop = {
dispatch(options, handler) {
const [kind, body] = PAGES[new URL(options.path, options.origin).pathname] ?? ['text/plain', Buffer.from('no such page\n')];
handler.onConnect(() => {});
queueMicrotask(() => {
handler.onHeaders(200, [Buffer.from('content-type'), Buffer.from(kind), Buffer.from('content-length'), Buffer.from(String(body.length))], () => {}, '');
handler.onData(body);
handler.onComplete([]);
});
return true;
},
close() {},
destroy() {},
};The rule for every client is the same: give the parser bytes, or decode them yourself from a declaration you trust, and use a guess only when nothing is declared. Parsing libraries that receive bytes can read the meta tag themselves; handing them a string that a client has already decoded wrongly removes that chance.
Unicode Character Inspector Paste a piece of mojibake such as ब and see each character's code point and name, and which bytes it came from.Write every file with its encoding named
The other half of the problem is writing. Python’s open() without an encoding uses the locale’s encoding, which on
many Windows systems is not UTF-8, so the same script writes different bytes on different machines. The defaults have
changed over the releases:
- 3.7
UTF-8 Mode, off by default opt in
PEP 540 adds a UTF-8 Mode, switched on with -X utf8 or PYTHONUTF8=1, in which open() uses UTF-8 instead of the locale's encoding. Without it, a file opened with no encoding is read and written in the locale's encoding.
- 3.10
A warning you can ask for find the gaps
PEP 597 adds EncodingWarning, shown with -X warn_default_encoding when open() is called without an encoding, and encoding="locale" for code that really means the locale.
- 3.14
The version this track pins the pin
Python 3.14.8 still uses the locale's encoding by default. On most Linux and macOS machines that is UTF-8; on Windows it often is not, so a script without encoding="utf-8" can write different bytes there.
- 3.15
UTF-8 by default new default
PEP 686 turns UTF-8 Mode on by default from Python 3.15. Code that names its encoding behaves the same on every version and every system, which is why the examples here always do.
The timeline has 4 events, from 3.7 to 3.15.
For CSV files, the csv module’s documentation adds a second rule: open the file with newline="", because the
module does its own newline handling, and without it quoted fields with line breaks are read wrongly and Windows gains
an extra \r on write. Spreadsheet apps add a third: many detect UTF-8 in a CSV file only by a byte-order mark, which
Python’s utf-8-sig codec writes before the first byte and skips on reading. The program writes the same three rows
both ways and reads each file back with each codec:
"""Write the same rows as UTF-8 and as UTF-8 with a BOM, and read each back two ways."""
import csv
from pathlib import Path
ROWS = [["sku", "name_hi", "price_inr"], ["NB-0001", "बासमती चावल 250 g", "25.00"], ["NB-0019", "जीरा 100 g", "48.00"]]
for encoding in ["utf-8", "utf-8-sig"]:
path = Path(f"products-{encoding}.csv")
with path.open("w", encoding=encoding, newline="") as f: # newline="": the csv module writes its own line ends
csv.writer(f).writerows(ROWS)
print(f"{path.name}: {path.stat().st_size} bytes, starting {path.read_bytes()[:6].hex(' ')}")
for reading in ["utf-8", "utf-8-sig"]:
with path.open(encoding=reading, newline="") as f:
first = next(csv.DictReader(f))
print(f" read as {reading:9} columns {list(first)} sku={first.get('sku')!r}")
path.unlink()Output
products-utf-8.csv: 110 bytes, starting 73 6b 75 2c 6e 61 read as utf-8 columns ['sku', 'name_hi', 'price_inr'] sku='NB-0001' read as utf-8-sig columns ['sku', 'name_hi', 'price_inr'] sku='NB-0001' products-utf-8-sig.csv: 113 bytes, starting ef bb bf 73 6b 75 read as utf-8 columns ['\ufeffsku', 'name_hi', 'price_inr'] sku=None read as utf-8-sig columns ['sku', 'name_hi', 'price_inr'] sku='NB-0001'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 write_csv.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The BOM costs 3 bytes, ef bb bf, and those 3 bytes are what a reader using plain utf-8 turns into \ufeff at
the start of the first column’s name. That is why first.get("sku") returned None there: the column was called
"\ufeffsku". Reading with utf-8-sig works for both files, so it is the safe choice for reading CSVs you did not
write.
Name the encoding everywhere
Write encoding="utf-8" (or "utf-8-sig" for a CSV meant for spreadsheet apps) on every open(), every
read_text() and every write_text(). Running with -X warn_default_encoding makes Python report each call that
leaves it out.
Checkpoint
Check: Client defaults that cause mojibake · Writing files with a named encoding
Normalise text before you compare it
Two strings can look identical and still differ. Unicode can write è as one character, U+00E8, or as e followed
by a combining accent, U+0300. The two are canonically equivalent, and Unicode normalization: Turning text into one standard form so that canonically equivalent strings compare equal. NFC composes characters such as e plus a combining accent into é wherever Unicode allows it.
turns either into one form: NFC composes characters where it can, NFD takes them apart. UAX #15 notes that the W3C recommends NFC for all
content, and warns against applying the compatibility forms, NFKC and NFKD, “blindly” to any text you happen to have.
Pages also hide characters you cannot see: a no-break space between a currency sign and its number, or a zero-width
space inside a word.
"""Two names that look the same and compare unequal, and the cleaning that makes them comparable."""
import unicodedata
composed = "Crème brûlée mix" # è and û as single characters
decomposed = unicodedata.normalize("NFD", composed) # e and u, each followed by a combining accent
print("look the same:", composed, "|", decomposed)
print("equal?", composed == decomposed, "| lengths", len(composed), len(decomposed))
print("equal after NFC?", unicodedata.normalize("NFC", composed) == unicodedata.normalize("NFC", decomposed))
print()
qa_one, qa_two = "\u0958", "\u0915\u093c" # DEVANAGARI LETTER QA, and KA followed by the sign NUKTA
for label, s in [("one code point", qa_one), ("two code points", qa_two)]:
nfc = unicodedata.normalize("NFC", s)
print(f"{label:16} {' '.join(f'U+{ord(c):04X}' for c in s):14} NFC: {' '.join(f'U+{ord(c):04X}' for c in nfc)}")
print()
price = "\u20b9\u00a01,250" # a price with a no-break space between the sign and the number, as many pages write it
print(repr(price), "->", [unicodedata.name(c) for c in price[:2]])
print("replace(' ', '') leaves", repr(price.replace(" ", "")))
def clean(text):
"""NFC, every kind of space made a plain space, zero-width spaces dropped; joiners kept, Indic text needs them."""
text = unicodedata.normalize("NFC", text)
text = "".join(" " if unicodedata.category(c) == "Zs" else c for c in text)
return text.replace("\u200b", "").replace("\ufeff", "")
print("cleaned:", repr(clean(price)), "| digits:", "".join(c for c in clean(price) if c.isdigit()))
print("joiner kept:", [unicodedata.name(c) for c in clean("\u0915\u094d\u200d\u0937")][-2:])Output
look the same: Crème brûlée mix | Crème brûlée mix
equal? False | lengths 16 19
equal after NFC? True
one code point U+0958 NFC: U+0915 U+093C
two code points U+0915 U+093C NFC: U+0915 U+093C
'₹\xa01,250' -> ['INDIAN RUPEE SIGN', 'NO-BREAK SPACE']
replace(' ', '') leaves '₹\xa01,250'
cleaned: '₹ 1,250' | digits: 1250
joiner kept: ['ZERO WIDTH JOINER', 'DEVANAGARI LETTER SSA']
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 normalise.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The two spellings of “Crème brûlée mix” had 16 and 19 characters and compared unequal until both were normalised to
NFC. NFC does not always compose: the Devanagari letter क़, U+0958, came out as two code points, U+0915 U+093C,
because Unicode excludes that letter from composition, so both spellings meet in the same two-character form. The
price held a no-break space, U+00A0, which replace(" ", "") does not touch; clean() turns every space character
into a plain one.
Keep the joiners in Indic text
The zero width joiner, U+200D, and non-joiner, U+200C, look like nothing, but the Unicode Standard’s chapter on
Devanagari describes how they change the way consonant clusters are drawn: half forms instead of conjuncts, or a
visible virama. Remove zero-width spaces, never the joiners, as clean() does.
Checkpoint
Check: Normalising before comparing
Practise: decode a page in the standard’s order
The exercise asks for the decoder of this lesson with the details real pages need: a quoted charset, a byte-order mark
for UTF-16, the http-equiv form of the meta tag, and the Encoding Standard’s labels, in which iso-8859-1 and
latin1 mean windows-1252. All its tests are sample tests:
Exercise · Medium · Python
Decode a page in the order the HTML standard uses
Write decode_body(content_type, body) in charset.py. It gets the response's Content-Type header (a string, or None when the response had none) and the body as bytes, and returns the body as text, choosing the encoding in this order:
- A byte-order mark at the start of the body:
EF BB BFis UTF-8,FE FFUTF-16 big-endian,FF FEUTF-16 little-endian (the listBOMSholds them). The mark is not part of the text. - The
charsetparameter ofContent-Type. Its name and value compare in any case, and the value may be in double quotes, as incharset="UTF-8". - A
<meta charset="...">, or a<meta http-equiv="Content-Type" content="...; charset=...">, in the first 1024 bytes of the body. - If nothing declared an encoding: UTF-8 when the bytes are valid UTF-8, otherwise windows-1252.
Map the labels as the Encoding Standard does: utf-8 and utf8 are UTF-8, and windows-1252, cp1252, iso-8859-1, latin1, ascii and us-ascii are all windows-1252 (Python's codec cp1252). A label you do not know counts as no declaration, so the next rule decides. The sample tests use the lesson's product pages in UTF-8, UTF-16 and windows-1252.
Starter code · charset.py
import codecs
import re
BOMS = [(codecs.BOM_UTF8, "utf-8"), (codecs.BOM_UTF16_BE, "utf-16-be"), (codecs.BOM_UTF16_LE, "utf-16-le")]
def decode_body(content_type, body):
"""The text of an HTML body: a BOM first, then the Content-Type charset, then a meta tag in the first 1024 bytes,
then UTF-8 if the bytes are valid UTF-8, else windows-1252."""
# Replace this: decoding everything as UTF-8 turns every legacy page into replacement characters.
return body.decode("utf-8", errors="replace")The sample tests · test_charset.py
import codecs
from charset import decode_body
HINDI = "बासमती चावल 250 g · ₹25.00"
OLD = "Kashmiri ‘Mongra’ saffron – 1 g"
def page(title, head=""):
return f"<!doctype html>\n<html><head>{head}<title>{title}</title></head></html>\n"
def test_the_header_charset():
"""a charset in Content-Type decides, in any case and quoted or not"""
assert decode_body("text/html; charset=utf-8", page(HINDI).encode("utf-8")) == page(HINDI)
assert decode_body('text/html; Charset="UTF-8"', page(HINDI).encode("utf-8")) == page(HINDI), "names and values compare in any case; quotes are not part of the value"
assert decode_body("text/html; charset=windows-1252", page(OLD).encode("cp1252")) == page(OLD)
def test_legacy_labels_mean_windows_1252():
"""iso-8859-1 and latin1 are labels for windows-1252 in the Encoding Standard"""
data = page(OLD).encode("cp1252")
assert decode_body("text/html; charset=iso-8859-1", data) == page(OLD), "0x96 is an en dash in windows-1252, a control character in true ISO-8859-1"
assert decode_body("text/html; charset=latin1", data) == page(OLD)
def test_a_bom_beats_the_header():
"""a byte-order mark decides even when the header says something else"""
assert decode_body("text/html; charset=windows-1252", codecs.BOM_UTF8 + page(HINDI).encode("utf-8")) == page(HINDI), "the UTF-8 BOM wins, and is not part of the text"
assert decode_body("text/html", codecs.BOM_UTF16_LE + page(HINDI).encode("utf-16-le")) == page(HINDI), "FF FE starts UTF-16 little-endian"
def test_a_meta_tag_when_the_header_is_silent():
"""<meta charset> or http-equiv Content-Type in the first 1024 bytes"""
html = page(HINDI, '<meta charset="utf-8">')
assert decode_body("text/html", html.encode("utf-8")) == html
old = page(OLD, '<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">')
assert decode_body("text/html", old.encode("cp1252")) == old, "the http-equiv form carries the charset in content"
def test_the_header_beats_the_meta_tag():
"""when the header and the page disagree, the header decides"""
old = page(OLD, '<meta charset="utf-8">')
assert decode_body("text/html; charset=windows-1252", old.encode("cp1252")) == old, "the meta tag is read only when the header is silent"
def test_a_meta_tag_too_far_down_is_not_read():
"""only the first 1024 bytes are prescanned"""
late = "<!--" + "x" * 1100 + "-->" + page(OLD, '<meta charset="utf-8">')
assert decode_body("text/html", late.encode("cp1252")) == late, "the meta tag is past byte 1024 and the bytes are not valid UTF-8: windows-1252"
def test_a_guess_when_nothing_is_declared():
"""valid UTF-8 is read as UTF-8; anything else as windows-1252"""
assert decode_body("text/html", page(HINDI).encode("utf-8")) == page(HINDI)
assert decode_body(None, page(OLD).encode("cp1252")) == page(OLD), "no header at all, and the bytes are not valid UTF-8"A hint
Write a small helper that turns a label into a codec name: strip spaces and quotes, lower-case it, then look it up in a dictionary of the labels above, returning None for anything else. For the header, re.search(r'charset\s*=\s*("[^"]*"|[^;\s]+)', content_type, re.I) finds the value; for the meta tag, search body[:1024] with a bytes pattern such as rb'<meta[^>]*?charset\s*=\s*["\']?\s*([\w-]+)'.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Then check what stayed with you, bytes, declarations, defaults, files and normalisation, with your weakest idea first:
Practice that adapts to you
Practise this lesson
10 questions about 5 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher): What is mojibake, and what causes it?
Mojibake is text that was encoded with one character encoding and decoded with another, so that each character turns
into one or more wrong ones, such as ब for the Devanagari letter ब or é for é. Nothing raises an error,
because tables such as windows-1252 give almost every byte some character. The usual cause in collected data is a
client default: a library decoding as ISO-8859-1 because the header had no charset, or a program reading UTF-8 bytes
with the computer’s locale encoding. The fix is to decode the original bytes with the right encoding, found from the
byte-order mark, the header or the meta tag; if only the mojibake is left and no byte was replaced, encoding it back
with the wrong table and decoding as UTF-8 repairs it.
From here, two interview questions go further: product names that show as mojibake in a database but fine in a browser, and a CSV export that some people’s spreadsheet app opens garbled. Open either on the question page of this module for its first step and its follow-ups:
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Fresher to mid
Product names in Hindi look right on the site but are stored in your database as text such as बा. Walk through how you find the cause, fix it, and repair the rows already stored.
-
Fresher to mid
Your collector's CSV export of Hindi product names looks right in your editor, but some colleagues see garbled text when they open it in their spreadsheet app. What is going on, and what do you change?
Key takeaways
- Text travels as bytes, and the encoding is the table that turns them back into characters; read with the wrong table, every multi-byte character becomes mojibake without any error.
- Decode a page in the HTML standard’s order, the byte-order mark, the header’s charset, a meta tag in the first 1024 bytes, then a guess, and keep the original bytes until you have.
- Client defaults cause most mojibake: Requests applies ISO-8859-1 to text without a charset and
fetch’stext()always decodes UTF-8, so take the bytes and decode them yourself. - Name the encoding on every file you open, use
newline=""for CSV, writeutf-8-sigfor spreadsheet apps and read CSVs you did not write withutf-8-sig. - Normalise text to NFC before comparing, turn no-break and other space characters into plain spaces, drop zero-width spaces, and keep the joiners Indic scripts need.
Terms in this lesson
- byte-order mark
- The character U+FEFF written at the very start of a text to identify its encoding: the bytes EF BB BF mean UTF-8, FE FF and FF FE the two byte orders of UTF-16.
- character encoding
- A table that turns characters into bytes and back, such as UTF-8 or windows-1252. Bytes carry no note of their encoding, so it must be declared or guessed.
- mojibake
- Text decoded with another encoding than the one it was written in, so that characters turn into wrong ones, such as é for é. No error is raised, which is why it spreads unnoticed.
- Unicode normalization
- Turning text into one standard form so that canonically equivalent strings compare equal. NFC composes characters such as e plus a combining accent into é wherever Unicode allows it.
References
- Encoding Standard (WHATWG)
- HTML Standard: determining the character encoding (WHATWG)
- RFC 9110: HTTP Semantics (section 8.3, Content-Type and charset) (IETF)
- Requests: Advanced usage (encodings) (Requests project)
- Requests: Quickstart (response content) (Requests project)
- HTTPX: Text encodings (the default encoding and auto-detection) (HTTPX project (Encode))
- Fetch Standard (section 5.3, the Body mixin) (WHATWG)
- codecs: utf-8-sig and Latin-1 (Python 3.14) (Python Software Foundation)
- csv: CSV file reading and writing (Python 3.14) (Python Software Foundation)
- Opening CSV UTF-8 files correctly in Excel (Microsoft)
- PEP 686: Make UTF-8 mode default (Python Software Foundation)
- PEP 540: Add a new UTF-8 Mode (Python Software Foundation)
- PEP 597: Add optional EncodingWarning (Python Software Foundation)
- unicodedata: Unicode Database (Python 3.14) (Python Software Foundation)
- UAX #15: Unicode Normalization Forms (Unicode Consortium)
- The Unicode Standard, chapter 12: South and Central Asia-I (Unicode Consortium)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress