UTF-8, BOMs and line endings in data files

A spreadsheet export that starts with three invisible bytes, EF BB BF, makes Python’s csv.DictReader name its first column '\ufeffname', and row["name"] fails with KeyError although the first line of the file plainly says name. Those bytes are a byte-order mark, one of four ways the text under a data file breaks a parser before the format’s own rules even start.

You will find which encodings each of the five formats allows and how it says which one a file uses, detect and remove a byte-order mark in Python and JavaScript, keep the line breaks inside quoted CSV fields intact, and refuse the characters a format forbids, with the offset of each.

Each format fixes its encoding, and only XML can name it inside the file

A file holds bytes, and the same word has different bytes in different encodings. café is 63 61 66 c3 a9 in UTF-8 and 63 61 66 e9 in Latin-1. Read the UTF-8 bytes as Latin-1 and you get café without any error; read the Latin-1 bytes as UTF-8 and the strict decoder stops at the fourth byte. This program does both, then gives the same bytes to XML and to TOML:

"""The same word as bytes in two encodings, read with the right and the wrong one, then given to XML and TOML."""
import tomllib
import xml.etree.ElementTree as ET

utf8 = "café".encode("utf-8")      # 63 61 66 c3 a9
latin1 = "café".encode("latin-1")  # 63 61 66 e9
print("UTF-8 bytes:  ", utf8.hex(" "), "| read as Latin-1:", utf8.decode("latin-1"))
try:
    latin1.decode("utf-8")
except UnicodeDecodeError as e:
    print("Latin-1 bytes:", latin1.hex(" "), "   | read as UTF-8:", e.reason, "at byte", e.start)

# XML can say inside the file which encoding it uses; without a declaration it must be UTF-8.
declared = b'<?xml version="1.0" encoding="ISO-8859-1"?><w>' + latin1 + b"</w>"
print("XML, declared ISO-8859-1:", ET.fromstring(declared).text)
try:
    ET.fromstring(b"<w>" + latin1 + b"</w>")
except ET.ParseError as e:
    print("XML, no declaration:     ", e)

# TOML must be UTF-8, and tomllib does not skip a byte-order mark.
try:
    tomllib.loads("\ufeffname = 'café'")
except tomllib.TOMLDecodeError as e:
    print("TOML with a BOM:", e)

UTF-8 bytes:   63 61 66 c3 a9 | read as Latin-1: café
Latin-1 bytes: 63 61 66 e9    | read as UTF-8: unexpected end of data at byte 3
XML, declared ISO-8859-1: café
XML, no declaration:      not well-formed (invalid token): line 1, column 6
TOML with a BOM: Invalid statement (at line 1, column 1)

The character encodingcharacter encoding: The rule that turns characters into bytes and back, such as UTF-8 or UTF-16. Bytes read with the wrong encoding become other characters without an error, or stop a strict decoder. is the contract between writer and reader, and each format settles it in its own way. The XML file named its encoding in its first line, so ElementTree read Latin-1 correctly; the same bytes without that line had to be UTF-8, and they were not. TOML allows only UTF-8, so tomllib has nothing to detect, and it refused a file that started with a byte-order mark. The five formats settle it like this:

CSV is the weak one. Nothing in the file says how to read it, so the reader has to be told, by the media type’s charset, by an agreement, or by a guess. When you write data files yourself, write UTF-8: it is the only encoding all five formats accept.

A byte-order mark is three bytes that JSON and CSV readers trip over

The byte-order markbyte-order mark: The character U+FEFF at the very start of a text: the bytes EF BB BF in UTF-8, FF FE or FE FF in UTF-16. It names the encoding, and many parsers refuse it unless it is removed first. is the character U+FEFF placed at the very start of a text. In UTF-16 it tells the reader the byte order, FF FE for little-endian and FE FF for big-endian; in UTF-8, where there is no byte order to tell, it is the three bytes EF BB BF. Python’s documentation calls a BOM in UTF-8 discouraged, and records that Microsoft’s Notepad wrote one to make UTF-8 files easier to recognise; some programs still write one at the start of their UTF-8 exports. This program writes a CSV file that way and reads it twice:

"""A CSV file that starts with a BOM, as some spreadsheet exports write it, read two ways."""
import csv

with open("branches.csv", "w", encoding="utf-8-sig", newline="") as f:
    csv.writer(f).writerows([["name", "city"], ["Deccan", "Pune"], ["Ikeja", "Lagos"]])

with open("branches.csv", "rb") as f:
    print("first bytes:", f.read(6).hex(" "))

for encoding in ["utf-8", "utf-8-sig"]:
    with open("branches.csv", encoding=encoding, newline="") as f:
        rows = list(csv.DictReader(f))
    print(f"{encoding:<10} header keys: {list(rows[0])}")
    try:
        print(f"{'':<10} first name: {rows[0]['name']}")
    except KeyError as e:
        print(f"{'':<10} KeyError: {e}")

first bytes: ef bb bf 6e 61 6d
utf-8      header keys: ['\ufeffname', 'city']
           KeyError: 'name'
utf-8-sig  header keys: ['name', 'city']
           first name: Deccan

Read as plain utf-8, the BOM became part of the first header, '\ufeffname', which looks like name when printed and fails like any other missing key. The utf-8-sig codec skips an optional UTF-8 BOM, so the second read is clean. JSON parsers react to the same three bytes in their own way, and the safe reader looks at the first bytes before decoding anything. Step through what it decides for four files:

The program behind that figure tests the three BOMs in turn, removes the one it finds and decodes the rest with the encoding it names, and UTF-8 when there is none:

"""Find the byte-order mark of a text, decode the text with the encoding it names, and parse the JSON inside."""
import codecs
import json

BOMS = [
    (codecs.BOM_UTF8, "utf-8"),        # EF BB BF
    (codecs.BOM_UTF16_LE, "utf-16-le"),  # FF FE
    (codecs.BOM_UTF16_BE, "utf-16-be"),  # FE FF
]


def sniff(data):
    """The encoding a BOM names and the bytes after it; UTF-8 when there is no BOM."""
    for bom, encoding in BOMS:
        if data.startswith(bom):
            return encoding, data[len(bom):], bom
    return "utf-8", data, b""


text = '{"city": "Pune"}'
files = {
    "UTF-8": text.encode("utf-8"),
    "UTF-8 with BOM": codecs.BOM_UTF8 + text.encode("utf-8"),
    "UTF-16LE with BOM": codecs.BOM_UTF16_LE + text.encode("utf-16-le"),
    "UTF-16BE with BOM": codecs.BOM_UTF16_BE + text.encode("utf-16-be"),
}
for name, data in files.items():
    encoding, body, bom = sniff(data)
    value = json.loads(body.decode(encoding, errors="strict"))
    print(f"{name:<18} first bytes {data[:4].hex(' ')}  BOM {bom.hex(' ') or 'none':<8} {encoding:<9} {value}")

print()
try:
    json.loads(files["UTF-8 with BOM"].decode("utf-8"))
except json.JSONDecodeError as e:
    print("decoded as plain utf-8:", e)
print("decoded as utf-8-sig:", json.loads(files["UTF-8 with BOM"].decode("utf-8-sig")))
print("the bytes themselves:", json.loads(files["UTF-8 with BOM"]))  # json.loads decodes bytes and drops a UTF-8 BOM

UTF-8              first bytes 7b 22 63 69  BOM none     utf-8     {'city': 'Pune'}
UTF-8 with BOM     first bytes ef bb bf 7b  BOM ef bb bf utf-8     {'city': 'Pune'}
UTF-16LE with BOM  first bytes ff fe 7b 00  BOM ff fe    utf-16-le {'city': 'Pune'}
UTF-16BE with BOM  first bytes fe ff 00 7b  BOM fe ff    utf-16-be {'city': 'Pune'}

decoded as plain utf-8: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
decoded as utf-8-sig: {'city': 'Pune'}
the bytes themselves: {'city': 'Pune'}

json.loads refused the BOM in text that had already been decoded, with a message that names the fix, decode using utf-8-sig. Given the bytes themselves, as in the last line, it sniffs them for you: it reads UTF-8, UTF-16 and UTF-32 and drops a UTF-8 BOM. In JavaScript the default TextDecoder removes a BOM for you, but Buffer.toString() and ignoreBOM: true keep it, and JSON.parse then fails on it. Decoding UTF-16 bytes as UTF-8 does not fail at once: each bad byte becomes U+FFFD, the replacement character:

// The same files in Node.js: TextDecoder removes a BOM by default; other ways of decoding keep it.
const json = new TextEncoder().encode('{"city": "Pune"}');
const withBom = new Uint8Array([0xef, 0xbb, 0xbf, ...json]);
const utf16le = new Uint8Array([0xff, 0xfe, ...[...'{"city": "Pune"}'].flatMap((c) => [c.charCodeAt(0), 0])]);

const hex = (bytes) => [...bytes.slice(0, 4)].map((b) => b.toString(16).padStart(2, '0')).join(' ');
// The first characters as code points, so an invisible BOM (U+FEFF) or a NUL (U+0000) shows.
const start = (text) => [...text.slice(0, 3)].map((c) => 'U+' + c.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')).join(' ');
const parsed = (text) => {
  try {
    return JSON.stringify(JSON.parse(text));
  } catch (e) {
    return e.name;
  }
};
const show = (label, text) => console.log(`${label.padEnd(32)} starts ${start(text)}  ${parsed(text)}`);

console.log(`UTF-8 with BOM, bytes ${hex(withBom)} ...`);
show('TextDecoder()', new TextDecoder().decode(withBom));
show('TextDecoder({ ignoreBOM: true })', new TextDecoder('utf-8', { ignoreBOM: true }).decode(withBom));
show("Buffer.toString('utf8')", Buffer.from(withBom).toString('utf8'));
console.log(`UTF-16LE with BOM, bytes ${hex(utf16le)} ...`);
show("TextDecoder('utf-16le')", new TextDecoder('utf-16le').decode(utf16le));
show("TextDecoder('utf-8')", new TextDecoder('utf-8').decode(utf16le));

UTF-8 with BOM, bytes ef bb bf 7b ...
TextDecoder()                    starts U+007B U+0022 U+0063  {"city":"Pune"}
TextDecoder({ ignoreBOM: true }) starts U+FEFF U+007B U+0022  SyntaxError
Buffer.toString('utf8')          starts U+FEFF U+007B U+0022  SyntaxError
UTF-16LE with BOM, bytes ff fe 7b 00 ...
TextDecoder('utf-16le')          starts U+007B U+0022 U+0063  {"city":"Pune"}
TextDecoder('utf-8')             starts U+FFFD U+FFFD U+007B  SyntaxError

CRLF ends CSV records, and quoted fields keep their own line breaks

A line endingline ending: The characters that end a line of text: LF (U+000A) on Linux and macOS, CR LF (U+000D U+000A) on Windows and at the end of every RFC 4180 CSV record. is LF (\n) on Linux and macOS and CR LF (\r\n) on Windows, and RFC 4180 ends every CSV record with CRLF. A field may contain a line break of its own, inside quotes: a postal address over two lines is the usual case. This program writes such a field with Python’s csv module and reads it back twice:

"""A CSV field with a line break inside it: written by the csv module, then read back two ways."""
import csv
import xml.etree.ElementTree as ET

with open("addresses.csv", "w", encoding="utf-8", newline="") as f:
    csv.writer(f).writerows([["branch", "address"], ["Deccan", "12 FC Road\r\nPune 411004"]])

with open("addresses.csv", "rb") as f:
    print("bytes on disk:", f.read())

with open("addresses.csv", encoding="utf-8", newline="") as f:
    print("newline='':   ", repr(list(csv.reader(f))[1][1]))
with open("addresses.csv", encoding="utf-8") as f:  # the default: universal newlines
    print("default:      ", repr(list(csv.reader(f))[1][1]))

# XML normalises every line break before the application sees it.
print("XML:          ", repr(ET.fromstring(b"<address>12 FC Road\r\nPune 411004</address>").text))

bytes on disk: b'branch,address\r\nDeccan,"12 FC Road\r\nPune 411004"\r\n'
newline='':    '12 FC Road\r\nPune 411004'
default:       '12 FC Road\nPune 411004'
XML:           '12 FC Road\nPune 411004'

The bytes on disk show both kinds of line break: \r\n after each record, and the \r\n inside the quoted address, because csv.writer uses '\r\n' as its line terminator by default. Opened with newline='', as the csv module’s documentation asks, the field came back exactly. Opened with the default, Python’s universal newlines turned the \r\n inside the field into \n before the csv module ever saw it, so the address changed without an error, and a checksum of the row would no longer match.

The other formats each settle line breaks in their own rules:

Some characters are not allowed at all: lone surrogates and control codes

Decoding can succeed and still leave characters the format refuses. Text pasted from other programs can bring control characters with it, such as a vertical tab, U+000B, or U+0001, and XML 1.0 allows neither. Its production [2] Char admits tab, line feed and carriage return, then every code point from U+0020 to U+D7FF, from U+E000 to U+FFFD and from U+10000 up, which leaves out the other C0 controls, the surrogates, U+FFFE and U+FFFF. This program finds them with their offsets before any XML is written:

"""Find the characters that XML 1.0 does not allow, with their offsets, before writing text into XML."""
import xml.etree.ElementTree as ET


def xml_char(cp):
    """XML 1.0, production [2] Char: tab, LF, CR and the ranges below, which leave out surrogates, FFFE and FFFF."""
    return cp in (0x9, 0xA, 0xD) or 0x20 <= cp <= 0xD7FF or 0xE000 <= cp <= 0xFFFD or 0x10000 <= cp <= 0x10FFFF


def forbidden(text):
    """(offset, code point) of every character XML 1.0 forbids."""
    return [(i, f"U+{ord(c):04X}") for i, c in enumerate(text) if not xml_char(ord(c))]


note = "Gift\x0bwrap\x01 please\ufffe"  # a vertical tab, a stray control and a noncharacter, as pasted text brings
print(forbidden(note))
print(forbidden("Tab\tand\r\nnew lines are fine"))

try:
    ET.fromstring(f"<note>{note}</note>")
except ET.ParseError as e:
    print("ElementTree:", e)

[(4, 'U+000B'), (9, 'U+0001'), (17, 'U+FFFE')]
[]
ElementTree: not well-formed (invalid token): line 1, column 10

ElementTree stopped at column 10, the vertical tab, because a file holding it is not well-formed: an XML parser must refuse the whole document, not skip the character. The scan names all three offenders, 4, 9 and 17, so they can be removed or replaced before writing.

JSON has a different gap. Its \u escapes can name half of a surrogate pair, such as \ud800, and RFC 8259 calls the behaviour of software that receives one unpredictable. A lone surrogatelone surrogate: A code point from U+D800 to U+DFFF that is not part of a pair. UTF-16 uses two of them together for one character above U+FFFF; on its own, one is not a character and has no UTF-8 form. is not a character, so it has no UTF-8 form:

"""A JSON escape for half of a surrogate pair: legal JSON text, but not a character."""
import json

pair = json.loads(r'"\ud83d\udcda"')  # a surrogate pair: one character, U+1F4DA (books)
lone = json.loads(r'"\ud800"')  # the first half alone
print("pair:", len(pair), f"U+{ord(pair):X}", pair.encode("utf-8"))
print("lone:", len(lone), f"U+{ord(lone):04X}")
print("dumps, ensure_ascii=True: ", json.dumps(lone))
try:
    lone.encode("utf-8")
except UnicodeEncodeError as e:
    print("encode('utf-8'):", e.reason)
try:
    with open("names.json", "w", encoding="utf-8") as f:
        f.write(json.dumps({"name": lone}, ensure_ascii=False))
except UnicodeEncodeError as e:
    print("writing the file:", e.reason)

pair: 1 U+1F4DA b'\xf0\x9f\x93\x9a'
lone: 1 U+D800
dumps, ensure_ascii=True:  "\ud800"
encode('utf-8'): surrogates not allowed
writing the file: surrogates not allowed

Python kept the half in a str and failed only when it had to write UTF-8, possibly far from where the data came in. JavaScript strings are UTF-16 code units, so a lone half is allowed in a string; JSON.stringify escapes it again, as ES2019’s well-formed JSON.stringify requires, but TextEncoder silently writes EF BF BD, the replacement character, so the data changes without a word:

// The same escape in JavaScript, whose strings are UTF-16 code units: a lone half is allowed in a string.
const lone = JSON.parse('"\\ud800"');
const hex = (s) => [...s].map((c) => 'U+' + c.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')).join(' ');
console.log('lone:', lone.length, hex(lone), 'well formed:', lone.isWellFormed());
console.log('JSON.stringify:', JSON.stringify(lone));
console.log('TextEncoder bytes:', [...new TextEncoder().encode(lone)].map((b) => b.toString(16)).join(' '));
console.log('toWellFormed():', hex(lone.toWellFormed()));

lone: 1 U+D800 well formed: false
JSON.stringify: "\ud800"
TextEncoder bytes: ef bf bd
toWellFormed(): U+FFFD

RFC 9839, a Proposed Standard, gives these sets names that a format or an API can require: Unicode Scalars (no surrogates), XML Characters (the Char production above) and Unicode Assignables, which also leaves out every control except tab, LF and CR and every noncharacter. Pick one for your format, check it where text enters, and refuse what falls outside it with the offset, as the scan does.

Practise: decode a data file safely

The exercise puts the lesson together in one function: it looks for a BOM, decodes strictly with the encoding the BOM names, and refuses every character XML 1.0 forbids, with its code point and offset, as the scan printed them:

import codecs


def normalise_text(data):
    """The text of `data` for a parser: BOM removed, decoded strictly, XML-forbidden characters refused."""
    return data.decode("utf-8")

import codecs

from normalise import normalise_text

TEXT = '{"city": "Pune"}'


def raises_value_error(data):
    """The message of the ValueError normalise_text(data) raises, or None."""
    try:
        normalise_text(data)
    except ValueError as e:
        return str(e)
    return None


def test_plain_utf8():
    """decodes UTF-8 without a BOM unchanged"""
    assert normalise_text(TEXT.encode("utf-8")) == TEXT
    assert normalise_text("caf\u00e9 \u20b9".encode("utf-8")) == "caf\u00e9 \u20b9"


def test_utf8_bom_removed():
    """removes a UTF-8 BOM"""
    assert normalise_text(codecs.BOM_UTF8 + TEXT.encode("utf-8")) == TEXT


def test_utf16_boms():
    """decodes UTF-16LE and UTF-16BE named by their BOM"""
    assert normalise_text(codecs.BOM_UTF16_LE + TEXT.encode("utf-16-le")) == TEXT
    assert normalise_text(codecs.BOM_UTF16_BE + TEXT.encode("utf-16-be")) == TEXT


def test_empty():
    """gives an empty string for no bytes, or for a BOM alone"""
    assert normalise_text(b"") == ""
    assert normalise_text(codecs.BOM_UTF8) == ""


def test_invalid_utf8():
    """refuses bytes that are not UTF-8"""
    assert raises_value_error(b"caf\xc3(") is not None


def test_forbidden_controls():
    """refuses a C0 control with its code point and offset"""
    assert raises_value_error("Gift\x0bwrap".encode("utf-8")) == "forbidden character U+000B at offset 4"
    assert raises_value_error(b"ok\x01") == "forbidden character U+0001 at offset 2"


def test_noncharacter():
    """refuses U+FFFE, counting the offset after the BOM is removed"""
    data = codecs.BOM_UTF8 + "ab\ufffe".encode("utf-8")
    assert raises_value_error(data) == "forbidden character U+FFFE at offset 2"


def test_allowed_whitespace_and_inner_bom():
    """keeps tab, CR, LF and a U+FEFF that is not at the start"""
    text = "a\tb\r\nc\ufeffd"
    assert normalise_text(text.encode("utf-8")) == text


def test_characters_above_ffff():
    """keeps a character above U+FFFF, in UTF-8 and in UTF-16 with a BOM"""
    text = "books \U0001f4da"
    assert normalise_text(text.encode("utf-8")) == text
    assert normalise_text(codecs.BOM_UTF16_LE + text.encode("utf-16-le")) == text

Then try the questions on encodings, byte-order marks, line endings and forbidden characters. A wrong answer brings back that idea before the others, with the reason spelt out:

Interview questions

Warm-up (fresher to 2–5 years): what is a byte-order mark, and why does JSON care?

A byte-order mark is the character U+FEFF at the very start of a text. In UTF-16 it says which byte comes first, FF FE for little-endian and FE FF for big-endian; in UTF-8 it is the three bytes EF BB BF, which can only mark the text as UTF-8, because UTF-8 has no byte order. RFC 8259 says JSON sent between systems must be UTF-8 and that implementations must not add a BOM, though parsers may ignore one. Given text that starts with U+FEFF, Python’s json.loads refuses it with “Unexpected UTF-8 BOM (decode using utf-8-sig)”, and JSON.parse refuses it too. The fix is to remove it while decoding, with the utf-8-sig codec in Python or the default TextDecoder in JavaScript, and never to write one.

Two harder questions follow, about an import that broke when the exporter changed and about addresses that changed in a round trip. Decide what you would check first, then open each first step:

Key takeaways

• The same characters have different bytes in different encodings; JSON between systems and TOML must be UTF-8, XML can name its encoding in its declaration, YAML detects it from the first bytes, and CSV says nothing inside the file.
• A byte-order mark is U+FEFF at the start of a text, EF BB BF in UTF-8; remove it before parsing with utf-8-sig or TextDecoder, and never write one into JSON.
• CSV ends records with CRLF and keeps line breaks inside quoted fields, so open CSV files with newline=''; XML and YAML normalise line breaks to LF themselves.
• XML 1.0 forbids most control characters, the surrogates, U+FFFE and U+FFFF; a JSON escape can name a lone surrogate, which has no UTF-8 form, so check characters where data enters and report the offset.