Python Module 8 – Files, paths and data formats
XML and HTML parsing without security holes
Read and build XML with ElementTree, refuse the DOCTYPE features behind XML attacks, escape user text with html.escape and parse simple HTML in Python.
What you will learn
- Parse and build XML with xml.etree.ElementTree
- Explain the XML attacks the Python documentation lists, such as entity expansion, and their defences
- Write a parser for untrusted XML that refuses any DOCTYPE and any document over a size limit
- Use html.escape for HTML output, once, at the moment a page is built
- Extract links from simple HTML and keep only the schemes on an allowlist
Before you start
What you’ll learn · Before you start
What you will learn
- Parse and build XML with xml.etree.ElementTree
- Explain the XML attacks the Python documentation lists, such as entity expansion, and their defences
- Write a parser for untrusted XML that refuses any DOCTYPE and any document over a size limit
- Use html.escape for HTML output, once, at the moment a page is built
- Extract links from simple HTML and keep only the schemes on an allowlist
Before you start
On this page
A small XML document can declare entities that expand into several gigabytes of text, as the Python documentation
warns, and a one-line comment from a reader can add a heading, or worse, to every page that shows it. Python reads
XML with xml.etree.ElementTree and helps with HTML through html.escape() and html.parser.HTMLParser; the
everyday work is easy, and the two habits that keep it safe are short: refuse what you do not need when reading XML,
and escape what you write into HTML.
You will read and build XML with ElementTree, see where the XML tricks live and write a parser that refuses them, escape readers’ text for HTML at the right moment, and collect the links of an untidy page, keeping only safe ones.
ElementTree turns a document into a tree you can search
An XML document is a tree. Every element (XML): One node of an XML tree: a tag such as item, optional attributes such as version="2.0", optional text and child elements. ElementTree gives each element its tag, attrib, text and children. has a tag (item), optional
attributes (version="2.0"), optional text and child elements. The example reads an RSS-style feed of
notices and then builds a new document from it:
import xml.etree.ElementTree as ET
# An RSS-style feed, as a server might send it.
feed = """<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>Campus notices</title>
<item>
<title>Library opens late on Friday</title>
<link>https://example.org/notices/library</link>
<category>library</category>
</item>
<item>
<title>Hostel fees & deadlines</title>
<link>https://example.org/notices/fees</link>
<category>hostel</category>
<category>fees</category>
</item>
<item>
<title>Sports day volunteers</title>
</item>
</channel>
</rss>"""
root = ET.fromstring(feed) # the <rss> element
print(root.tag, root.get("version")) # attributes come from get()
channel = root.find("channel") # the first matching child
print("feed:", channel.findtext("title"))
for item in channel.findall("item"): # every matching child, in order
title = item.findtext("title")
link = item.findtext("link", default="(no link)")
tags = [c.text for c in item.iter("category")]
print(f"- {title} | {link} | {tags}")
# Building XML: elements, attributes and text. ElementTree escapes & and < itself.
out = ET.Element("notices", count=str(len(channel.findall("item"))))
for item in channel.findall("item"):
ET.SubElement(out, "notice").text = item.findtext("title")
ET.indent(out)
print(ET.tostring(out, encoding="unicode"))Output
rss 2.0 feed: Campus notices - Library opens late on Friday | https://example.org/notices/library | ['library'] - Hostel fees & deadlines | https://example.org/notices/fees | ['hostel', 'fees'] - Sports day volunteers | (no link) | [] <notices count="3"> <notice>Library opens late on Friday</notice> <notice>Hostel fees & deadlines</notice> <notice>Sports day volunteers</notice> </notices>
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 feed_titles.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The feed is the tree in the diagram: rss at the root, its channel, and under the channel its own title and
three items:
The tree ElementTree builds from the feed
Text description of the diagram
The diagram is a column, from the root of the tree down.
- The root element, rss, with the attribute version set to 2.0.
- Its only child, channel.
- The children of channel, in the order of the document: first the channel's own title element, whose text is Campus notices, then three item elements. Item 1 has the title Library opens late on Friday, one link and one category, library. Item 2 has the title Hostel fees & deadlines, one link and two categories, hostel and fees. Item 3 has only a title, Sports day volunteers: no link and no category.
Each call of the example looks at a different part of that tree:
| Call in the example | What it returns |
|---|---|
ET.fromstring(feed) |
the root element, rss |
root.get("version") |
the attribute’s value, '2.0' |
root.find("channel") |
the first child named channel |
channel.findtext("title") |
'Campus notices', the channel’s own title |
channel.findall("item") |
the three items, in document order |
item.findtext("link", default="(no link)") |
'(no link)' for the third item |
item.iter("category") |
the second item’s two category elements, hostel and fees |
find() and findall() look one level down unless the path says more, such as "channel/item" or ".//category"
for any depth, while iter() walks every matching element below, however deep. For a file, ET.parse("feed.xml")
returns an ElementTree whose getroot() is the same root element. Entities such as & arrive decoded: the
second title reads Hostel fees & deadlines.
For files too big to hold as one tree, ET.iterparse(path, events=("end",)) hands over each element as soon as it
is complete. The tree still grows as it is read, so call elem.clear() once you have used an element, and for a
long run of records also remove each finished one from its parent, as the documentation’s pull-parsing section shows.
Namespaced documents, such as Atom feeds, write tags as {http://www.w3.org/2005/Atom}entry; pass a namespaces
dictionary to find() and findall() to use short prefixes instead.
Checkpoint
Check: Reading an XML tree
Element and SubElement build XML that escapes itself
The last part of the first example builds a document. ET.Element(tag, attributes) makes the root,
ET.SubElement(parent, tag) adds a child and returns it, and assigning .text sets its text. ET.indent(), added in
Python 3.9, adds line breaks and spaces so the result is readable, and ET.tostring(elem, encoding="unicode") returns
a string. To save a file, wrap the root in ET.ElementTree(root) and call
write(path, encoding="utf-8", xml_declaration=True).
Notice the & in its output: ElementTree escapes &, < and > in text and attribute values itself. The next
example builds the same kind of document twice, once by joining strings and once with SubElement, and checks each
result by parsing it back:
import xml.etree.ElementTree as ET
titles = ["Library opens late on Friday", "Hostel fees & deadlines", "Exam in room <B12>"]
# Joining strings: the titles go into the document exactly as they are.
joined = "<notices>" + "".join(f"<notice>{t}</notice>" for t in titles) + "</notices>"
# Building elements: ElementTree escapes &, < and > in text as it writes.
root = ET.Element("notices")
for t in titles:
ET.SubElement(root, "notice").text = t
built = ET.tostring(root, encoding="unicode")
# The check: parse each document back and compare the titles with the ones that went in.
for label, text in [("joined strings", joined), ("SubElement", built)]:
try:
back = [n.text for n in ET.fromstring(text)]
print(f"{label}: parses, titles come back unchanged: {back == titles}")
except ET.ParseError as err:
print(f"{label}: not XML, ParseError: {err}")
print(built)Output
joined strings: not XML, ParseError: not well-formed (invalid token): line 1, column 75 SubElement: parses, titles come back unchanged: True <notices><notice>Library opens late on Friday</notice><notice>Hostel fees & deadlines</notice><notice>Exam in room <B12></notice></notices>
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 build_vs_join.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The two titles that cause the trouble, and what each way of building did with them:
| Title | Joined into the string | Written by SubElement |
|---|---|---|
Hostel fees & deadlines |
a bare &, which XML does not allow |
Hostel fees & deadlines |
Exam in room <B12> |
a <B12> tag nobody meant |
Exam in room <B12> |
The joined document is not XML at all: the parser stops at the second title’s &. The built one parses back to
exactly the titles that went in. Build XML this way rather than by joining strings, for the same reason that SQL is
built with placeholders: the library knows the escaping rules, and string formatting does not.
Checkpoint
Check: Building XML that stays XML
The entity tricks of untrusted XML live in the DOCTYPE
XML is older and richer than the data it usually carries. A document may start with a
DOCTYPE: A document type declaration near the start of an XML document, <!DOCTYPE …>, which may declare entities. Feeds, invoices and API responses never need one, so code that parses untrusted XML can refuse it., a document type declaration, and inside it declare
entity (XML): A named piece of text that a reference such as &name; stands for. &, <, >, " and ' are built in; any other entity is declared in a DOCTYPE, which the expansion tricks depend on.: named pieces of text that &name; later expands. That feature is the root of
the first two attacks of the Python documentation’s XML security section; the other two need no DOCTYPE:
- Exponential entity expansion, also called “billion laughs”: entities that refer to other entities several times each, so a tiny file expands into gigabytes of text.
- Quadratic blowup: one large entity repeated thousands of times, which avoids limits on nesting.
- Large tokens: a token so long that the parser re-reads it over and over, which costs quadratic time.
- Decompression bombs: a small compressed stream, such as a gzipped upload, that unpacks into an enormous one.
The documentation names
xmlrpcas vulnerable to it.
Python’s parsers are built on the Expat library. The documentation states that Expat does not open local files or
network connections by default, and that Expat versions below 2.7.2 may be vulnerable to the first three problems.
Python bundles a copy of Expat, but it can also be built against the operating system’s copy, so the Python version
alone does not tell you which Expat runs: check pyexpat.EXPAT_VERSION on the machines that run your code.
Two checks on the way in, escaping on the way out
Text description of the diagram
The flowchart follows text from outside the program, from top to bottom.
- XML arrives from outside: an upload, a feed or an API response.
- The first check asks whether the document is too large. If it is, the document is refused with a clear error.
- The second check asks whether the document declares a DOCTYPE. If it does, it is refused as well, because a DOCTYPE is where entity declarations live, and untrusted data has no need for them.
- Only then does ElementTree build the tree, and the program reads plain values from it.
- Every value that goes into an HTML page passes through html.escape() first, and only then becomes part of the page.
The defence that removes the entity tricks altogether is simple, because feeds, invoices and API responses never need a DOCTYPE: refuse any document that has one. The OWASP cheat sheet on XML entity attacks also advises switching DTDs off wherever possible, and points out that this alone does not stop every denial of service, so a size limit belongs beside it. The diagram puts both checks on the way in, and the escaping of the later sections on the way out.
Checkpoint
Check: Where XML tricks live
A parser for untrusted XML refuses any DOCTYPE
ElementTree builds its tree through a target, normally a TreeBuilder. A target may define a doctype()
method, and the parser calls it when it reaches a DOCTYPE. A subclass whose doctype() raises therefore stops
parsing right there, before the declarations inside are used. The example runs four samples through such a parser,
then gives the sample with the DOCTYPE to ET.fromstring() without the guard:
import pyexpat
import xml.etree.ElementTree as ET
print("Expat:", pyexpat.EXPAT_VERSION)
class NoDoctype(ET.TreeBuilder):
"""Builds the tree as usual, but refuses any document that has a DOCTYPE."""
def doctype(self, name, pubid, system):
raise ValueError(f"a DOCTYPE is not accepted here (found {name!r})")
def parse_untrusted(text, max_chars=10_000):
"""Parse XML from outside the program: a size limit first, then no DOCTYPE at all."""
if len(text) > max_chars:
raise ValueError(f"document too large: {len(text):,} characters")
return ET.fromstring(text, parser=ET.XMLParser(target=NoDoctype()))
samples = {
"plain": "<order id='17'><item>rice</item><item>tea</item></order>",
"with DOCTYPE": '<!DOCTYPE order [<!ENTITY shop "Gupta Stores">]><order>&shop;</order>',
"too large": "<log>" + "x" * 20_000 + "</log>",
"broken": "<order><item>rice</order>",
}
for label, text in samples.items():
try:
root = parse_untrusted(text)
print(f"{label}: accepted, {len(root)} child elements")
except ValueError as err:
print(f"{label}: refused, {err}")
except ET.ParseError as err:
print(f"{label}: not well-formed, {err}")
# The same document without the guard: fromstring() accepts the DOCTYPE and expands the entity it declares.
unguarded = ET.fromstring(samples["with DOCTYPE"])
print(f"without the guard: accepted, &shop; became {unguarded.text!r}")Output
Expat: expat_2.8.5 plain: accepted, 2 child elements with DOCTYPE: refused, a DOCTYPE is not accepted here (found 'order') too large: refused, document too large: 20,011 characters broken: not well-formed, mismatched tag: line 1, column 19 without the guard: accepted, &shop; became 'Gupta Stores'
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 refuse_doctype.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Version note
The first line differs in the browser, which runs the Expat that Pyodide bundles, 2.7.3, instead of the 2.8.5 linked into this lesson’s CPython 3.14.8. Both are above the 2.7.2 that the documentation names; the rest of the output is the same.
In your browser, Pyodide 314.0.7 (CPython 3.14.2) prints:
Expat: expat_2.7.3 plain: accepted, 2 child elements with DOCTYPE: refused, a DOCTYPE is not accepted here (found 'order') too large: refused, document too large: 20,011 characters broken: not well-formed, mismatched tag: line 1, column 19 without the guard: accepted, &shop; became 'Gupta Stores'
Step through the four samples, and the one without the guard, check by check:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- too large. A log element with 20,011 characters is refused before the parser reads any of it, so an oversized document costs almost nothing.
- with DOCTYPE. This document declares an entity in a DOCTYPE. The parser calls doctype() as soon as it reaches the DOCTYPE, and NoDoctype's doctype() raises, so the declaration is never used.
- broken. <order><item>rice</order> closes order while item is still open. The parser gives up with ParseError, mismatched tag, at line 1, column 19.
- plain. An order with two items, no DOCTYPE and well under the limit passes every check, and the root of its tree has 2 child elements.
- Without the guard. Parsed by ET.fromstring() alone, the document with the DOCTYPE is accepted and &shop; becomes 'Gupta Stores'. Harmless here, it is the feature that every entity trick of the documentation depends on.
The plain order is accepted. The document with a DOCTYPE is refused even though its entity is harmless, which is the
point: the rule needs no judgement about what a DOCTYPE contains. Without the guard the same document was accepted
and &shop; expanded to 'Gupta Stores', the feature every entity trick depends on. The size check runs before any
parsing, so an oversized document costs nothing, and malformed XML still raises ET.ParseError, with the line and
column where the parser gave up. Callers can catch ValueError and ET.ParseError and answer with a clear error
message.
For a ready-made alternative, the third-party package defusedxml offers drop-in versions of the standard parsers.
By default it refuses entity declarations and external references, and with forbid_dtd=True it refuses any
DOCTYPE, as the example does. The OWASP cheat sheet recommends it for untrusted XML in Python. Whichever you use, keep
the limits that no parser setting replaces: a maximum size, a time limit for the request, and iterparse() with
clear() for large documents.
Key idea
Refuse what untrusted XML never needs: check the size before parsing, refuse any DOCTYPE while parsing, and read only plain values from the tree.
Checkpoint
Check: A parser for untrusted XML
html.escape() keeps a reader’s text from becoming markup
The problem in HTML runs the other way: your program writes text into a page, and if that text contains markup, the
browser reads it as markup. html.escape(s) replaces &, < and > with the entities &, < and >,
and, with quote=True, the default, both quote characters too:
from html import escape, unescape
name = 'Priya "PJ" D\'Souza'
comment = "Use <h1> for the title & <p> for each paragraph."
# Pasted in as it is, the comment's tags become real tags in the page.
print(f"<p>{comment}</p>")
# Escaped, the browser shows the same characters as text.
print(f"<p>{escape(comment)}</p>")
# Inside a quoted attribute, the quotes in the name must be escaped too (quote=True is the default).
print(f'<span title="{escape(name)}">{escape(name, quote=False)}</span>')
print(unescape(escape(comment)) == comment)Output
<p>Use <h1> for the title & <p> for each paragraph.</p> <p>Use <h1> for the title & <p> for each paragraph.</p> <span title="Priya "PJ" D'Souza">Priya "PJ" D'Souza</span> True
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 escape_text.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
These are the five characters it replaces, and where each one matters:
| Character | escape() writes |
Where it matters |
|---|---|---|
& |
& |
text and attribute values |
< |
< |
text and attribute values |
> |
> |
text and attribute values |
" |
" |
quoted attribute values, with quote=True |
' |
' |
quoted attribute values, with quote=True |
The first line of the output shows the problem: pasted in as it is, the comment’s <h1> and <p> would become real
elements of the page and change its layout, the first step of the weakness called
cross-site scripting: A weakness in which text from one user reaches a page without escaping, so the browser reads it as markup and can run code in other visitors' browsers. Escaping output for its place in the page prevents it.. Escaped, the browser shows the exact characters the
reader typed. The third line shows the attribute case: a title attribute is wrapped in double quotes, so a double
quote inside the name would end the attribute early, and escaping it to " keeps the value whole. Between tags,
quote=False is enough, as the visible name shows. html.unescape() turns entities back into characters.
Where escaping happens matters as much as whether it happens. Follow one comment from the reader who writes it to the reader who sees it:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- The comment. Priya posts "Use <h1> for the title & <p> for each paragraph." The comment explains HTML, so it contains tags.
- Stored as typed. The server stores the comment exactly as it was typed. Escaped now, the database would hold < instead of <, and escaping again when the page is built would show readers < on the page.
- A page is requested. Another reader opens the page of comments, and the server builds it for them.
- Escaped on the way out. The server puts escape(comment) into the page, so <h1> becomes <h1> and & becomes &.
- Shown as text. The browser turns the entities back into characters and shows exactly what Priya typed, with no new heading on the page.
- Without escape(). Pasted in raw, the comment's <h1> and <p> would become real elements of the page and change its layout. A comment written to add elements that run code would reach every reader the same way.
Three rules keep escaping reliable. Escape at the moment you write into the page, not when you store the text, so
the database keeps what the user typed and nothing is escaped twice. Quote every attribute value, because an
unquoted value ends at the first space. And remember what escaping cannot do: it keeps text from becoming markup, but
it does not make a link safe or code inside a <script> element safe. The OWASP cheat sheet on cross-site scripting
asks for an encoder that matches each context, and for links, an allowlist: A list of what is accepted, such as the URL schemes http and https; anything not on it is refused. Unlike a denylist, it also refuses values nobody thought of. of http and
https addresses. Most template engines can escape values automatically, a setting called
autoescaping: A template-engine setting that escapes every value put into an HTML template unless the template marks it as safe, so a forgotten call to an escape function cannot let markup through.; check that it is switched on.
Escaping twice
escape(escape("Fees & forms")) gives Fees &amp; forms, and the reader sees Fees & forms on the page.
Store the text as typed and escape it once, when the page is built.
Checkpoint
Check: Escaping text for HTML
HTMLParser calls your methods, tag by tag
html.parser.HTMLParser reads HTML and calls methods you define: handle_starttag(tag, attrs),
handle_endtag(tag) and handle_data(data). Tag names arrive in lower case and attributes as a list of
(name, value) pairs, and with convert_charrefs=True, the default, entities such as & arrive as plain
characters. The example collects a page’s title and its links:
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit
class LinkCollector(HTMLParser):
"""Collects the page title and every link's address and text."""
def __init__(self):
super().__init__() # convert_charrefs=True: & and friends arrive as plain characters
self.title = ""
self.links = []
self._in_title = False
self._href = None
def handle_starttag(self, tag, attrs):
if tag == "title":
self._in_title = True
elif tag == "a":
self._href = dict(attrs).get("href")
self.links.append([self._href, ""])
def handle_endtag(self, tag):
if tag == "title":
self._in_title = False
elif tag == "a":
self._href = None
def handle_data(self, data):
if self._in_title:
self.title += data
elif self._href is not None:
self.links[-1][1] += data.strip()
# Real pages are untidy: upper-case tags, a missing </p>, an entity inside an attribute.
page = """<HTML><head><title>Timetable & rooms</title></head>
<body><p>Lectures move to <A HREF="/rooms?block=B&floor=2">Block B</A>
<p>Questions? <a href="mailto:office@example.org">Write to the office</a>
or see <a href="https://example.org/faq">the FAQ</a>, <a href="ftp://files.example.org/old">the old files</a>.
</body></HTML>"""
parser = LinkCollector()
parser.feed(page)
parser.close()
print("title:", parser.title)
base = "https://college.example.org/notices/"
for href, text in parser.links:
url = urljoin(base, href) # a relative link becomes a full address
keep = urlsplit(url).scheme in {"http", "https"} # an allowlist of schemes, never a denylist
print(f"{'keep' if keep else 'skip'} {text:<20} {url}")Output
title: Timetable & rooms keep Block B https://college.example.org/rooms?block=B&floor=2 skip Write to the office mailto:office@example.org keep the FAQ https://example.org/faq skip the old files ftp://files.example.org/old
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 link_parser.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Step through the first calls the parser makes on that untidy page:
The drawing shows the whole scene; its 6 steps are listed below.
Every step as text
- A start tag. The parser meets <title> and calls handle_starttag('title', []). LinkCollector notes that it is now inside the title.
- Text, decoded. The text Timetable & rooms arrives through handle_data() as "Timetable & rooms": with convert_charrefs=True, the default, entities come as plain characters.
- Upper case becomes lower. <A HREF="/rooms?block=B&floor=2"> arrives as handle_starttag('a', [('href', '/rooms?block=B&floor=2')]): the tag and the attribute name in lower case, the entity in the value decoded.
- The link's text. handle_data('Block B') comes while LinkCollector holds a link's address, so it becomes that link's text.
- The end tag. </A> arrives as handle_endtag('a'), and LinkCollector stops collecting text for that link.
- A missing </p>. The first <p> is never closed. The parser does not mind; it reports the next <p> as another start tag and carries on, because it reports what it meets instead of checking the document.
The page has upper-case tags, a missing </p> and an entity inside an attribute, and the parser copes with all
three, because it reports what it sees instead of insisting on a correct document. HTMLParser suits small jobs on
pages you know. For scraping many untidy real-world pages, the third-party package BeautifulSoup builds a searchable
tree and finds elements by tag, class or CSS selector, and it can use the standard library’s parser underneath.
Regular expressions are a poor fit for HTML: attributes in any order, comments and optional closing tags break them.
Checkpoint
Check: Reading HTML with callbacks
An allowlist decides which links to keep
The collector’s last loop decides which of the four links to keep. urljoin() turns the relative link into a full
address, and urlsplit(url).scheme gives each link’s scheme, which the code checks against an allowlist: it keeps
http and https and skips the rest, here a mailto: and an ftp: link. Step through the four links:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Block B. The href /rooms?block=B&floor=2 is relative. urljoin() puts it on the page's base address, https://college.example.org/notices/, and the scheme of the full address is https, so the link is kept.
- Write to the office. mailto:office@example.org is already a full address. Its scheme, mailto, is not on the allowlist, so the link is skipped.
- the FAQ. https://example.org/faq stays as it is, because it is a full address already, and its scheme is https, so it is kept.
- the old files. ftp://files.example.org/old has the scheme ftp, which is not on the allowlist, so it is skipped too.
- A scheme nobody listed. The allowlist names what is accepted, so a scheme nobody thought of, such as javascript:, is refused without anyone writing a rule for it. A denylist would have to name every bad scheme in advance.
An allowlist names what is accepted, so a scheme nobody thought of is refused by default; a denylist, which names
what is refused, lets through every scheme its author forgot. The same pattern fits wherever a value from outside
decides what a program does, such as the file extensions an upload may have. A link that passes still goes into the
page through html.escape(), quoted, as the previous section showed: allowing and escaping are two separate steps.
Checkpoint
Check: Allowing only safe links
Practise: read a feed safely and show comments escaped
The exercise asks for parse_feed(), which refuses a document over a size limit or with any DOCTYPE and returns
each item’s title and link, with empty strings for missing ones, and for render_comment(), which escapes a name and
a comment into one line of HTML. Its sample tests run in your browser:
Exercise · Medium · Python
Read a feed safely and show comments escaped
Write two functions in feed_tools.py. Keep the names and parameters: the tests import them.
parse_feed(xml_text, max_chars=200_000) reads an RSS-style feed that came from outside the program and returns a list of (title, link) tuples, one per <item> of the <channel>, in order.
- A missing
<title>or<link>gives an empty string in its place. - Raise
ValueErrorwhen the text is longer thanmax_charscharacters, and when the document declares a DOCTYPE, whatever the DOCTYPE contains. Check the length before parsing. - Text that is not well-formed XML may raise
xml.etree.ElementTree.ParseError, asfromstring()does.
render_comment(name, text) returns one comment as HTML, exactly in this shape:
<p class="comment"><b>NAME</b>: TEXT</p>
where NAME and TEXT are the arguments with &, <, >, " and ' escaped, so a reader's text always appears as text.
Starter code · feed_tools.py
import xml.etree.ElementTree as ET
from html import escape
def parse_feed(xml_text, max_chars=200_000):
"""Return [(title, link), ...] for the feed's items; ValueError for a DOCTYPE or a document that is too long."""
root = ET.fromstring(xml_text)
return [(item.findtext("title"), item.findtext("link")) for item in root.iter("item")]
def render_comment(name, text):
"""One comment as HTML: <p class="comment"><b>NAME</b>: TEXT</p>, with both values escaped."""
return f'<p class="comment"><b>{name}</b>: {text}</p>'The sample tests · test_feed_tools.py
import xml.etree.ElementTree as ET
from feed_tools import parse_feed, render_comment
FEED = """<?xml version="1.0"?>
<rss version="2.0"><channel>
<title>Notices</title>
<item><title>Exam timetable</title><link>https://example.org/exams</link></item>
<item><title>Fees & forms</title></item>
<item><link>https://example.org/untitled</link></item>
</channel></rss>"""
def test_titles_and_links():
"""every item's title and link, in order, with empty strings for missing ones"""
assert parse_feed(FEED) == [
("Exam timetable", "https://example.org/exams"),
("Fees & forms", ""),
("", "https://example.org/untitled"),
]
def test_channel_title_is_not_an_item():
"""only items count, not the channel's own title"""
assert len(parse_feed(FEED)) == 3
def test_refuses_doctype():
"""a document that declares a DOCTYPE is refused, even a harmless one"""
text = '<!DOCTYPE rss [<!ENTITY org "Example College">]><rss><channel><item><title>&org;</title></item></channel></rss>'
try:
parse_feed(text)
except ValueError:
return
assert False, "the DOCTYPE was accepted"
def test_refuses_long_documents():
"""a document longer than max_chars is refused"""
try:
parse_feed(FEED, max_chars=len(FEED) - 1)
except ValueError:
assert parse_feed(FEED, max_chars=len(FEED)) # exactly at the limit is fine
return
assert False, "max_chars was ignored"
def test_broken_xml_still_raises_parse_error():
"""text that is not XML raises ParseError"""
try:
parse_feed("<rss><channel></rss>")
except ET.ParseError:
return
assert False, "broken XML was accepted"
def test_render_comment_escapes():
"""markup characters in the name and the text appear as text"""
html = render_comment('Ravi "R" <admin>', "Use <b> & 'quotes'")
assert html == '<p class="comment"><b>Ravi "R" <admin></b>: Use <b> & 'quotes'</p>'
def test_render_plain_text_unchanged():
"""plain text passes through as it is"""
assert render_comment("Asha", "Thanks!") == '<p class="comment"><b>Asha</b>: Thanks!</p>'A hint
Subclass xml.etree.ElementTree.TreeBuilder and give it a doctype(self, name, pubid, system) method that raises ValueError; pass it to the parser with ET.fromstring(text, parser=ET.XMLParser(target=YourBuilder())). Then root.find("channel"), findall("item") and findtext("title", default="") give you each item's values. For render_comment(), html.escape() escapes all five characters when quote is left at its default.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Reading trees, building XML, the DOCTYPE rule, escaping, parsing HTML and the link allowlist all come back in the practice below, starting from the idea your answers show you know least; a wrong pick names the slip behind it:
Practice that adapts to you
Practise this lesson
15 questions about 7 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Why must text from users go through html.escape() before it is put into an HTML page?
Because the browser cannot tell your markup from theirs. A page is one string of HTML, and anything in it that looks
like a tag is treated as one, so a comment containing <h1> or an unclosed <b> changes the page’s structure, and
a carefully written one can add elements that run code in other visitors’ browsers, which is called cross-site
scripting. html.escape() replaces &, < and > with entities, and by default both quote characters, so the
browser displays the characters as text instead of reading them as markup. It should run when the page is built,
not when the text is stored, so nothing is escaped twice. It is not the whole story: a link a user supplies also
needs its scheme checked against an allowlist, because escaping does not change what an address does.
Key takeaways
- ElementTree reads a document into a tree of elements:
find()andfindall()look one level down unless the path says more,findtext()takes a default, anditer()walks any depth. - Build XML with
ElementandSubElement, which escape&and<as they write; a joined string with a bare&is not XML at all. - The entity tricks of untrusted XML need a DOCTYPE, and Expat below 2.7.2 may be vulnerable: check
pyexpat.EXPAT_VERSIONwhere your code runs. - For untrusted XML, cap the size before parsing and refuse any DOCTYPE, through a
TreeBuilderwhosedoctype()raises or defusedxml’sforbid_dtd=True. html.escape()every value you write into a page once, when the page is built, withquote=Trueinside attributes.HTMLParserreports tags and text through your methods; keep only the links whose scheme is on an allowlist ofhttpandhttps.
Terms in this lesson
- allowlist
- A list of what is accepted, such as the URL schemes http and https; anything not on it is refused. Unlike a denylist, it also refuses values nobody thought of.
- autoescaping
- A template-engine setting that escapes every value put into an HTML template unless the template marks it as safe, so a forgotten call to an escape function cannot let markup through.
- cross-site scripting
- A weakness in which text from one user reaches a page without escaping, so the browser reads it as markup and can run code in other visitors' browsers. Escaping output for its place in the page prevents it.
- DOCTYPE
- A document type declaration near the start of an XML document, <!DOCTYPE …>, which may declare entities. Feeds, invoices and API responses never need one, so code that parses untrusted XML can refuse it.
- element (XML)
- One node of an XML tree: a tag such as item, optional attributes such as version="2.0", optional text and child elements. ElementTree gives each element its tag, attrib, text and children.
- entity (XML)
- A named piece of text that a reference such as &name; stands for. &, <, >, " and ' are built in; any other entity is declared in a DOCTYPE, which the expansion tricks depend on.
References
- xml.etree.ElementTree: the ElementTree XML API (Python Software Foundation)
- XML processing modules: XML security (Python Software Foundation)
- html: HyperText Markup Language support (html.escape) (Python Software Foundation)
- html.parser: simple HTML and XHTML parser (Python Software Foundation)
- urllib.parse: parse URLs into components (Python Software Foundation)
- Extensible Markup Language (XML) 1.0 (World Wide Web Consortium)
- XML External Entity Prevention Cheat Sheet (OWASP Foundation)
- Cross Site Scripting Prevention Cheat Sheet (OWASP Foundation)
- defusedxml (Python Package Index) (Python Package Index)
- Beautiful Soup Documentation (Beautiful Soup project)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress