XML syntax and well-formedness

One bare ampersand, in an item called Fish & Chips, is enough to make a parser reject a whole product feed. XML has no repair mode: the specification calls a broken rule a fatal error, after which a processor “MUST NOT continue normal processing”, so every other item of the feed is lost with the one that broke the rule.

The rules that prevent this are few and exact. You will write documents that every parser accepts, turn a parser’s line and column into the mistake behind them, escape text and attribute values correctly, see what CDATA sections, comments and processing instructions are for, and decide when a value belongs in an element and when in an attribute.

Six rules make a document well-formed

Here is one order of a bookshop as XML. It has elements (<branch>Pune</branch>), attributes (sku="BK-101"), an empty element written as one tag (<note/>), a comment and, on the first line, the XML declaration:

<?xml version="1.0" encoding="UTF-8"?>
<!-- One bookshop order from the Pune branch -->
<order id="BK-10042" placed="2026-03-14T18:45:00+05:30">
  <branch>Pune</branch>
  <customer type="member">Asha Rao</customer>
  <items>
    <item sku="BK-101" qty="2">Data Formats in Practice</item>
    <item sku="BK-205" qty="1">Fish &amp; Chips: A Kochi Cookbook</item>
  </items>
  <amount currency="INR">379.00</amount>
  <note/>
</order>

A document like this is well-formedwell-formed: Following the syntax rules of XML 1.0: one root element, tags that match and nest, quoted attribute values with no name repeated, and < and & escaped. A parser refuses any document that is not well-formed. when it keeps six rules of the XML 1.0 specification:

1. There is exactly one root elementroot element: The one element of an XML document that contains all the others, also called the document element. Comments and processing instructions may stand outside it, but no other element may., here order, and it contains every other element. Comments and processing instructions may sit before and after it, but no second element.
2. Every start tag has an end tag with the same name, or the element is written as an empty-element tagOne tag that opens and closes an element at once, written with a slash just before its closing bracket. It means the same as a start tag followed at once by its end tag.. Names are case-sensitive: <Item> is not closed by </item>.
3. Elements nest: the element opened last is closed first, so <b><i>sale</b></i> is not allowed.
4. Every attribute value is in quotes, single or double, and an attribute name appears at most once in a tag.
5. The characters < and & never appear on their own in text or in attribute values; the next sections show how to write them.
6. The XML declaration, if there is one, is the very first thing in the file, with not even a blank line before it.

Names have rules of their own. A name starts with a letter or an underscore and may then contain digits, hyphens and dots, so 1st-floor is not a name. Names that start with xml, in any mix of capitals, are reserved for the standards. The colon is allowed, but the specification asks authors to keep it for namespaces, the subject of the next lesson.

The program below puts ten small documents, each breaking one rule, through Python’s ElementTree and prints what it reported. Press Edit, mend one of the documents and run it again: its line then says accepted.

"""Ten small documents, each breaking one rule of well-formed XML, through Python's ElementTree."""
import xml.etree.ElementTree as ET

BROKEN = [
    ("two root elements", "<order/><order/>"),
    ("end tag does not match", "<item>Pens</itm>"),
    ("tags overlap", "<b><i>sale</b></i>"),
    ("names differ in case", "<Item>Pens</item>"),
    ("attribute not quoted", "<item qty=2/>"),
    ("attribute written twice", '<item qty="2" qty="3"/>'),
    ("bare ampersand in text", "<name>Fish & Chips</name>"),
    ("< in an attribute value", '<rule test="a<b"/>'),
    ("name starts with a digit", "<1st-floor/>"),
    ("declaration not first", '\n<?xml version="1.0"?><order/>'),
]

print(f"{'rule broken':<26} {'line':>4} {'col':>4}  message")
for rule, text in BROKEN:
    try:
        ET.fromstring(text)
        print(f"{rule:<26} accepted")
    except ET.ParseError as err:
        line, column = err.position
        message = str(err).split(":")[0]
        print(f"{rule:<26} {line:>4} {column:>4}  {message}")

rule broken                line  col  message
two root elements             1    8  junk after document element
end tag does not match        1   12  mismatched tag
tags overlap                  1   12  mismatched tag
names differ in case          1   12  mismatched tag
attribute not quoted          1   10  not well-formed (invalid token)
attribute written twice       1   14  duplicate attribute
bare ampersand in text        1   12  not well-formed (invalid token)
< in an attribute value       1   13  not well-formed (invalid token)
name starts with a digit      1    1  not well-formed (invalid token)
declaration not first         2    0  XML or text declaration not at start of entity

Each document produced a parse error, and the messages are brief: three different mistakes, tags that overlap, names that differ in case and an end tag spelt wrongly, all read mismatched tag. The message names the kind of error; the position says where to look.

A parser stops at its first error and reports where it noticed it

A parser reads the text from the start and keeps a list of the elements that are still open. Every end tag must match the element opened last, and the first time one does not, the parser stops. It cannot know where the real mistake was, only where the text stopped making sense. Step through a document whose item on line 2 is never closed:

The parser accepted lines 2 and 3, because an item inside an item breaks no rule of XML, and stopped on line 4. The next program prints the reported line of that document, of a bare ampersand and of the first document with its last line deleted, each with a caret under the reported column:

"""Show where ElementTree stopped: the reported line, with a caret under the reported column."""
import xml.etree.ElementTree as ET

DOCS = {
    "an item that is never closed": "<items>\n  <item>Pens\n  <item>Pads</item>\n</items>\n",
    "a bare ampersand": "<menu>\n  <name>Fish & Chips</name>\n</menu>\n",
    "the same item, </items> deleted": "<items>\n  <item>Pens\n  <item>Pads</item>\n",
}

for title, text in DOCS.items():
    try:
        ET.fromstring(text)
    except ET.ParseError as err:
        line, column = err.position
        print(f"{title}: {str(err).split(':')[0]}")
        print(f"  reported at line {line}, column {column} (column {column + 1} when counted from 1)")
        lines = text.splitlines()
        if line > len(lines):
            lines.append("")  # the parser reached the end of the text, after the last line break
        for number, source in enumerate(lines, start=1):
            print(f"  {number} | {source}")
            if number == line:
                print("    | " + " " * column + "^")
        print()

an item that is never closed: mismatched tag
  reported at line 4, column 2 (column 3 when counted from 1)
  1 | <items>
  2 |   <item>Pens
  3 |   <item>Pads</item>
  4 | </items>
    |   ^

a bare ampersand: not well-formed (invalid token)
  reported at line 2, column 14 (column 15 when counted from 1)
  1 | <menu>
  2 |   <name>Fish & Chips</name>
    |               ^
  3 | </menu>

the same item, </items> deleted: no element found
  reported at line 4, column 0 (column 1 when counted from 1)
  1 | <items>
  2 |   <item>Pens
  3 |   <item>Pads</item>
  4 | 
    | ^

Two habits follow from this run. First, read upwards from the reported line: a missing end tag is reported where the first end tag that does not match arrives, two lines later here and possibly hundreds in a long file. Second, check how your tool counts. ElementTree counts lines from 1 and columns from 0, and it points at the first character it could not accept: for the bare ampersand that is the space after &, because an ampersand must start a reference such as &. An editor that counts columns from 1 shows the same place one column further on.

Well-formed is not the same as valid

A document can keep every rule of XML and still be wrong for the program that reads it. The three orders below break the shop’s rules, a quantity written as a word, an order without an amount and an element nobody defined, and ElementTree accepts all of them:

"""Three orders that break the shop's rules but none of XML's: the parser accepts every one."""
import xml.etree.ElementTree as ET

ORDERS = {
    "quantity written as a word": '<order id="BK-10043"><item sku="BK-101" qty="two"/></order>',
    "amount missing": '<order id="BK-10044"><branch>Pune</branch></order>',
    "element the shop never defined": '<order id="BK-10045"><discount>100%</discount></order>',
}

for problem, text in ORDERS.items():
    root = ET.fromstring(text)
    print(f"{problem}: parsed, root <{root.tag}> with {[child.tag for child in root]}")

quantity written as a word: parsed, root <order> with ['item']
amount missing: parsed, root <order> with ['branch']
element the shop never defined: parsed, root <order> with ['discount']

Well-formed means the text follows XML’s own syntax. Validvalid: Of an XML document: well-formed and also following a schema, such as a DTD or an XSD, that says which elements and attributes the document has, in which order and of which types. means it also follows a schemaschema: A set of rules that says which values a document may hold, such as which fields it needs and the type of each. JSON Schema and XML Schema (XSD) write those rules as documents of their own. More in “Choosing a data format: JSON, YAML, TOML, XML, CSV”, the rules for one kind of document: which elements and attributes an order has, in which order, and of which types. XML 1.0 itself defines validity against a DTD, the document type definition that a document type declaration contains or points to. Schemas are also written in XML Schema (XSD) or RELAX NG, and a later lesson of this module validates with XSD. Step through the checks a document passes on its way into a program:

Each check catches what the earlier ones cannot see, and none replaces another. A parser never refuses qty="two", and a schema that lists elements and their types, as a DTD or an XSD 1.0 schema does, cannot check that an amount equals the sum of the items. XSD 1.1 can state such a rule as an assertion; without one, the rule is your code’s job, on data that passed both checks.

Escape five characters, and only where they need it

XML reserves < for tags and & for references, so text that contains them must spell them differently. Five entity referencesentity reference: A name between & and ; that stands for text, such as &lt; for < and &amp; for &. XML predefines five of them; any other must be declared in the document's DTD. are predefined, and every parser knows them without being told:

Any other character can be written as a character referencecharacter reference: A character written by its Unicode code point between &# and ;, in decimal or in hexadecimal after &#x, so &#8377; and &#x20B9; are both the rupee sign. A parser replaces it with the character., its Unicode code point in decimal or hexadecimal: ₹ and ₹ are both the rupee sign. A parser replaces every reference with its character, so the program that reads the document never sees the reference itself. Named entities from HTML, such as  , are not predefined in XML, and a parser stops on them with undefined entity unless the document declares them.

Attribute values hold one more surprise. A parser normalises them: every line break and tab written in an attribute value is read back as a space. To keep one, write it as the character reference 
 or 	. The program shows both traps, then the two ways to escape correctly, with the helpers of xml.sax.saxutils and by letting ElementTree write the element:

"""Escaping text and attribute values: by hand, with xml.sax.saxutils, and with ElementTree's serializer."""
import xml.etree.ElementTree as ET
from xml.sax.saxutils import escape, quoteattr

name = "Fish & Chips <Kochi>"
note = 'Ask for "extra" lime\nNo onions'

# 1. An f-string with no escaping: not well-formed.
raw = f"<item note=\"{note}\">{name}</item>"
try:
    ET.fromstring(raw)
except ET.ParseError as err:
    print("f-string, no escaping:", err)

# 2. A raw line break inside an attribute value is read back as a space.
print("raw line break read as:", repr(ET.fromstring('<item note="one\ntwo"/>').get("note")))

# 3. escape() for text and quoteattr() for an attribute value (it adds the quotes).
print("escape(name):    ", escape(name))
print("quoteattr(note): ", quoteattr(note))
back = ET.fromstring(f"<item note={quoteattr(note)}>{escape(name)}</item>")
print("read back:       ", repr(back.text), repr(back.get("note")))

# 4. ElementTree escapes for you, and writes the line break as a character reference.
item = ET.Element("item", note=note)
item.text = name
written = ET.tostring(item, encoding="unicode")
print("ElementTree wrote:", written)
again = ET.fromstring(written)
print("same values back:", again.text == name and again.get("note") == note)

# 5. A character reference names any character by its code point.
print(ET.fromstring("<price>&#x20B9;379 or &#8377;379</price>").text)

f-string, no escaping: not well-formed (invalid token): line 1, column 21
raw line break read as: 'one two'
escape(name):     Fish & Chips <Kochi>
quoteattr(note):  'Ask for "extra" lime
No onions'
read back:        'Fish & Chips <Kochi>' 'Ask for "extra" lime\nNo onions'
ElementTree wrote: <item note="Ask for &quot;extra&quot; lime&#10;No onions">Fish &amp; Chips &lt;Kochi&gt;</item>
same values back: True
₹379 or ₹379

The f-string without escaping failed at column 21, the e of extra: the quote before it had already ended the attribute value. The raw line break came back as a space. escape() handled the text and quoteattr() the attribute value, choosing single quotes because the note contains double ones and writing the line break as 
. ElementTree did the same work on its own and read back exactly the values it was given.

CDATA is just text; comments and processing instructions are not data

Three kinds of markup look unusual, and only one of them holds data. A CDATA sectionCDATA section: A block of text between <![CDATA[ and ]]> in which < and & need no escaping. A parser reports its content as ordinary text, and the section ends at the first ]]>., <![CDATA[ … ]]>, lets you write < and & without escaping them, which is handy for a snippet of code or a rule that people edit by hand. It ends at the first ]]>, so that sequence cannot appear inside it. A comment, <!-- … -->, is a note for people; it may not contain two hyphens in a row. A processing instructionprocessing instruction: Markup of the form <?target data?> that passes an instruction to an application, such as xml-stylesheet for a browser. It is not part of the document's data, and many parsers drop it., <?target data?>, passes an instruction to one application, as <?xml-stylesheet … ?> tells a browser which stylesheet to use. The XML declaration looks like one but is not: it may only stand at the start.

This menu uses all three, and the program shows what ElementTree keeps of them:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="menu.xsl"?>
<!-- Menu of the Kochi branch -->
<menu>
  <!-- prices include tax -->
  <item price="&#x20B9;250">Fish &amp; Chips</item>
  <rule><![CDATA[show if a < b && open]]></rule>
  <?print-hint bold?>
</menu>

"""What ElementTree keeps of comments, a CDATA section, processing instructions and character references."""
import xml.etree.ElementTree as ET

root = ET.parse("menu.xml").getroot()
for child in root:
    print(f"{child.tag}: {child.text!r} {child.attrib}")
print("written back:")
print(ET.tostring(root, encoding="unicode"))

# A TreeBuilder that keeps comments and processing instructions, but only those inside the root element.
builder = ET.TreeBuilder(insert_comments=True, insert_pis=True)
root = ET.parse("menu.xml", parser=ET.XMLParser(target=builder)).getroot()
print("with insert_comments and insert_pis:")
print(ET.tostring(root, encoding="unicode"))

item: 'Fish & Chips' {'price': '₹250'}
rule: 'show if a < b && open' {}
written back:
<menu>
  
  <item price="₹250">Fish &amp; Chips</item>
  <rule>show if a &lt; b &amp;&amp; open</rule>
  
</menu>
with insert_comments and insert_pis:
<menu>
  <!-- prices include tax -->
  <item price="₹250">Fish &amp; Chips</item>
  <rule>show if a &lt; b &amp;&amp; open</rule>
  <?print-hint bold?>
</menu>

The CDATA section arrived as ordinary text, and when the tree was written back it came out escaped: show if a < b && open. To the parser, a CDATA section and escaped text are two spellings of the same characters, so no program should depend on which one a file uses. Comments and processing instructions were dropped, which ElementTree does by default. A TreeBuilder with insert_comments and insert_pis keeps them, but only those inside the root element: the comment and the stylesheet instruction above <menu> were still lost.

Elements or attributes: let the value decide

The same order could be written <item sku="BK-101" qty="2"/> or with <sku> and <qty> elements, and both are well-formed. The choice matters when the data changes, because an attribute can do less than an element. Its name may appear once per tag, its value is a single string, and the specification says the order of attributes is not significant. Mixed content shows the difference most clearly. In directions such as <p>Take <b>exit 4</b> to <i>Station Road</i>, then <b>exit 2</b>.</p>, text and elements alternate, and their order is the meaning:

"""Mixed content: text and elements in one order. What ElementTree keeps, and what a naive JSON conversion loses."""
import json
import xml.etree.ElementTree as ET

first = ET.fromstring("<p>Take <b>exit 4</b> to <i>Station Road</i>, then <b>exit 2</b>.</p>")
second = ET.fromstring("<p>Take <i>Station Road</i> to <b>exit 4</b>, then <b>exit 2</b>.</p>")

# ElementTree keeps the order: the text before the first child, and each child's tail after it.
print("text:", repr(first.text))
for child in first:
    print(f"<{child.tag}> text={child.text!r} tail={child.tail!r}")


def naive(element):
    """Elements become keys and the pieces of text are joined: a common shortcut for record-like XML."""
    out = {}
    for child in element:
        out.setdefault(child.tag, []).append(child.text)
    out["#text"] = (element.text or "") + "".join(child.tail or "" for child in element)
    return out


for p in (first, second):
    print("sentence:  ", "".join(p.itertext()))
    print("naive JSON:", json.dumps(naive(p)))
print("same JSON for both sentences:", naive(first) == naive(second))

text: 'Take '
<b> text='exit 4' tail=' to '
<i> text='Station Road' tail=', then '
<b> text='exit 2' tail='.'
sentence:   Take exit 4 to Station Road, then exit 2.
naive JSON: {"b": ["exit 4", "exit 2"], "i": ["Station Road"], "#text": "Take  to , then ."}
sentence:   Take Station Road to exit 4, then exit 2.
naive JSON: {"i": ["Station Road"], "b": ["exit 4", "exit 2"], "#text": "Take  to , then ."}
same JSON for both sentences: True

ElementTree keeps the order: the text before the first child, then each child’s text and its tail, the text that follows it. The naive conversion makes each element name a key and joins the loose text, and two sets of directions that lead to different places, “exit 4 to Station Road” and “Station Road to exit 4”, turned into the same JSON. Books, contracts and documentation are full of mixed contentmixed content: The content of an element in which text and child elements alternate, such as a paragraph with words in bold. The order of its pieces is part of its meaning., which is where XML does better than JSON; for record-like data, either format works.

For each value of your own document, answer three questions and see which form fits:

In short: elements for values that repeat, may grow parts or carry markup, and attributes for short metadata about an element, such as its id, a code or a unit.

Practise: write one element that parses back exactly

The exercise asks you to write the escaping of this lesson as a function: an element with its attributes and text, which ElementTree must read back exactly, also when the text holds ]]> and an attribute holds a line break:

def element(tag, attrs, text):
    """One XML element as text, with the attribute values and the text escaped."""
    attributes = "".join(f' {name}="{value}"' for name, value in attrs.items())
    return f"<{tag}{attributes}>{text}</{tag}>"

import xml.etree.ElementTree as ET

from element import element


def test_plain_text():
    """writes a start tag, the text and an end tag"""
    assert element("branch", {}, "Pune") == "<branch>Pune</branch>"


def test_ampersand_and_angle_brackets():
    """escapes & and < in text, so the name parses back as written"""
    written = element("name", {}, "Fish & Chips <Kochi>")
    assert ET.fromstring(written).text == "Fish & Chips <Kochi>"


def test_no_cdata_end_in_text():
    """keeps the sequence ]]> out of the text"""
    written = element("rule", {}, "a]]>b")
    assert "]]>" not in written
    assert ET.fromstring(written).text == "a]]>b"


def test_quotes_in_an_attribute():
    """escapes a double quote inside a double-quoted attribute value"""
    written = element("item", {"note": 'Ask for "extra" lime'}, "Chai")
    assert ET.fromstring(written).get("note") == 'Ask for "extra" lime'


def test_line_break_and_tab_in_an_attribute():
    """writes a line break and a tab as character references, so they survive"""
    written = element("item", {"note": "one\ntwo\tthree"}, "Chai")
    assert ET.fromstring(written).get("note") == "one\ntwo\tthree"


def test_ampersand_and_less_than_in_an_attribute():
    """escapes & and < inside an attribute value, and keeps a carriage return"""
    written = element("rule", {"test": "a < b && open\r"}, "Chai")
    assert ET.fromstring(written).get("test") == "a < b && open\r"


def test_attributes_in_order_and_empty_element():
    """keeps the attributes in order and writes an empty element as a single tag"""
    assert element("item", {"sku": "BK-101", "qty": "2"}, "") == '<item sku="BK-101" qty="2"/>'

After that, test yourself on the six ideas of the lesson. Each question picks the idea you have shown the least so far, and the reasoning appears as soon as you answer:

Interview questions

Warm-up, for freshers and up to 5 years: how does a well-formed XML document differ from a valid one?

A well-formed document follows the syntax of XML itself: one root element, every start tag matched by an end tag in the right order, attribute values in quotes with no name repeated, and < and & escaped in text. Every XML parser checks this, and a document that is not well-formed is not XML at all, so the parser stops with a fatal error. A valid document is well-formed and also follows a schema for its kind of document, a DTD, an XSD or a RELAX NG grammar, which says which elements and attributes may appear, in which order and with which types. Validity is checked only when a program asks a validating parser to do it. So <qty>two</qty> is well-formed, and it is invalid only against a schema that says a quantity is a whole number.

Two harder questions follow: a feed that one ampersand broke, and the design of an export that people and programs both read. Settle on what you would do first in each case, then follow the question’s link and compare it with the first step given there:

Key takeaways

• A well-formed document has one root, end tags that match the element opened last, quoted attribute values with no name repeated, escaped < and &, and the XML declaration first or not at all.
• A parser stops at its first fatal error and reports where it noticed it, which for a missing end tag can be lines after the mistake; ElementTree counts lines from 1 and columns from 0.
• Well-formed is checked by every parser; valid means the document also follows a schema, and your own code checks the rules your schema does not state.
• Escape < and & everywhere, > in ]]>, and the quote that delimits an attribute; write line breaks in attribute values as 
, or let a library write the XML.
• CDATA sections are another spelling of text, comments and processing instructions are not data, and ElementTree drops the last two by default.
• Use elements for values that repeat, grow or carry markup, and attributes for short metadata; mixed content needs elements because their order is the meaning.