SEO & Digital Marketing Module 2 – Technical SEO: crawling and indexing
XML sitemaps that help, not hurt
Build a valid XML sitemap and index within the 50,000-URL and 50 MB limits, set honest lastmod dates, submit it to Search Console and fix what it reports.
What you will learn
- Build a valid XML sitemap and a sitemap index within the 50,000-URL and 50 MB limits
- Choose the addresses a sitemap lists, the canonical pages that answer 200 and may be indexed
- Set each lastmod to the last real change of a page, and leave out priority and changefreq
- Use Search Console or robots.txt to submit a sitemap, and fix the errors a check reports
Before you start
What you’ll learn · Before you start
What you will learn
- Build a valid XML sitemap and a sitemap index within the 50,000-URL and 50 MB limits
- Choose the addresses a sitemap lists, the canonical pages that answer 200 and may be indexed
- Set each lastmod to the last real change of a page, and leave out priority and changefreq
- Use Search Console or robots.txt to submit a sitemap, and fix the errors a check reports
Before you start
On this page
Pune Bakes’ shop software wrote a XML sitemap: A file, often sitemap.xml at a site's root, that lists the addresses of the pages a site wants Google to know about, each with the date it last changed. It helps discovery but guarantees neither crawling nor indexing. of every address it knew: 12 of them, 6 of which did not belong there, from a redirect and a page that no longer exists to a sorted copy and a thank-you page. Every entry also claimed that its page had changed that night, because the software stamped each one with the time of its nightly build.
A sitemap like that is a list Google has no reason to trust, and Google uses its dates only when they prove accurate. By the end of this page you can build a valid sitemap and, past 50,000 addresses, a sitemap index: A file that lists several sitemaps, each with its address and an optional lastmod, so that a site with more than 50,000 URLs or 50 MB can still submit one file. It cannot list another index.; choose the addresses it lists; set lastmod: The date, or date and time, that a sitemap entry gives for its page's last significant change: the main content, the structured data or the links. Google uses it only when a site's values prove accurate. values that are true; and submit it, read Search Console’s report on it and fix the errors a check finds.
A sitemap lists the pages you want found; it promises nothing
An XML sitemap is a file that tells Google about the pages and files of a site (Learn about sitemaps). It helps Google discover pages, and Google names three kinds of site that are likely to need one: a large site, a new site with few external links, and a site with a lot of rich media such as video and images. A site of about 500 pages or fewer, linked well enough that every page can be reached from the others, may not need one. Either way it is a hint: Google says a sitemap “doesn’t guarantee that all the items in your sitemap will be crawled and indexed”.
Google accepts XML sitemaps, RSS 2.0 and Atom 1.0 feeds, and plain text files with one address per line (Build and submit a sitemap). This lesson uses XML, the format that carries a date for every page. Step through what Google does with the bakery’s sitemap:
The drawing shows the whole scene; its 5 steps are listed below.
Every step as text
- Where the sitemap is. The bakery's robots.txt has the line Sitemap followed by the sitemap's full address, and the owner has also submitted it in the Sitemaps report of Search Console, which shows how it was processed.
- The sitemap is fetched. Googlebot downloads sitemap.xml and reads its six entries, each a loc and a lastmod.
- What changed. The mango mousse page has a lastmod newer than Google's last crawl of it. Google uses lastmod only when a site's values have proved consistently and verifiably accurate, so honest dates are what make this signal count.
- Crawled, perhaps. Googlebot fetches the changed page. A sitemap is a hint that helps Google find pages; it does not guarantee that every address will be crawled, or crawled soon.
- The report. The Sitemaps report shows Success and the number of pages discovered in the file. The Page indexing report, filtered to submitted pages, then shows which of them are indexed and why the others are not.
Checkpoint
Check: What a sitemap does and does not do
The format: urlset, url, loc, and five characters to escape
An XML sitemap is a urlset element of the sitemaps.org namespace with one url element per page; each url has a
loc, the page’s address, and may have a lastmod
(sitemaps.org protocol). The file must be UTF-8 encoded, every address must
be a full address with its protocol and shorter than 2,048 characters, and every value must be entity-escaped, as in
any XML file:
| Character | Written as |
|---|---|
& |
& |
' |
' |
" |
" |
> |
> |
< |
< |
The official schema also fixes the order inside a url: loc first, then lastmod, changefreq and priority
(sitemap.xsd). The program below builds the bakery’s
sitemap from what a crawl of the site found; ElementTree, Python’s XML library, writes the escapes itself
(ElementTree):
"""Build the bakery's sitemap from what a crawl of the site found.
Each row is one address the crawl met: its status code, the canonical its page names (None: the page itself), whether
it carries noindex, and the day its content last changed. Only canonical pages that answer 200, may be indexed and sit
on the site's own origin go in (Google Search Central, "Build and submit a sitemap"). ElementTree writes the XML, so
characters such as & are escaped for us.
"""
import xml.etree.ElementTree as ET
SITE = "https://punebakes.example"
NS = "http://www.sitemaps.org/schemas/sitemap/0.9"
PAGES = [
("https://punebakes.example/", 200, None, False, "2026-10-02"),
("https://punebakes.example/cakes/", 200, None, False, "2026-09-28"),
("https://punebakes.example/cakes/eggless-truffle", 200, None, False, "2026-09-14"),
("https://punebakes.example/cakes/mango-mousse", 200, None, False, "2026-10-09"),
("https://punebakes.example/collections?occasion=diwali&size=large", 200, None, False, "2026-10-05"),
("https://punebakes.example/menu/", 200, None, False, "2026-08-30"),
("https://punebakes.example/cakes/?sort=price", 200, "https://punebakes.example/cakes/", False, "2026-09-28"),
("https://punebakes.example/cakes/mango-mousse?utm_source=insta", 200,
"https://punebakes.example/cakes/mango-mousse", False, "2026-10-09"),
("https://punebakes.example/old-offers", 301, None, False, "2026-03-01"),
("https://punebakes.example/thank-you", 200, None, True, "2026-06-11"),
("http://punebakes.example/menu/", 301, None, False, "2026-08-30"),
("https://punebakes.example/cakes/red-velvet", 404, None, False, "2026-05-20"),
]
def reason_to_leave_out(url, status, canonical, noindex):
"""Why an address does not belong in the sitemap, or None if it does."""
if not url.startswith(SITE + "/"):
return f"not on {SITE}"
if status != 200:
return f"answers {status}, not 200"
if canonical not in (None, url):
return f"its canonical is {canonical.removeprefix(SITE)}"
if noindex:
return "carries noindex"
return None
ET.register_namespace("", NS)
urlset = ET.Element(f"{{{NS}}}urlset")
left_out = []
for url, status, canonical, noindex, changed in PAGES:
reason = reason_to_leave_out(url, status, canonical, noindex)
if reason:
left_out.append((url.removeprefix(SITE), reason))
continue
entry = ET.SubElement(urlset, f"{{{NS}}}url")
ET.SubElement(entry, f"{{{NS}}}loc").text = url
ET.SubElement(entry, f"{{{NS}}}lastmod").text = changed
print(f"{len(PAGES)} addresses found, {len(left_out)} left out:")
for path, reason in left_out:
print(f" {path}: {reason}")
ET.indent(urlset)
print(f"sitemap.xml, {len(urlset)} URLs:")
print('<?xml version="1.0" encoding="UTF-8"?>')
print(ET.tostring(urlset, encoding="unicode"))Output
12 addresses found, 6 left out:
/cakes/?sort=price: its canonical is /cakes/
/cakes/mango-mousse?utm_source=insta: its canonical is /cakes/mango-mousse
/old-offers: answers 301, not 200
/thank-you: carries noindex
http://punebakes.example/menu/: not on https://punebakes.example
/cakes/red-velvet: answers 404, not 200
sitemap.xml, 6 URLs:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://punebakes.example/</loc>
<lastmod>2026-10-02</lastmod>
</url>
<url>
<loc>https://punebakes.example/cakes/</loc>
<lastmod>2026-09-28</lastmod>
</url>
<url>
<loc>https://punebakes.example/cakes/eggless-truffle</loc>
<lastmod>2026-09-14</lastmod>
</url>
<url>
<loc>https://punebakes.example/cakes/mango-mousse</loc>
<lastmod>2026-10-09</lastmod>
</url>
<url>
<loc>https://punebakes.example/collections?occasion=diwali&size=large</loc>
<lastmod>2026-10-05</lastmod>
</url>
<url>
<loc>https://punebakes.example/menu/</loc>
<lastmod>2026-08-30</lastmod>
</url>
</urlset>
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 build_sitemap.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Look at the fifth loc: the address collections?occasion=diwali&size=large is written with &, which becomes the
address itself again when the XML is read. Build a sitemap with an XML library like this one, never by joining strings,
and the escaping and the namespace take care of themselves.
Checkpoint
Check: The XML format and escaping
Only canonical pages that answer 200 and may be indexed
Google’s rule for what a sitemap lists is short: the addresses you want people to find in its results, and of a set
of duplicate pages, only the canonical one
(Build and submit a sitemap). The
first lines the program printed show what that took out of the bakery’s 12 addresses: two copies whose canonical names
another address, a redirect, a page with noindex, a page that answers 404, and an http address, which a sitemap on
https may not list. Step through the checks:
The drawing shows the whole scene; its 4 steps are listed below.
Every step as text
- The mango mousse page. /cakes/mango-mousse answers 200, names itself as canonical and has no noindex, so it passes all three checks and goes in, with the day its content last changed as its lastmod.
- The old offers page. /old-offers answers 301, a redirect. A sitemap lists the addresses Google should show, and a redirect's target is the one to list, so this address is left out.
- A sorted copy. /cakes/?sort=price answers 200, but its canonical points to /cakes/. Of a set of duplicates, only the canonical belongs in the sitemap.
- The thank-you page. /thank-you answers 200 and is its own canonical, but it carries noindex. Listing a page the site asks Google not to index sends two opposite signals.
Change one row of the crawl
In the program above, press Edit, change the 301 of /old-offers to 200 and run again: the address now passes
every check and joins the sitemap, which has 7 URLs.
Each address left out is a contradiction the sitemap would otherwise make. A redirect says the page lives elsewhere, a noindex says do not show it, and a copy’s canonical names another address; listing any of them asks Google to show an address that the site itself says not to show. In the Page indexing report, filtered to submitted pages, such addresses appear among the pages not indexed, each with its reason (Page indexing report).
Key idea
List exactly the addresses you want shown, canonical, answering 200 and allowed to be indexed, so that every entry agrees with every other signal the site sends.
Checkpoint
Check: Which addresses belong in it
lastmod must be true, and Google ignores priority and changefreq
Google ignores <priority> and <changefreq>. It uses <lastmod> only when a site’s values are consistently and
verifiably accurate, which it can tell by setting them beside each page’s own last modification, and the value should
be the date of the last significant update of the page: a change to its main content, its structured data or its
links, not to the copyright date
(Build and submit a sitemap). The
format is W3C Datetime: a date such as 2026-10-09, or a date and time with its zone, such as
2026-10-09T08:30:00+05:30 (sitemaps.org protocol). Walk through six
changes to the bakery’s mango mousse page:
- Published
The page goes live lastmod set
The page is published with its recipe, photos and price. Its lastmod is the day of publication.
- New price
The price and a photo change lastmod moves
The main content changed, which Google counts as significant, so lastmod moves to this day.
- Footer year
The footer's copyright year changes lastmod stays
Google says an update to the copyright date is not significant. The page looks the same to a reader, so lastmod stays.
- Nightly build
The site is rebuilt overnight lastmod stays
The files are written again, but the page's content is the same. A lastmod set to the build time would claim a change that never happened.
- New links
Links to two new recipes are added lastmod moves
Links on the page count as a significant change, like the main content and the structured data, so lastmod moves.
- New structured data
Recipe markup is added lastmod moves
The page gains structured data for its recipe, which Google also lists as significant, so lastmod moves again.
The timeline has 6 events, from Published to New structured data.
So the bakery’s program takes each page’s lastmod from the day its content last changed, the last field of each crawl row, and never from the time of the build. A site that stamps every entry with its build time claims thousands of changes a night that never happened, and lastmod values that are not accurate are values Google does not use.
Checkpoint
Check: Honest lastmod, ignored priority
Past 50,000 URLs or 50 MB, split the sitemap and add an index
One sitemap holds at most 50,000 URLs or 50 MB uncompressed, whichever comes first; a compressed file must still be no
larger than 50 MB once uncompressed (sitemaps.org protocol). Larger sites
split their URLs into several sitemaps and list those in a sitemap index, a
sitemapindex element with one sitemap entry per file, each holding a loc and an optional lastmod
(sitemap index files):
| Limit | Value |
|---|---|
| URLs in one sitemap | 50,000 |
| Size of one sitemap, uncompressed | 50 MB, that is 52,428,800 bytes |
| Sitemaps listed in one index | 50,000 |
| Sitemap index files per site in Search Console | 500 |
The sitemaps an index lists must be on the same site as the index, in its folder or below it, and an index cannot list another index. The program below splits Spice Box’s 120,000 product pages and prints the index it would publish at the site’s root:
"""Split a large catalogue into sitemaps of at most 50,000 URLs and list them in a sitemap index.
A sitemap may hold 50,000 URLs or 50 MB (52,428,800 bytes) uncompressed, whichever comes first, and an index may list
50,000 sitemaps (sitemaps.org protocol; Google Search Central, "Build and submit a sitemap"). The shop's 120,000
product pages are made up; every URL and date comes from its number. The program writes each file in memory, counts its
bytes, and prints the index it would publish.
"""
from xml.sax.saxutils import escape
SITE = "https://spicebox.example"
MAX_URLS = 50_000
MAX_BYTES = 50 * 1024 * 1024
HEAD = '<?xml version="1.0" encoding="UTF-8"?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n'
products = [(f"{SITE}/products/item-{n:06d}", f"2026-{1 + n % 9:02d}-{1 + n % 28:02d}") for n in range(1, 120_001)]
parts = []
for start in range(0, len(products), MAX_URLS):
chunk = products[start:start + MAX_URLS]
body = "".join(f" <url><loc>{escape(url)}</loc><lastmod>{day}</lastmod></url>\n" for url, day in chunk)
text = HEAD + body + "</urlset>\n"
parts.append((f"products-{len(parts) + 1}.xml", len(chunk), len(text.encode("utf-8")), max(d for _, d in chunk)))
print(f"{len(products):,} product URLs, at most {MAX_URLS:,} a sitemap:")
for name, count, size, newest in parts:
print(f" {name}: {count:,} URLs, {size:,} bytes ({size / MAX_BYTES:.1%} of 50 MB)")
print(f"50 MB comes first only if an entry averages over {MAX_BYTES / MAX_URLS:,.1f} bytes")
print("sitemap_index.xml:")
print('<?xml version="1.0" encoding="UTF-8"?>')
print('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">')
for name, count, size, newest in parts:
print(f" <sitemap><loc>{SITE}/sitemaps/{name}</loc><lastmod>{newest}</lastmod></sitemap>")
print("</sitemapindex>")Output
120,000 product URLs, at most 50,000 a sitemap: products-1.xml: 50,000 URLs, 4,950,110 bytes (9.4% of 50 MB) products-2.xml: 50,000 URLs, 4,950,110 bytes (9.4% of 50 MB) products-3.xml: 20,000 URLs, 1,980,110 bytes (3.8% of 50 MB) 50 MB comes first only if an entry averages over 1,048.6 bytes sitemap_index.xml: <?xml version="1.0" encoding="UTF-8"?> <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap><loc>https://spicebox.example/sitemaps/products-1.xml</loc><lastmod>2026-09-28</lastmod></sitemap> <sitemap><loc>https://spicebox.example/sitemaps/products-2.xml</loc><lastmod>2026-09-28</lastmod></sitemap> <sitemap><loc>https://spicebox.example/sitemaps/products-3.xml</loc><lastmod>2026-09-28</lastmod></sitemap> </sitemapindex>
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 split_sitemaps.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
Each full sitemap of 50,000 URLs takes 4,950,110 bytes, 9.4 % of the 50 MB allowed, so here the count of URLs decides. The size limit comes first only when an entry averages more than 1,048.6 bytes, which a plain entry rarely does but an entry with many image tags can: an entry may list up to 1,000 images (image sitemaps).
Checkpoint
Check: Limits, splitting and the index
Submit it, then fix what the checks report
Google finds a sitemap in three ways: you submit it in Search Console’s Sitemaps report, you submit it with the Search
Console API, or you add a line Sitemap: with its full address anywhere in robots.txt
(Build and submit a sitemap). The
Sitemaps report then gives each submitted sitemap a status
(Sitemaps report):
| Status | What it means |
|---|---|
| Success | Loaded and processed with no errors; its URLs go on to be crawled |
| Has errors | Fetched, with one or more errors; the URLs that parsed still go on |
| Couldn’t fetch | Not downloaded at all; Google tries again for a few days |
Its error messages name the faults a check can catch before you submit, among them “Parsing error”, usually a special character left unescaped, “Invalid date”, “Invalid URL”, “URL not allowed”, for an address above the sitemap’s folder or on another host or protocol, and “Incorrect sitemap index format: Nested sitemap indexes”. The program below runs those checks on three of the bakery’s sitemaps:
"""Check sitemaps against the rules of the sitemaps.org schema and Google's documentation, with line numbers.
The XML must be well-formed; inside <url>, loc comes first, then lastmod, changefreq and priority (the schema's
sequence); loc is a full address on the sitemap's own host; lastmod is a W3C date or date-time; changefreq is one of
seven words; priority lies between 0.0 and 1.0. Three sitemaps of the bakery: two broken, one repaired.
"""
import re
import xml.parsers.expat
from urllib.parse import urlsplit
HOST = "punebakes.example"
ORDER = ["loc", "lastmod", "changefreq", "priority"]
FREQUENCIES = {"always", "hourly", "daily", "weekly", "monthly", "yearly", "never"}
W3C_DATE = re.compile(r"\d{4}-\d{2}-\d{2}(T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}:\d{2}))?")
SITEMAPS = {
"ampersand.xml": """<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://punebakes.example/collections?occasion=diwali&size=large</loc>
</url>
</urlset>""",
"values.xml": """<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://punebakes.example/cakes/</loc>
<lastmod>30/09/2026</lastmod>
</url>
<url>
<lastmod>2026-10-09</lastmod>
<loc>https://punebakes.example/cakes/mango-mousse</loc>
</url>
<url>
<loc>https://punebakes.example/menu/</loc>
<priority>1.5</priority>
</url>
<url>
<loc>/offers/</loc>
</url>
<url>
<loc>https://www.punebakes.example/</loc>
</url>
</urlset>""",
"repaired.xml": """<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://punebakes.example/collections?occasion=diwali&size=large</loc>
<lastmod>2026-10-05</lastmod>
</url>
<url>
<loc>https://punebakes.example/cakes/mango-mousse</loc>
<lastmod>2026-10-09T08:30:00+05:30</lastmod>
</url>
</urlset>""",
}
def problems(text):
"""[(line, problem)] for one sitemap."""
found, seen, value = [], [], []
parser = xml.parsers.expat.ParserCreate()
def start(name, attrs):
if name == "url":
seen.clear()
elif name in ORDER:
value.clear()
if seen and ORDER.index(name) < ORDER.index(seen[-1]):
found.append((parser.CurrentLineNumber, f"{name} comes after {seen[-1]}; loc must be first"))
seen.append(name)
def end(name):
line, text = parser.CurrentLineNumber, "".join(value).strip()
if name == "loc":
u = urlsplit(text)
if not u.scheme:
found.append((line, f'loc "{text}" is not a full address'))
elif u.hostname != HOST:
found.append((line, f"loc {text} is not on {HOST}"))
elif name == "lastmod" and not W3C_DATE.fullmatch(text):
found.append((line, f'lastmod "{text}" is not a W3C date such as 2026-09-30'))
elif name == "changefreq" and text not in FREQUENCIES:
found.append((line, f'changefreq "{text}" is not one of the seven words'))
elif name == "priority" and not 0.0 <= float(text) <= 1.0:
found.append((line, f"priority {text} is outside 0.0 to 1.0"))
parser.StartElementHandler, parser.EndElementHandler = start, end
parser.CharacterDataHandler = value.append
try:
parser.Parse(text, True)
except xml.parsers.expat.ExpatError as e:
found.append((e.lineno, f"{xml.parsers.expat.ErrorString(e.code)}: is a & written as &?"))
return found
for name, text in SITEMAPS.items():
found = problems(text)
count = f"{len(found)} problem" + ("s" if len(found) != 1 else "")
print(f"{name}: {'valid' if not found else count}")
for line, problem in found:
print(f" line {line}: {problem}")Output
ampersand.xml: 1 problem line 4: not well-formed (invalid token): is a & written as &? values.xml: 5 problems line 5: lastmod "30/09/2026" is not a W3C date such as 2026-09-30 line 9: loc comes after lastmod; loc must be first line 13: priority 1.5 is outside 0.0 to 1.0 line 16: loc "/offers/" is not a full address line 19: loc https://www.punebakes.example/ is not on punebakes.example repaired.xml: valid
Recorded with Python 3.14.8 on macOS 26 arm64. To run it yourself: mise exec python@3.14.8 -- python3 check_sitemap.py
Runs on this device, in your browser. The first run downloads Python (about 13.5 MB), which is kept for the next runs.
Your run, in this browser
The first file is not XML at all: an unescaped & stops the parser on line 4, so nothing after it is read. The second
is well-formed but breaks five rules: a date written day first, lastmod before loc, a priority of 1.5, a relative
address and an address on the www host. The third, with the & escaped and the dates in W3C form, is valid. A
sitemap that Google found only through robots.txt does
not show in the Sitemaps report, so submit it there as well to see its status, and deleting a sitemap from the report
does not make Google forget it or the addresses it listed.
Checkpoint
Check: Submitting and fixing errors
Practise: repair a 30-address sitemap
Spice Box’s shop software writes every address it knows, relative ones included, stamps them all with the build time
and gives each a priority. Write build() so that it returns the sitemap Google should get; the sample tests parse it,
compare its addresses with the 16 that belong and check every lastmod:
Exercise · Medium · Python
Repair a 30-address sitemap
Spice Box's shop software writes a sitemap of every address it knows, with the time of each nightly build as every lastmod and a priority on every entry. CRAWL in sitemap.py lists the 30 addresses, as a crawl of https://spicebox.example found them: the address as the software writes it (some are relative), its status code, the canonical its page names (None for the page itself), whether it carries noindex, and the day its content last changed.
Write build() so that it returns the text of a correct sitemap:
- only the addresses that answer 200, name themselves as canonical and carry no noindex;
- every
loca full address onhttps://spicebox.example, with&escaped; - every
lastmodthe day the content last changed, never the build time; - no
priorityand nochangefreq, which Google ignores; - the XML declaration and the
urlsetelement of the sitemaps.org namespace.
The sample tests parse your sitemap, compare its addresses with the ones that belong in it, and check every lastmod.
Starter code · sitemap.py
import xml.etree.ElementTree as ET
SITE = "https://spicebox.example"
NS = "http://www.sitemaps.org/schemas/sitemap/0.9"
BUILD_TIME = "2026-10-18T02:00:00+05:30" # when the shop software last rebuilt the site
# address as written, status, canonical (None: the page itself), noindex, the day its content last changed
CRAWL = [
("/", 200, None, False, "2026-10-02"),
("/spices/", 200, None, False, "2026-09-30"),
("/spices/kashmiri-chilli", 200, None, False, "2026-10-04"),
("/spices/black-pepper", 200, None, False, "2026-09-21"),
("/spices/cardamom", 200, None, False, "2026-08-17"),
("/spices/turmeric", 200, None, False, "2026-09-02"),
("/spices/cinnamon", 200, None, False, "2026-07-29"),
("/spices/cloves", 200, None, False, "2026-09-12"),
("https://spicebox.example/spices/saffron", 200, None, False, "2026-10-06"),
("https://spicebox.example/blends/garam-masala", 200, None, False, "2026-09-25"),
("/blends/", 200, None, False, "2026-09-25"),
("/blends/chai-masala", 200, None, False, "2026-08-08"),
("/gift-boxes?occasion=diwali&size=large", 200, None, False, "2026-10-01"),
("/gift-boxes?occasion=wedding&size=small", 200, None, False, "2026-09-18"),
("/about", 200, None, False, "2026-03-15"),
("/shipping", 200, None, False, "2026-06-30"),
("/spices/?sort=price", 200, "https://spicebox.example/spices/", False, "2026-09-30"),
("/spices/?sort=name", 200, "https://spicebox.example/spices/", False, "2026-09-30"),
("/spices/kashmiri-chilli?utm_source=newsletter", 200, "https://spicebox.example/spices/kashmiri-chilli", False, "2026-10-04"),
("/spices/kashmiri-chilli?ref=homepage", 200, "https://spicebox.example/spices/kashmiri-chilli", False, "2026-10-04"),
("http://spicebox.example/spices/turmeric", 301, None, False, "2026-09-02"),
("https://www.spicebox.example/blends/", 301, None, False, "2026-09-25"),
("/old-offers", 301, None, False, "2026-01-20"),
("/spices/long-pepper", 404, None, False, "2026-04-02"),
("/cart", 200, None, True, "2026-05-11"),
("/checkout", 200, None, True, "2026-05-11"),
("/account", 200, None, True, "2026-05-11"),
("/search?q=chilli", 200, None, True, "2026-02-09"),
("/thank-you", 200, None, True, "2026-02-09"),
("/print/spices/cardamom", 200, "https://spicebox.example/spices/cardamom", False, "2026-08-17"),
]
def build():
"""Return the text of the sitemap: an XML declaration and a urlset of the addresses that belong in it."""
lines = ['<?xml version="1.0" encoding="UTF-8"?>', f'<urlset xmlns="{NS}">']
for address, status, canonical, noindex, changed in CRAWL:
lines.append(f" <url><loc>{address}</loc><lastmod>{BUILD_TIME}</lastmod><priority>0.8</priority></url>")
lines.append("</urlset>")
return "\n".join(lines)The sample tests · test_sitemap.py
import xml.etree.ElementTree as ET
from sitemap import BUILD_TIME, CRAWL, NS, SITE, build
BELONGS = [
"/", "/spices/", "/spices/kashmiri-chilli", "/spices/black-pepper", "/spices/cardamom", "/spices/turmeric",
"/spices/cinnamon", "/spices/cloves", "/spices/saffron", "/blends/garam-masala", "/blends/", "/blends/chai-masala",
"/gift-boxes?occasion=diwali&size=large", "/gift-boxes?occasion=wedding&size=small", "/about", "/shipping",
]
def entries():
"""{loc: the url element} of the sitemap that build() returns."""
root = ET.fromstring(build().encode("utf-8"))
assert root.tag == f"{{{NS}}}urlset"
return {url.find(f"{{{NS}}}loc").text: url for url in root}
def test_well_formed_with_declaration_and_namespace():
"""starts with the XML declaration and parses as a urlset of the sitemaps.org namespace"""
assert build().startswith('<?xml version="1.0" encoding="UTF-8"?>')
assert entries()
def test_only_the_addresses_that_belong():
"""keeps the 16 canonical, indexable pages that answer 200, as full addresses, and drops the other 14"""
assert set(entries()) == {SITE + path for path in BELONGS}
def test_ampersands_escaped():
"""writes the & of the gift-box addresses as &"""
text = build()
assert "occasion=diwali&size=large" in text
assert "occasion=diwali&size=" not in text
def test_lastmod_is_the_content_change():
"""gives every page the day its content last changed, never the build time"""
changed = {(SITE + a if a.startswith("/") else a): day for a, _, _, _, day in CRAWL}
for loc, url in entries().items():
assert url.find(f"{{{NS}}}lastmod").text == changed[loc], loc
assert BUILD_TIME not in build()
def test_no_priority_or_changefreq():
"""leaves out priority and changefreq, which Google ignores"""
for url in entries().values():
assert url.find(f"{{{NS}}}priority") is None
assert url.find(f"{{{NS}}}changefreq") is NoneA hint
Turn a relative address into a full one first (SITE + path), then apply the three checks to it: status 200, a canonical that is None or the address itself, and no noindex. xml.etree.ElementTree escapes & for you when it writes the text of an element.
Results of the sample tests
| Test | Result | Details |
|---|
What your code printed
The sample tests run on this device, in your browser (Pyodide): nothing is sent to mysmartcopilot.com. The first run downloads Python (about 13.5 MB), which is kept for the next runs. A check in your browser is feedback for you, not proof that the code is right for every input.
Last, a round of practice on the six ideas of sitemaps, starting where your answers have been weakest:
Practice that adapts to you
Practise this lesson
18 questions about 6 ideas of this lesson. Each answer is explained at once, and what you get right is kept in this browser.
Interview questions
Warm-up (fresher to 2–5 years): does submitting a sitemap guarantee that Google crawls and indexes its pages?
No. A sitemap is a hint that helps Google discover pages, most of all on large sites, new sites with few links and sites with many images or videos, and Google says it doesn’t guarantee that every item in it will be crawled and indexed. Submitting it does not even guarantee that Google downloads it, although the Sitemaps report shows whether it was fetched and processed. What a sitemap can do is list exactly the canonical, indexable pages a site wants shown, with lastmod dates that tell Google which pages changed; if those dates prove accurate, Google can use them. Whether each page is then indexed depends on the page, which the Page indexing report shows when it is filtered to submitted pages.
Two more questions follow, about a sitemap rebuilt every night and a sitemap full of pages that are not indexed. Make the first decision of each before you open it:
Checkpoint
Your first decision
Model answers open with any pass: ₹79 for 1 week ($2.99), one payment that never renews.
-
Mid to senior
An online shop with 2 million URLs regenerates its sitemaps every night and sets every lastmod to the build time. Its new products take weeks to be crawled. What is wrong, and what would you build instead?
-
Fresher to mid
The Page indexing report, filtered to submitted pages, shows 40 % of a site's sitemap URLs as not indexed: redirects, pages marked noindex and duplicates with another canonical. How do you clean up the sitemap, and what do you change upstream so that it stays clean?
Key takeaways
- A sitemap helps Google discover pages, most on large, new or media-rich sites, and guarantees neither crawling nor indexing.
- An XML sitemap is a UTF-8
urlsetofurlentries, each with a fulllocfirst, and its values are entity-escaped, so&becomes&. - It lists only the addresses the site wants shown: canonical, answering 200 and without noindex.
- Each lastmod is the date of the page’s last significant change, never the build time, and Google ignores priority and changefreq.
- Past 50,000 URLs or 50 MB the URLs are split into several sitemaps listed in an index, which cannot list another index.
- A sitemap is submitted in Search Console, with the API or with a robots.txt line, and checked before submission for the errors the Sitemaps report names.
Terms in this lesson
- lastmod
- The date, or date and time, that a sitemap entry gives for its page's last significant change: the main content, the structured data or the links. Google uses it only when a site's values prove accurate.
- sitemap index
- A file that lists several sitemaps, each with its address and an optional lastmod, so that a site with more than 50,000 URLs or 50 MB can still submit one file. It cannot list another index.
- XML sitemap
- A file, often sitemap.xml at a site's root, that lists the addresses of the pages a site wants Google to know about, each with the date it last changed. It helps discovery but guarantees neither crawling nor indexing.
References
- Learn about sitemaps (Google Search Central)
- Build and submit a sitemap (Google Search Central)
- Manage your sitemaps with a sitemap index file (Google Search Central)
- Image sitemaps (Google Search Central)
- Sitemaps XML format (sitemaps.org)
- XML Schema for Sitemap files (sitemap.xsd) (sitemaps.org)
- Manage your sitemaps using the Sitemaps report (Google Search Console Help)
- Page indexing report (Google Search Console Help)
- How to specify a canonical URL with rel="canonical" and other methods (Google Search Central)
- xml.etree.ElementTree: the ElementTree XML API (Python Software Foundation)
Related tools
Report a problem with this lesson
Kept only in this browser. Your Learn progress