Set up a site for search and Search Console

Pune Bakes’ site answers at four addresses: http:// and https://, each with and without www. The bakery added only https://punebakes.example/ to Search Console, so its reports left out everything Google had to say about the other three, and a fault on any of them would never have shown.

Setting a site up for search starts by closing blind spots like that one. By the end of this page you can choose the right kind of Search Console property and verify it with a DNS record, send every address to one preferred address, read what URL InspectionURL Inspection tool: The Search Console tool that shows what Google has stored about one URL, from its last crawl to the canonical Google chose, and can test the live page now or ask for it to be indexed. says Google has stored about a page, decide whether the site takes part in AI Overviews and AI Mode, and add the site to Bing.

A Domain property sees every address; a URL-prefix property sees one

Search Console reports on a property, and there are two kinds (Add a website property). A Domain propertyDomain property: A Search Console property for a whole domain, such as example.com: it covers every protocol and every subdomain (www, m, shop and the rest), and it is verified through a DNS record., such as punebakes.example, includes every subdomain (www, m, staging and the rest) and every protocol, http and https alike. A URL-prefix propertyURL-prefix property: A Search Console property for the addresses that start with one prefix, protocol and host included, such as https://example.com/blog/. Each protocol and each subdomain needs its own. includes only the addresses that start with its prefix, protocol and host included: https://punebakes.example/ leaves out http://punebakes.example/ and https://www.punebakes.example/.

The program below applies the help page’s two rules to seven addresses and four properties. Run it, then compare the last two properties, which differ by one character:

"""Which Search Console property sees which address of the bakery's sites?

A Domain property covers every protocol and every subdomain of its domain. A URL-prefix property covers only the
addresses that start with its prefix, protocol and host included (Search Console Help, "Add a website property").
"""

from urllib.parse import urlsplit

PROPERTIES = [
    ("Domain", "punebakes.example"),
    ("URL-prefix", "https://punebakes.example/"),
    ("URL-prefix", "https://punebakes.example/recipes"),
    ("URL-prefix", "https://punebakes.example/recipes/"),
]

ADDRESSES = [
    "https://punebakes.example/cakes/",
    "http://punebakes.example/cakes/",
    "https://www.punebakes.example/cakes/",
    "https://staging.punebakes.example/cakes/",
    "https://punebakes.example/recipes/mango-mousse",
    "https://punebakes.example/recipes-archive/old-menu",
    "https://punebakes-shop.example/cakes/",
]


def covers(kind, name, address):
    """True if a property of this kind and name covers the address."""
    if kind == "Domain":
        host = urlsplit(address).hostname
        return host == name or host.endswith("." + name)
    return address.startswith(name)


for kind, name in PROPERTIES:
    seen = [a for a in ADDRESSES if covers(kind, name, a)]
    print(f"{kind} {name}")
    print(f"  covers {len(seen)} of {len(ADDRESSES)} addresses:")
    for address in seen:
        print(f"  {address}")

Domain punebakes.example
  covers 6 of 7 addresses:
  https://punebakes.example/cakes/
  http://punebakes.example/cakes/
  https://www.punebakes.example/cakes/
  https://staging.punebakes.example/cakes/
  https://punebakes.example/recipes/mango-mousse
  https://punebakes.example/recipes-archive/old-menu
URL-prefix https://punebakes.example/
  covers 3 of 7 addresses:
  https://punebakes.example/cakes/
  https://punebakes.example/recipes/mango-mousse
  https://punebakes.example/recipes-archive/old-menu
URL-prefix https://punebakes.example/recipes
  covers 2 of 7 addresses:
  https://punebakes.example/recipes/mango-mousse
  https://punebakes.example/recipes-archive/old-menu
URL-prefix https://punebakes.example/recipes/
  covers 1 of 7 addresses:
  https://punebakes.example/recipes/mango-mousse

The Domain property covers six of the seven; only punebakes-shop.example, a different domain, is outside it. The URL-prefix property for the site’s root covers three, because the other protocol and the other hosts do not start with its prefix. Without its final /, the recipes property also covers /recipes-archive/old-menu, a page that is not a recipe at all, which is why the help page says to end a path’s prefix with /.

A DNS TXT record proves the domain is yours

Search Console shows a site’s data only to people who prove they control it (Verify your site ownership). A Domain property is verified through DNS, unless the site runs on a Google product such as Blogger: usually with a TXT record whose value starts with google-site-verification=. You add it with the company that runs your domain’s DNS, on the domain itself, with the host or name field left empty or set to @. Step through the bakery’s verification:

Most failures come from where the record ends up. The program below reads three versions of the bakery’s DNS zone, written in the zone-file format that DNS servers load, and looks for the token where Search Console looks for it: on punebakes.example. itself. Run it, then press Edit and fix the first zone by changing www to @:

"""Is the Search Console token where a Domain property looks for it?

Three versions of the bakery's DNS zone, written in the zone-file format DNS servers read (RFC 1035, section 5).
A name that does not end with a dot is relative: the zone's $ORIGIN is added to it. The program finds every TXT
record, prints its full name, and says whether the domain itself holds the token. Nothing is sent over the network.
"""

TOKEN = "google-site-verification=Xq3tPb7Rk2LmN9vW4yZc8Hs1Dj6Fg0Ae5UoIiKpQwEr"  # made up for this lesson

ZONES = {
    "1. record on www": """
$ORIGIN punebakes.example.
@                  3600 IN A     192.0.2.10
@                  3600 IN TXT   "v=spf1 mx -all"
www                3600 IN TXT   "google-site-verification=Xq3tPb7Rk2LmN9vW4yZc8Hs1Dj6Fg0Ae5UoIiKpQwEr"
""",
    "2. name without a final dot": """
$ORIGIN punebakes.example.
@                  3600 IN A     192.0.2.10
@                  3600 IN TXT   "v=spf1 mx -all"
punebakes.example  3600 IN TXT   "google-site-verification=Xq3tPb7Rk2LmN9vW4yZc8Hs1Dj6Fg0Ae5UoIiKpQwEr"
""",
    "3. record on @": """
$ORIGIN punebakes.example.
@                  3600 IN A     192.0.2.10
@                  3600 IN TXT   "v=spf1 mx -all"
@                  3600 IN TXT   "google-site-verification=Xq3tPb7Rk2LmN9vW4yZc8Hs1Dj6Fg0Ae5UoIiKpQwEr"
""",
}


def txt_records(zone):
    """Yield (full name, text) for each TXT record of a zone."""
    origin = ""
    for line in zone.strip().splitlines():
        fields = line.split(None, 4)
        if fields[0] == "$ORIGIN":
            origin = fields[1]
            continue
        name, _ttl, _class, rtype, data = fields
        if name == "@":
            name = origin
        elif not name.endswith("."):
            name = f"{name}.{origin}"
        if rtype == "TXT":
            yield name, data.strip('"')


for title, zone in ZONES.items():
    print(title)
    found = False
    for name, text in txt_records(zone):
        shown = text if len(text) <= 32 else text[:32] + "..."
        print(f"  {name}")
        print(f"    TXT {shown}")
        found = found or (name == "punebakes.example." and text == TOKEN)
    verdict = "verifies" if found else "fails: no token on punebakes.example."
    print(f"  Domain property: {verdict}")

1. record on www
  punebakes.example.
    TXT v=spf1 mx -all
  www.punebakes.example.
    TXT google-site-verification=Xq3tPb7...
  Domain property: fails: no token on punebakes.example.
2. name without a final dot
  punebakes.example.
    TXT v=spf1 mx -all
  punebakes.example.punebakes.example.
    TXT google-site-verification=Xq3tPb7...
  Domain property: fails: no token on punebakes.example.
3. record on @
  punebakes.example.
    TXT v=spf1 mx -all
  punebakes.example.
    TXT google-site-verification=Xq3tPb7...
  Domain property: verifies

Two of the three fail. In the first, the record sits on www.punebakes.example., which a Domain property never reads. The second is the subtle one: in a zone file, a name that does not end with a dot is relative, so the zone’s origin is added to it (RFC 1035, section 5.1), and punebakes.example became punebakes.example.punebakes.example.. A DNS panel that adds the domain to whatever you type does the same, so follow Google’s instruction and leave the host field empty or type @. The third zone adds the token beside the SPF record that was already there: a domain can hold many TXT records, so never delete one to make room.

Owners, users and the agency that left

Verifying makes you a verified owner. Owners add everyone else under Settings, then Users and permissions (Managing owners, users and permissions):

Give each person their own Google account at the lowest level that does the job, instead of one shared login. When an agency’s contract ends, remove its access and then delete its verification tokens: a removed owner whose token is still on the site can verify again.

One preferred address over HTTPS, and the rest redirect to it

Four addresses that answer with the same pages are four copies of each page. Google’s advice for a page reachable at several addresses is to pick one as preferred and redirect the others to it (Redirects and Google Search). It also prefers HTTPS pages over equivalent HTTP pages as canonical, except when the HTTPS page has an invalid certificate, insecure dependencies, a redirect to HTTP or a canonical tag that points to HTTP (duplicate URLs). Follow the bakery’s three redirects:

Each address reaches the preferred one in a single permanent redirect; the lessons on canonical URLs and on redirects later in this module take the details further. HTTPS is also part of page experience, where Google’s checklist asks “Are your pages served in a secure fashion?” (page experience). Search Console’s HTTPS report, shown for a Domain property or an HTTPS URL-prefix property, counts the site’s indexed HTTP and HTTPS URLs and gives a reason for each HTTP one. Five of the reasons each name a cause you can fix:

The report lists a sample of URLs, not all of them, so fix the cause each reason names rather than only the URLs it shows.

URL Inspection shows what Google stored, and a live test what it sees now

The bakery published a mango mousse page last week, and it has not appeared in Google. Before changing anything, inspect it: paste its address into the box at the top of Search Console. The URL Inspection tool answers from two places (URL Inspection tool), and mixing them up is the easiest way to misread it. Step through one inspection:

Two habits follow from the table. After you fix a page, the indexed result goes on showing the old state until Google crawls the page again, so check your fix with the live test and read the indexed result for what Google has. And Request indexing is for a handful of pages: there is a daily limit, and Google says that “requesting a recrawl multiple times for the same URL won’t get it crawled any faster” (Ask Google to recrawl). For many pages at once, a sitemap is the tool, as the sitemaps lesson of this module shows.

The live test’s crawler, Google-InspectionTool, obeys robots.txt rules written for its own name and for Googlebot (common crawlers), so a rule that blocks Googlebot fails the test as well. For the whole site, the Page indexing reportPage indexing report: The Search Console report that lists a site's pages Google has and has not indexed, with a reason for each page left out, such as a robots.txt block, a noindex rule or a duplicate. More in “How Google Search works: crawl, index, serve” gives the reason for each page Google left out, and Google advises that “you should not expect all URLs on your site to be indexed, only the canonical pages” (Page indexing report).

The Search generative AI control: Include, Exclude or Inherit

Pune Bakes’ recipes come from a cookbook author whose contract keeps them out of AI answers, while the rest of the site wants every visit it can get. Search Console holds one setting for this decision, the Search generative AI controlSearch generative AI control: A Search Console setting, Include, Exclude or Inherit, that decides whether a property's links and content may appear in AI Overviews, AI Mode and Discover's generative AI features., under Settings, then Search generative AI (Search generative AI control). Google has rolled it out to all websites, and it has three values:

• Include, the default of a property with no parent: the site’s links and content may appear in Search generative AI features.
• Exclude: links to the site and its content stop appearing in AI Overviews, AI Mode and the generative AI features of Discover, generally within a few days, and the site gets no traffic or impressions from them. Google expects to update the list of features over time.
• Inherit, the default of a property that has a parent: the property follows its closest parent.

Step through the bakery’s properties as their owners set them:

Three limits keep the decision honest. Exclude is not a ranking or inclusion signal for the rest of Search, so the recipes keep their places in ordinary results. It does not affect AI training: Google points to Google-Extended, a robots.txt token, for that. And other sites’ content still appears in those features, so an excluded recipe can be replaced by a competitor’s. To keep only part of a page’s text out of AI Overviews and AI Mode, the page-level snippet rules are the finer tool, and the lesson on noindex and snippet controls covers them.

Bing Webmaster Tools and IndexNow cover other engines

Google is one search engine of several. Bing Webmaster Tools can import sites that are already verified in Search Console, with their sitemaps, and verifies them automatically, up to 100 sites at a time (Bing Webmaster Blog), so adding Bing takes minutes once Search Console is set up.

Bing also takes part in IndexNowIndexNow: A protocol through which a site tells participating search engines, such as Bing, which of its addresses were added, updated or deleted. Google is not among the participants., a protocol through which a site tells search engines which of its addresses were added, updated or deleted (IndexNow documentation). The site picks a key of 8 to 128 letters, digits or dashes, publishes it in a text file named after the key at its root, and sends up to 10,000 addresses in one POST request. The participating engines share what they receive; the published list names Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot, and not Google (participating engines), which finds pages through links and sitemaps. Check a request before you send it:

{
  "host": "punebakes.example",
  "key": "5f1c8e3a9b2d4f6a",
  "keyLocation": "https://punebakes.example/5f1c8e3a9b2d4f6a.txt",
  "urlList": [
    "https://punebakes.example/cakes/mango-mousse",
    "https://punebakes.example/cakes/eggless-truffle",
    "https://punebakes.example/offers/"
  ]
}

"""Check an IndexNow request before it is sent, against the rules of the IndexNow documentation.

payload.json is the bakery's list of changed pages. A second payload, written below, has three mistakes. The
program prints the problems, or the request it would send. It sends nothing.
"""

import json
import re
from urllib.parse import urlsplit

MAX_URLS = 10_000  # URLs in one POST
KEY = re.compile(r"[A-Za-z0-9-]{8,128}")  # 8 to 128 letters, digits or dashes


def problems(p):
    """The reasons an engine would refuse this request, or ignore some of its URLs."""
    found = []
    if not KEY.fullmatch(p["key"]):
        found.append(f'key "{p["key"]}": 8 to 128 letters, digits or dashes')
    key_file = p.get("keyLocation", f"https://{p['host']}/{p['key']}.txt")
    key_url = urlsplit(key_file)
    if key_url.hostname != p["host"]:
        found.append(f"key file is not on {p['host']}")
    scope = key_url.path.rsplit("/", 1)[0] + "/"
    if len(p["urlList"]) > MAX_URLS:
        found.append(f"{len(p['urlList'])} URLs: at most {MAX_URLS:,} a request")
    for url in p["urlList"]:
        u = urlsplit(url)
        if u.hostname != p["host"]:
            found.append(f"{url}: not on {p['host']} (422)")
        elif not u.path.startswith(scope):
            found.append(f"{url}: outside the key file's folder {scope}")
    return found


good = json.load(open("payload.json"))
bad = {
    "host": "punebakes.example",
    "key": "pb_key",
    "keyLocation": "https://punebakes.example/keys/pb_key.txt",
    "urlList": [
        "https://www.punebakes.example/cakes/",
        "https://punebakes.example/offers/",
    ],
}

for name, payload in (("payload.json", good), ("second payload", bad)):
    print(name)
    found = problems(payload)
    for problem in found:
        print(f"  problem: {problem}")
    if not found:
        print("  POST /indexnow")
        print("  Content-Type: application/json; charset=utf-8")
        print(f"  {len(payload['urlList'])} URLs, all on {payload['host']}")
        print(f"  the engine then fetches {payload['keyLocation']}")
        print("  a 200 answer means received, not crawled or indexed")

payload.json
  POST /indexnow
  Content-Type: application/json; charset=utf-8
  3 URLs, all on punebakes.example
  the engine then fetches https://punebakes.example/5f1c8e3a9b2d4f6a.txt
  a 200 answer means received, not crawled or indexed
second payload
  problem: key "pb_key": 8 to 128 letters, digits or dashes
  problem: https://www.punebakes.example/cakes/: not on punebakes.example (422)
  problem: https://punebakes.example/offers/: outside the key file's folder /keys/

The bakery’s own request passes. The second fails three of the documentation’s rules: its key is too short and holds an underscore, its www address is not on the request’s host, which earns a 422 answer, and a key file in /keys/ vouches only for addresses under /keys/. Even a request that passes promises nothing: a 200 answer means the engine received the list, not that it will crawl or index any of it.

Practise: set the bakery up

The exercise gives you the owner’s notes: four addresses, one preferred, a section to report on its own, a licence to respect and a staging copy to keep private. Choose the properties, the redirects, the AI settings and the protection; the sample tests check each against this lesson:

ADDRESSES = [
    "http://punebakes.example",
    "http://www.punebakes.example",
    "https://punebakes.example",
    "https://www.punebakes.example",
]
PREFERRED = "https://punebakes.example"


def properties():
    """Return the properties to create, as (kind, name) pairs: kind "domain" or "url-prefix"."""
    return [("url-prefix", "https://punebakes.example/")]


def redirects():
    """Return {address: target} for every address that must redirect, each straight to its final address."""
    return {}


def ai_settings():
    """Return {property name: "include", "exclude" or "inherit"} for every property you create."""
    return {}


def staging_protection():
    """Return how the staging copy is kept out of Google: "robots-txt", "noindex" or "password"."""
    return "robots-txt"

from setup_plan import ADDRESSES, PREFERRED, ai_settings, properties, redirects, staging_protection


def test_properties():
    """creates one Domain property for the whole site and a URL-prefix property for the recipes, with its final /"""
    assert set(properties()) == {("domain", "punebakes.example"), ("url-prefix", "https://punebakes.example/recipes/")}


def test_redirects():
    """sends each of the three other addresses straight to the preferred one, and never the preferred one itself"""
    plan = redirects()
    assert set(plan) == set(ADDRESSES) - {PREFERRED}
    assert all(target == PREFERRED for target in plan.values())


def test_ai_settings():
    """includes the whole site and excludes only the recipe section, which covers everything beneath it"""
    settings = ai_settings()
    assert set(settings) == {name for _, name in properties()}
    assert settings["punebakes.example"] == "include"
    assert settings["https://punebakes.example/recipes/"] == "exclude"


def test_staging():
    """keeps the staging copy out with a password, which robots.txt and noindex cannot promise for a private copy"""
    assert staging_protection() == "password"

Then answer the lesson’s questions on its six ideas; they begin with whichever idea your answers show is weakest:

Interview questions

Warm-up (fresher to 2–5 years): how does a Domain property differ from a URL-prefix property?

A Domain property, entered as a bare domain such as example.com, includes every subdomain and every protocol: http and https, www, m, staging and any other subdomain. It is verified with a DNS record, and it is the property to look at first, because nothing on the domain escapes it. A URL-prefix property includes only the addresses that start with its prefix, protocol and host included, so https://example.com/ leaves out http://example.com/ and https://www.example.com/. It can also be verified with an HTML file or tag, Google Analytics or Tag Manager, and it suits a section reported or set on its own, such as https://example.com/blog/. A section’s prefix should end with a slash, or it also covers addresses such as /blog-archive/.

Two more questions follow, about an agency that left and a publisher that wants out of AI features. Make the first decision of each before you open it:

Key takeaways

• A Domain property covers every protocol and subdomain and is verified through DNS; a URL-prefix property covers only the addresses that start with its prefix, so a section’s prefix ends with /.
• The verification record goes on the domain itself, @, beside the TXT records already there, and stays there; owners add people at the lowest level that does the job and delete a departing owner’s tokens.
• One preferred address over HTTPS, with every other protocol and host sent to it in one permanent redirect, and the HTTPS report’s reason read for any HTTP page that is left.
• URL Inspection’s indexed result is what Google stored at the last crawl and the live test is the page now; neither promises indexing, and repeated requests do not speed crawling.
• Exclude in the Search generative AI control removes a property’s links and content from AI Overviews, AI Mode and Discover’s generative features, changes no other ranking and does not touch training; Inherit follows the closest parent.
• Bing can import a verified site from Search Console, and IndexNow tells Bing and the other participating engines, not Google, which addresses changed.