How Google Search works: crawl, index, serve

Pune Bakes’ clicks from Google fell by 45 % in one week, and the owner’s first thought was that Google had punished her. It had not: a theme update on her own site had told Google to drop every product page.

To find a cause like that in minutes, you need a picture of what Google does with a page. Google Search works in three stages, and a page can stop at any of them. By the end of this page you can name the three stages and what stops a page at each, read the reasons Search Console gives for a page that is missing, recognise the kinds of result a results page mixes, AI Overviews and AI Mode among them, and tell a drop that Google caused from one that you caused yourself.

Every page passes three stages, and none is guaranteed

Google describes the work in three stages, and not every page passes through all of them (how Search works):

• Crawlingcrawling: The first stage of Google Search: Googlebot downloads a page whose address Google has found, from a link or a sitemap, and renders it by running its JavaScript.: Google “downloads text, images, and videos from pages it found on the internet” with automated programs; the main one is GooglebotGooglebot: Google's main crawler, the program that downloads pages for Google Search. It reads a site's robots.txt rules before it fetches pages.. Before it can download a page, Google must learn that the address exists, from links on pages it already knows or from a sitemap. While crawling, it also renders the page: it runs the page’s JavaScript in a recent version of Chrome, as a browser would.
• Indexingindexing: The second stage of Google Search: Google analyses a crawled page's text, images and tags, groups duplicates, picks a canonical and stores the result in its index.: Google analyses the text, the images and the videos of the page, along with key tags such as the title and alt attributes, and “stores the information in the Google index”, a large database. It also groups pages that are near-duplicates and picks one of each group as the canonicalcanonical URL: The address Google picks as the most representative of a group of duplicate pages; it is the one that may be shown in results. A site's preferred canonical is a hint, not a rule., the one that may be shown in results.
• Servingserving: The third stage of Google Search: for each search, Google chooses the indexed pages it judges most relevant and useful, with factors such as the searcher's location, language and device.: when someone searches, Google picks the results it judges most relevant and of the highest quality for that query, from hundreds of factors that include the searcher’s location, language and device.

None of the three is promised. Google states that it doesn’t guarantee to crawl, index or serve a page, even one that follows its guidance, and that it “doesn’t accept payment to crawl a site more frequently, or rank it higher”. Step through one new page of the bakery and notice where each common failure would stop it:

The page passed every stage, but it could have stopped at three places: a robots.txt block at crawling, a noindex rule at indexing, and a better answer from another site at serving. Google’s own rules for a site are gathered in Search Essentials, in three parts: technical requirements, spam policies and key best practices. Most of this track’s SEO lessons work on one of the three.

Search Console names the stage where a page stopped

The technical requirements are the bare minimum a page needs to be eligible: Googlebot is not blocked (the page is neither behind a login nor disallowed in robots.txt), the page works, answering “200 (success)” rather than an error, and its content is in a format Google can index and breaks no spam policy (technical requirements). When Google does not index a page, for one of these failures or for another cause, Search Console’s Page indexing reportPage indexing report: The Search Console report that lists a site's pages Google has and has not indexed, with a reason for each page left out, such as a robots.txt block, a noindex rule or a duplicate. gives the reason (Page indexing report). Each reason belongs to a stage:

A page that is indexed but never shown for the words you expect has stopped at serving; Search Console’s Performance report, which shows the queries a page appears for, is where that shows. One reason sits outside the table: “Indexed, though blocked by robots.txt” means Google indexed an address it could not crawl, often because other pages link to it. That is why Google says robots.txt “is not a mechanism for keeping a web page out of Google” (robots.txt).

A results page mixes many kinds of result

Search for “eggless cake near me” and the page that comes back is not ten blue links. Google’s gallery of visual elements names the common kinds of result, and three more parts sit beside them:

Two rows matter for money. For local listings, Google says “There’s no way to request or pay for a better local ranking on Google” (local ranking). For ads, Google sells only the ad space, so “no one can buy better placement in organic Search results” (Ads on Search).

The last row is the newest. AI Overviews give a quick answer to a complex question with links to explore; AI Mode suits questions that need comparisons and reasoning. Both may use query fan-outquery fan-out: How AI Overviews and AI Mode may build an answer: by running several related searches across subtopics and data sources, then linking to the pages they draw on., “issuing multiple related searches across subtopics and data sources”, to build an answer (AI features). To be shown as a supporting link, a page only needs to be indexed and eligible to be shown with a snippet: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” Their traffic is counted in Search Console’s Performance report under the Web search type.

Ranking systems change, and Google publishes the big changes

Serving is done by many ranking systems working together. Google’s guide to its ranking systems names them: BERT, MUM and neural matching for understanding language, link analysis systems and PageRank, a reviews system, freshness systems, a site diversity system, original content systems and spam detection systems, among others. Some are retired: the helpful content system, announced in 2022, became part of the core ranking systems in 2024.

Google changes these systems all the time. Several times a year it makes a broad change to them, a core updatecore update: A broad change to Google's ranking systems, made several times a year and announced on the Search Status Dashboard with the days its rollout began and ended. It does not target particular sites., and announces it; smaller core updates run continuously and are not announced (core updates). Spam updates, notable improvements to the systems that detect spam, are announced the same way (spam updates). Each announced update appears on the Search Status Dashboard with the day its rollout began and the day it ended, and Google’s advice on reading one follows those dates:

A core update does not single out sites: Google says core updates “don’t target specific sites or individual web pages”. When a page falls in one, other pages have been judged more relevant or more useful, which is why the advice is to wait, compare and review the content honestly rather than rush a quick fix.

A drop in clicks has more than one cause

Google’s guide to traffic drops lists six causes: an algorithmic update, technical issues, security issues, spam issues, seasonality and changing interests, and site moves. Only the first is a change Google makes to its ranking. The program below lines up 35 weeks of the bakery’s web search clicks with the dashboard’s dates and the bakery’s own change log, which sits at the top of the program, and for each week that fell by 15 % or more it names the likeliest cause. The clicks are invented; the update dates are the dashboard’s. Run it, then press Edit, move the theme update in SITE_CHANGES to 2026-03-25, while the March core update was rolling out, and run it again to see which cause the program now names first:

"""Find the weeks when organic clicks fell, and line each one up with Google's updates and the site's own changes.

weekly_clicks.csv: 35 weeks of a fictional bakery's clicks from Google web search, and the same week a year before
(invented data, shaped like a Search Console export). ranking_updates.json: Google's announced ranking updates from
the Search Status Dashboard. SITE_CHANGES below: what the bakery changed on its site, and when; move a day and run
the program again to see the verdicts follow the dates.
A week counts as a drop when clicks fell 15 % or more from the week before. Discover updates are left out, because
this export counts web search only.
"""
import csv
import json
from datetime import date, timedelta

DROP = -0.15
# The bakery's own log of changes to punebakes.example: the day, and what changed.
SITE_CHANGES = [
    ("2026-02-02", "New photos on the ten best-selling product pages"),
    ("2026-07-08", "A theme update added a noindex tag to every product page"),
    ("2026-07-15", "The noindex tag removed, and recrawls requested with URL Inspection"),
]

with open("weekly_clicks.csv", encoding="utf-8") as f:
    weeks = [(date.fromisoformat(r["week"]), int(r["clicks"]), int(r["same_week_last_year"])) for r in csv.DictReader(f)]
with open("ranking_updates.json", encoding="utf-8") as f:
    updates = [u for u in json.load(f)["updates"] if "Discover" not in u["name"]]

clicks_in = {start: clicks for start, clicks, _ in weeks}


def monday(day):
    return day - timedelta(days=day.weekday())


print(f"{len(weeks)} weeks of web search clicks. Weeks that fell 15 % or more from the week before:\n")
for (start0, clicks0, last0), (start, clicks, last) in zip(weeks, weeks[1:]):
    change = clicks / clicks0 - 1
    if change > DROP:
        continue
    before_year = last / last0 - 1
    print(f"week of {start}: {clicks0:,} -> {clicks:,} clicks ({change:+.0%}); a year before: {before_year:+.0%}")
    window = (start0, start + timedelta(days=6))  # this week and the week before
    mine = [(day, what) for day, what in SITE_CHANGES if window[0] <= date.fromisoformat(day) <= window[1]]
    rolling = [u for u in updates if date.fromisoformat(u["begin"]) <= window[1] and date.fromisoformat(u["end"]) >= window[0]]
    for day, what in mine:
        print(f"  site change on {day}: {what}")
    for u in rolling:
        print(f"  rolling out: {u['name']}, {u['begin']} to {u['end']}")
    if mine:
        print("  -> look at your own change first")
    elif before_year <= DROP:
        print("  -> seasonal: the same week fell as much a year before")
    elif rolling:
        last_one = max(rolling, key=lambda u: u["end"])
        begin, end = date.fromisoformat(last_one["begin"]), date.fromisoformat(last_one["end"])
        before = monday(begin) - timedelta(days=7)  # the full week before the rollout began
        after = monday(end + timedelta(days=7)) + timedelta(days=7)  # a full week, a week after it ended
        b, a = clicks_in[before], clicks_in.get(after)
        print(f"  -> an update was rolling out: wait a full week after it ends on {end}, then compare")
        if a is not None:
            print(f"     week of {before}: {b:,} clicks; week of {after}: {a:,} clicks ({a / b - 1:+.0%})")
    else:
        print("  -> no update, no change, not seasonal: check Search Console's reports for errors")
    print()

35 weeks of web search clicks. Weeks that fell 15 % or more from the week before:

week of 2026-02-16: 1,420 -> 1,080 clicks (-24%); a year before: -25%
  -> seasonal: the same week fell as much a year before

week of 2026-03-30: 1,120 -> 905 clicks (-19%); a year before: +1%
  rolling out: March 2026 spam update, 2026-03-24 to 2026-03-25
  rolling out: March 2026 core update, 2026-03-27 to 2026-04-08
  -> an update was rolling out: wait a full week after it ends on 2026-04-08, then compare
     week of 2026-03-16: 1,110 clicks; week of 2026-04-20: 910 clicks (-18%)

week of 2026-05-11: 1,210 -> 930 clicks (-23%); a year before: -19%
  -> seasonal: the same week fell as much a year before

week of 2026-07-13: 1,020 -> 560 clicks (-45%); a year before: +0%
  site change on 2026-07-08: A theme update added a noindex tag to every product page
  site change on 2026-07-15: The noindex tag removed, and recrawls requested with URL Inspection
  -> look at your own change first

week,clicks,same_week_last_year
2026-01-05,980,900
2026-01-12,1000,910
2026-01-19,1010,905
2026-01-26,1030,930
2026-02-02,1060,960
2026-02-09,1420,1300
2026-02-16,1080,980
2026-02-23,1070,970
2026-03-02,1090,990
2026-03-09,1100,1000
2026-03-16,1110,1010
2026-03-23,1120,1005
2026-03-30,905,1015
2026-04-06,890,1020
2026-04-13,900,1030
2026-04-20,910,1035
2026-04-27,920,1040
2026-05-04,1210,1300
2026-05-11,930,1050
2026-05-18,940,1045
2026-05-25,975,1050
2026-06-01,990,1060
2026-06-08,1000,1065
2026-06-15,1005,1070
2026-06-22,1010,1080
2026-06-29,1015,1085
2026-07-06,1020,1090
2026-07-13,560,1095
2026-07-20,640,1100
2026-07-27,860,1110
2026-08-03,990,1105
2026-08-10,1015,1110
2026-08-17,1030,1120
2026-08-24,1035,1125
2026-08-31,1040,1130

{
  "source": "Google Search Status Dashboard, its published history of ranking updates (https://status.search.google.com/incidents.json): the name and the first and last day of each rollout, in UTC",
  "updates": [
    {
      "name": "December 2025 core update",
      "begin": "2025-12-11",
      "end": "2025-12-29"
    },
    {
      "name": "February 2026 Discover update",
      "begin": "2026-02-05",
      "end": "2026-02-27"
    },
    {
      "name": "March 2026 spam update",
      "begin": "2026-03-24",
      "end": "2026-03-25"
    },
    {
      "name": "March 2026 core update",
      "begin": "2026-03-27",
      "end": "2026-04-08"
    },
    {
      "name": "May 2026 core update",
      "begin": "2026-05-21",
      "end": "2026-06-02"
    },
    {
      "name": "June 2026 spam update",
      "begin": "2026-06-24",
      "end": "2026-06-26"
    },
    {
      "name": "August 2026 spam update",
      "begin": "2026-08-18",
      "end": "2026-08-21"
    },
    {
      "name": "September 2026 spam update",
      "begin": "2026-09-24",
      "end": "2026-10-08"
    }
  ]
}

Four weeks fell, for three different reasons. Two drops, of 24 % and 23 %, came the week after a peak in demand, and the same weeks fell by 25 % and 19 % a year before: seasonality, which nothing on the site can fix. One fall, of 19 %, came while a core update and a spam update were rolling out; a full week after the core update ended, clicks were still 18 % below the week before it began, so the update did cost the bakery traffic, and the answer is a review of its pages, not a panic. The largest fall, 45 %, came the week after the bakery’s own theme update added noindex to every product page, with no update running: the cause was the bakery’s own, and the fix was in its hands.

Practise: find where each page stopped

The exercise gives you eight things that went wrong on eight sites, from a noindex tag on a working page to a fall during a core update. For each, choose the stage where the page stopped and the first check you would make; the sample tests check both:

STAGES = {"crawl", "index", "serve"}
CHECKS = {
    "robots-txt",  # read the site's robots.txt rules for that URL
    "server-logs",  # read the server's logs and errors for Googlebot's requests
    "internal-links",  # find which pages link to it, and whether the sitemap lists it
    "url-inspection",  # inspect the URL in Search Console: indexing allowed, canonical, last crawl
    "content-review",  # ask honestly whether the page is useful and different from others
    "performance-report",  # see which queries the page gets impressions for, if any
    "status-dashboard",  # see when Google's updates began and ended
}

SCENARIOS = [
    "noindex-200", "blocked-but-linked", "server-5xx", "discovered-not-indexed",
    "duplicate-canonical", "crawled-not-indexed", "indexed-no-impressions", "drop-during-update",
]


def diagnose():
    """Return {scenario id: (stage, first check)} for the eight scenarios in the prompt."""
    # For example "noindex-200": ("index", "url-inspection").
    return {}

from diagnosis import CHECKS, SCENARIOS, STAGES, diagnose


def test_every_scenario_is_answered():
    """answers all eight scenarios with a known stage and a known check"""
    answers = diagnose()
    assert set(answers) == set(SCENARIOS)
    for scenario, (stage, check) in answers.items():
        assert stage in STAGES, scenario
        assert check in CHECKS, scenario


def test_stages():
    """puts each problem at the stage where it stops the page"""
    stage = {s: a[0] for s, a in diagnose().items()}
    assert {s for s in SCENARIOS if stage[s] == "crawl"} == {"blocked-but-linked", "server-5xx", "discovered-not-indexed"}
    assert {s for s in SCENARIOS if stage[s] == "index"} == {"noindex-200", "duplicate-canonical", "crawled-not-indexed"}
    assert {s for s in SCENARIOS if stage[s] == "serve"} == {"indexed-no-impressions", "drop-during-update"}


def test_first_checks_for_crawling():
    """starts a crawling problem where the crawler was stopped or never sent"""
    check = {s: a[1] for s, a in diagnose().items()}
    assert check["blocked-but-linked"] == "robots-txt"
    assert check["server-5xx"] == "server-logs"
    assert check["discovered-not-indexed"] == "internal-links"


def test_first_checks_for_indexing_and_serving():
    """reads what Google saw before changing the page, and the dates before blaming the page"""
    check = {s: a[1] for s, a in diagnose().items()}
    assert check["noindex-200"] == "url-inspection"
    assert check["duplicate-canonical"] == "url-inspection"
    assert check["crawled-not-indexed"] == "content-review"
    assert check["indexed-no-impressions"] == "performance-report"
    assert check["drop-during-update"] == "status-dashboard"

Finally, practise the five ideas of the page, weakest first, as your answers show it:

Interview questions

Warm-up (fresher to 2–5 years): how are crawling, indexing and serving different?

Crawling is Google downloading a page with Googlebot, after it has learned the address from a link or a sitemap; it includes rendering the page’s JavaScript. Indexing is Google analysing what it downloaded, the text, images, title and other tags, grouping duplicates, choosing a canonical and storing the result in its index. Serving is choosing, for each search, the indexed pages that are most relevant and useful for that searcher. A page can stop at any stage: a robots.txt block stops crawling, a noindex rule stops indexing, and a better answer elsewhere stops serving. Google guarantees none of the three, and no payment buys any of them.

Two more questions follow, about a page that never appeared and a fall in clicks. Make the first decision of each before you open it:

Key takeaways

• Google Search works in three stages, crawling, indexing and serving, and Google guarantees none of them and sells none of them.
• Search Console’s Page indexing report gives a reason for each page left out, and each reason belongs to a stage; robots.txt controls crawling, not whether an address can appear.
• A results page mixes text and rich results, images, videos, exploration features, Business Profile listings, labelled ads and AI features, and AI Overviews and AI Mode need no more than an indexed page that may show a snippet.
• Ranking systems change all the time; core and spam updates are announced on the Search Status Dashboard, and an update is judged a full week after it ends against the week before it began.
• A drop in clicks has several possible causes, so line it up with Google’s updates, your own changes and last year’s same weeks before you change anything.