Redirects, status codes and soft 404s

Pune Bakes moved its menu twice in a year, and an old link to its mango cake now passes three redirects before it reaches the page. Every visit through that link costs four requests, and one wrong rule in the redirect map sends visitors round in a loop that never reaches a page at all.

A redirect is one of the answers a server gives, and each answer’s status code tells Google something different about an address: here, moved, gone or busy. By the end of this page you can choose between 301, 308, 302 and 307, clean a redirect map so that every old address reaches its page in one hop, answer pages that are gone with 404 or 410 instead of a soft 404soft 404: A page that says something is missing, or holds almost nothing, while its status code says 200. Search Console reports it as a soft 404; a missing page should answer 404 or 410., and pause a site for maintenance without hurting how Google crawls it.

Every answer starts with a status code, and Google acts on it

An HTTP response starts with a three-digit status codestatus code: The three-digit number at the start of every HTTP response that says what happened: 2xx success, 3xx a redirect, 4xx a client error such as 404, 5xx a server error. Google's crawlers act on it before they read the page., whose first digit gives its class: 2xx success, 3xx redirection, 4xx a client error, 5xx a server error (RFC 9110, section 15). Google’s crawlers read the code before they read the page, and they treat the common codes like this (HTTP status codes and Google Search):

Whatever a redirecting or failing answer contains is ignored, so a helpful message in a 404 page is for visitors only. Step through five of the bakery’s addresses, each answering with a different family of code:

The codes differ in meaning even where Google handles two of them alike, as it does 301 and 308, so choose the one that says what happened: other search engines and other readers of the web use them too.

A permanent redirect moves the address Google shows; a temporary one keeps it

Google treats a redirect as a signal about which address should be canonical, and the difference between the two kinds is the address people see in results (Redirects and Google Search). A permanent redirectpermanent redirect: A redirect with status 301 or 308, or an instant meta refresh, that says the content has moved for good. Google treats the target as canonical and shows it in results. makes Google show the target; a temporary redirecttemporary redirect: A redirect with status 302, 303 or 307, or a delayed meta refresh, that says the move will be undone. Google follows it but usually keeps showing the source address in results. makes it keep showing the source, although the target may still be indexed if other signals point at it. Google lists the methods in the order in which it interprets them most reliably:

Google’s advice is to “use a permanent server-side redirect whenever possible”. A JavaScript redirect works only if Google renders the page successfully, so it comes after both of those, and a link with a note comes last of all. The pairs differ in one more way, which matters for forms: when a browser follows a 301 or a 302 it may turn a POST into a GET, losing the form’s data, while 307 and 308 forbid any change of method (RFC 9110, sections 15.4.2 to 15.4.9). Google treats 308 like 301 and 307 like 302.

After a permanent move Google keeps the old address for a while as an alternate name of the new one, and may still show it when a search hints that people trust the old address more; such names fade away without any action. On Apache the simplest rule is Redirect permanent "/one" "https://punebakes.example/two"; on Nginx it is return 301 inside a location.

Chains and loops: every old address reaches its page in one hop

A redirect chainredirect chain: Two or more redirects in a row, A to B to C. Google's crawlers follow up to 10 hops, but every hop costs visitors a request, so old addresses should point straight at the final page. grows a rule at a time: each move adds one, and nobody goes back to point the older rules at the newest address. Google’s crawlers follow up to 10 hops by default (HTTP status codes and Google Search), but every hop costs each visitor one more request, and Google’s site-move guide asks for no chains at all, redirecting “to the final destination directly”, or at least “ideally no more than 3 and fewer than 5” hops (How to move a site). Follow the old link to the mango cake:

A redirect loopredirect loop: Redirects that lead back to an address already passed, A to B to A, so no page is ever reached. Browsers stop with an error, and Search Console reports a redirect error. is worse: it never reaches a page at all, and the Page indexing report lists such addresses under “Redirect error”, with a chain too long, a loop, an address that grew past the maximum length and a bad or empty address as its causes (Page indexing report). Here is the bakery’s redirect map as the server holds it, last year’s move and this month’s renames together:

old,new
/menu/mango-cake,/cakes/mango-cake
/menu/choco-truffle,/cakes/choco-truffle
/menu/red-velvet,/cakes/red-velvet-heart
/menu/rasmalai,/cakes/rasmalai-cake
/menu/black-forest,/cakes/black-forest
/menu/pineapple,/cakes/pineapple-cake
/menu/butterscotch,/cakes/butterscotch-cake
/menu/coffee-walnut,/cakes/coffee-walnut
/menu/plum-cake,/cakes/plum-cake
/menu/blueberry,/cakes/blueberry-cheesecake
/menu/fruit-cake,/cakes/fruit-cake
/menu/,/cakes/
/menu/cupcakes,/cupcakes/
/menu/brownies,/brownies/
/menu/cookies,/cookies/
/menu/breads,/breads/
/cakes/mango-cake,/cakes/mango-mousse
/cakes/choco-truffle,/cakes/eggless-truffle
/about-us,/about/
/contact-us,/contact/
/offers,/offers/diwali
/offers/diwali,/offers
/order-online,/order/
/gift-hampers,/hampers/

The program follows each old address to where the map finally sends it, using the 20 addresses that answer 200, and writes the rules again for Apache and Nginx:

"""Clean Pune Bakes' redirect map, then write it as Apache and Nginx rules.

redirect_map.csv holds every redirect the bakery's server has today: those of last year's move from /menu/ to the
new folders, and this month's renames. LIVE lists the addresses that answer 200. The program follows
each old address to where the map finally sends it, points it there in one hop, and reports chains, loops and redirects
to pages that no longer exist. It writes redirects.conf (Apache) and redirects-nginx.conf (Nginx), with full https
targets, so that the same rules on the http host also send old links to the final page in one hop.
"""

import csv

SITE = "https://punebakes.example"

LIVE = {  # the addresses that answer 200, from a crawl of the site
    "/cakes/", "/cakes/mango-mousse", "/cakes/eggless-truffle", "/cakes/red-velvet-heart", "/cakes/rasmalai-cake",
    "/cakes/black-forest", "/cakes/pineapple-cake", "/cakes/butterscotch-cake", "/cakes/coffee-walnut",
    "/cakes/blueberry-cheesecake", "/cakes/fruit-cake", "/cupcakes/", "/brownies/", "/cookies/", "/breads/",
    "/about/", "/contact/", "/offers/", "/order/", "/hampers/",
}

rules = {row["old"]: row["new"] for row in csv.DictReader(open("redirect_map.csv", encoding="utf-8"))}


def follow(address):
    """The addresses the map sends `address` through, ending where it stops or where a loop begins again."""
    path = [address]
    while path[-1] in rules:
        nxt = rules[path[-1]]
        if nxt in path:
            return path + [nxt], True
        path.append(nxt)
    return path, False


chains, loops, gone, final = [], {}, [], {}
for old in rules:
    path, loop = follow(old)
    if loop:
        loops.setdefault(frozenset(path), path)  # each loop once, from the first address that enters it
    elif path[-1] not in LIVE:
        gone.append(path)
    else:
        final[old] = path[-1]
        if len(path) > 2:
            chains.append(path)

print(f"{len(rules)} redirects read, {len(LIVE)} live pages")
print(f"{len(chains)} chains, each now one hop:")
for path in chains:
    print("  " + " -> ".join(path))
print(f"{len(loops)} loop, left out until someone decides where it should go:")
for path in loops.values():
    print("  " + " -> ".join(path))
print(f"{len(gone)} redirect to a page that no longer exists, written as 410 Gone:")
for path in gone:
    print("  " + " -> ".join(path) + " (not a live page)")

apache = [f"RedirectMatch 301 ^{old}$ {SITE}{new}" for old, new in final.items()]
apache += [f"RedirectMatch 410 ^{path[0]}$" for path in gone]
nginx = [f"location = {old} {{ return 301 {SITE}{new}; }}" for old, new in final.items()]
nginx += [f"location = {path[0]} {{ return 410; }}" for path in gone]
open("redirects.conf", "w", encoding="utf-8").write("\n".join(apache) + "\n")
open("redirects-nginx.conf", "w", encoding="utf-8").write("\n".join(nginx) + "\n")
print(f"\nredirects.conf, {len(apache)} Apache rules; the first two and the last:")
print("\n".join("  " + line for line in apache[:2] + apache[-1:]))
print(f"redirects-nginx.conf, {len(nginx)} Nginx locations; the first two and the last:")
print("\n".join("  " + line for line in nginx[:2] + nginx[-1:]))

24 redirects read, 20 live pages
2 chains, each now one hop:
  /menu/mango-cake -> /cakes/mango-cake -> /cakes/mango-mousse
  /menu/choco-truffle -> /cakes/choco-truffle -> /cakes/eggless-truffle
1 loop, left out until someone decides where it should go:
  /offers -> /offers/diwali -> /offers
1 redirect to a page that no longer exists, written as 410 Gone:
  /menu/plum-cake -> /cakes/plum-cake (not a live page)

redirects.conf, 22 Apache rules; the first two and the last:
  RedirectMatch 301 ^/menu/mango-cake$ https://punebakes.example/cakes/mango-mousse
  RedirectMatch 301 ^/menu/choco-truffle$ https://punebakes.example/cakes/eggless-truffle
  RedirectMatch 410 ^/menu/plum-cake$
redirects-nginx.conf, 22 Nginx locations; the first two and the last:
  location = /menu/mango-cake { return 301 https://punebakes.example/cakes/mango-mousse; }
  location = /menu/choco-truffle { return 301 https://punebakes.example/cakes/eggless-truffle; }
  location = /menu/plum-cake { return 410; }

It found what grows in every real map: 2 chains, now each one hop; 1 loop between /offers and /offers/diwali, which a person must settle, since only someone at the bakery knows where offers should go; and 1 redirect to the plum cake, which no longer exists, now a 410. Two choices in its rules are deliberate. Apache’s Redirect matches the start of a path, so Redirect 301 /menu/ /cakes/ would also catch every address below /menu/; RedirectMatch with ^ and $ matches one address exactly (mod_alias), as Nginx’s location = does (ngx_http_rewrite_module). And the targets are full https addresses, so the same rules placed on the http host send old links there in one hop. Apache also uses the first rule that matches, so these exact rules go before any rule that sends all of http to https. This session writes the rules and lets Apache check them:

# A minimal Apache httpd configuration whose one job is to load the rules redirect_map.py wrote, so that
# httpd -t can check them. The module paths are those of macOS; on Linux they are usually under
# /usr/lib/apache2/modules/. On a real server the same lines go in the site's VirtualHost, or in .htaccess.
ServerName punebakes.example
LoadModule mpm_prefork_module /usr/libexec/apache2/mod_mpm_prefork.so
LoadModule unixd_module /usr/libexec/apache2/mod_unixd.so
LoadModule authz_core_module /usr/libexec/apache2/mod_authz_core.so
LoadModule alias_module /usr/libexec/apache2/mod_alias.so
DocumentRoot "."
ErrorLog /dev/stderr

Include redirects.conf

"""Writes the bakery's redirect rules and checks them with Apache httpd, the way a terminal shows it.

Each command is printed after "$ ", then what it printed: standard output and standard error together. python3 runs
redirect_map.py, which writes redirects.conf; httpd -t then reads apache-test.conf, which includes it, and reports
whether every rule is valid. httpd -t starts no server and opens no port. To replay the session, type its commands in
a terminal on a computer with Python 3 and Apache httpd 2.4 installed.
"""

import shlex
import subprocess
import sys

COMMANDS = [
    "python3 redirect_map.py > summary.txt",
    "grep -c RedirectMatch redirects.conf",
    "httpd -v | head -n 1",
    'httpd -t -d "$PWD" -f apache-test.conf',
]

failed = 0
for command in COMMANDS:
    print(f"$ {command}")
    run = command.replace("python3 ", f"{shlex.quote(sys.executable)} ", 1)  # the pinned Python of this run
    done = subprocess.run(run, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
                          env={"PATH": "/usr/sbin:/usr/bin:/bin"})
    print(done.stdout, end="")
    failed += done.returncode != 0
sys.exit(1 if failed else 0)

$ python3 redirect_map.py > summary.txt
$ grep -c RedirectMatch redirects.conf
22
$ httpd -v | head -n 1
Server version: Apache/2.4.67 (Unix)
$ httpd -t -d "$PWD" -f apache-test.conf
Syntax OK

Apache 2.4.67 accepts all 22. The Nginx file is checked the same way with nginx -t on a server that runs Nginx.

Pages that are gone answer 404 or 410, never 200

A page that no longer exists should say so in its status. 404 Not Found says that the server has no current page at the address, without saying whether for now or for good; 410 Gone says that the page is no longer available and that this “is likely to be permanent” (RFC 9110, sections 15.5.5 and 15.5.11). For Google the two work alike: an address that answers any 4xx but 429 is taken out of the index over time, and a 404 is “a strong signal not to crawl that URL again” (crawl budget). Search Console lists such addresses as Not found (404), and Google notes that a 404 is not necessarily a problem when a page was removed without a replacement; the ones to fix are those the site still links to or lists in its sitemap, and moved content gets a 301 instead (Page indexing report).

The trouble starts when the status and the page disagree. A page that answers 200 while it shows a “not found” message, or very little, is what Google calls a soft 404, and such pages “will continue to be crawled, and waste your budget”. Two more cases count the same way: old addresses sent in bulk to one unrelated page, such as the home page, which “might be treated as a soft 404 error” (How to move a site), and a filter that finds nothing, for which Google asks for a 404 (faceted navigation). The program below checks six of the bakery’s answers for exactly that disagreement:

"""Does each answer's status code say what its page says? Six of Pune Bakes' addresses, checked one by one.

Each row is what the bakery's server sends: the status, the Location of a redirect, and the start of the page's main
text. A page that tells the visitor something is missing while its status says 200 is what Google calls a soft 404,
and a removed product redirected to the home page may be treated as one.
"""

ANSWERS = [  # address, status, Location (redirects only), the start of the main text
    ("/cakes/blueberry-cheesecake", 200, None, "Blueberry cheesecake, baked on a biscuit base"),
    ("/cakes/plum-cake", 404, None, "We couldn't find that cake. Here are this week's bakes"),
    ("/cakes/strawberry-tart", 410, None, "The strawberry tart is no longer made"),
    ("/cakes/black-forest-mini", 200, None, "Sorry, this cake is no longer available"),
    ("/cakes/?flavour=durian", 200, None, "No cakes match these filters"),
    ("/cakes/vanilla-sponge", 301, "/", None),
]
SAYS_MISSING = ("no longer available", "couldn't find", "no cakes match", "not found")


def verdict(address, status, location, text):
    missing = text is not None and any(p in text.lower() for p in SAYS_MISSING)
    if status == 200 and missing:
        return "soft 404: the page says missing, the status says 200"
    if status in (301, 308) and location == "/":
        return "home page for a removed product: may count as a soft 404"
    if status in (404, 410):
        return f"{status}: the page is gone and the status says so"
    return f"{status}: the status matches the page"


for address, status, location, text in ANSWERS:
    print(f"{address}\n  {verdict(address, status, location, text)}")

/cakes/blueberry-cheesecake
  200: the status matches the page
/cakes/plum-cake
  404: the page is gone and the status says so
/cakes/strawberry-tart
  410: the page is gone and the status says so
/cakes/black-forest-mini
  soft 404: the page says missing, the status says 200
/cakes/?flavour=durian
  soft 404: the page says missing, the status says 200
/cakes/vanilla-sponge
  home page for a removed product: may count as a soft 404

Three answers are right, three are not. The black forest mini page and the durian filter both tell the visitor that nothing is there while their status says 200, and the vanilla sponge, which the bakery stopped making, sends its visitors to the home page. Each needs the status that matches its page: 404 or 410 for the two missing pages, and for the sponge either a 301 to a close replacement, such as another sponge cake, or a 410.

A short pause answers 503, and robots.txt keeps answering 200

Sometimes a site must stop for a few hours: a new database, a payment provider’s outage, a sale that overloads the server. Google’s first advice is to keep the site online with less in it, for example by turning off the cart, which leaves search visibility unchanged; a whole site switched off should be a matter of “a few days at most” (Temporarily pause or disable a website). For a pause of a day or two Google suggests an informational error page with status 503 and a Retry-AfterRetry-After: An HTTP response header, such as Retry-After: 3600, that tells a client how many seconds to wait, or until which date and time, before trying again. It goes with 503 and 429 answers. header, either a date or a number of seconds, as in Retry-After: 3600 (RFC 9110, section 10.2.3). And it gives one warning in plain words: never answer 503 for robots.txt itself, because that “blocks all crawling”.

For a longer closure Google suggests a stand-in page that answers 200; what it advises against is a 403, 404 or 410 or a noindex, each of which removes the site’s addresses from Search, and a robots.txt block of the whole site, which may remove them too. A server that is only overloaded by one client may answer 429 Too Many Requests instead, which RFC 6585 defines for rate limiting, and which Google’s crawlers treat like a 5xx: they slow down. Here is the bakery’s maintenance switch for Apache, which answers 503 for everything except robots.txt and the maintenance page itself:

# Pune Bakes' maintenance switch for Apache httpd 2.4: while it is in the configuration, every address
# answers 503 Service Unavailable with Retry-After: 3600 (try again in an hour), except robots.txt and the
# maintenance page itself, which keep answering 200. Remove the block when the work is done.
# The module paths are those of macOS; on Linux they are usually under /usr/lib/apache2/modules/.
ServerName punebakes.example
LoadModule mpm_prefork_module /usr/libexec/apache2/mod_mpm_prefork.so
LoadModule unixd_module /usr/libexec/apache2/mod_unixd.so
LoadModule authz_core_module /usr/libexec/apache2/mod_authz_core.so
LoadModule rewrite_module /usr/libexec/apache2/mod_rewrite.so
LoadModule headers_module /usr/libexec/apache2/mod_headers.so
DocumentRoot "."
ErrorLog /dev/stderr

RewriteEngine On
RewriteCond %{REQUEST_URI} !=/robots.txt
RewriteCond %{REQUEST_URI} !=/maintenance.html
RewriteRule ^ - [R=503,L]
ErrorDocument 503 /maintenance.html
Header always set Retry-After "3600" "expr=%{REQUEST_STATUS} == 503"

"""Checks the bakery's maintenance configuration with Apache httpd, the way a terminal shows it.

Each command is printed after "$ ", then what it printed: standard output and standard error together. httpd -t reads
the configuration and reports whether its syntax is valid; it starts no server and opens no port. To replay the
session, type its commands in a terminal on a computer with Apache httpd 2.4 installed.
"""

import subprocess
import sys

COMMANDS = [
    "httpd -v | head -n 1",
    'httpd -t -d "$PWD" -f maintenance.conf',
]

failed = 0
for command in COMMANDS:
    print(f"$ {command}")
    done = subprocess.run(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
                          env={"PATH": "/usr/sbin:/usr/bin:/bin"})
    print(done.stdout, end="")
    failed += done.returncode != 0
sys.exit(1 if failed else 0)

$ httpd -v | head -n 1
Server version: Apache/2.4.67 (Unix)
$ httpd -t -d "$PWD" -f maintenance.conf
Syntax OK

[R=503] works because a RewriteRule may return any valid status, and with one outside the 300s it serves no target and stops (RewriteRule flags); Header always with expr= adds Retry-After only to the 503 answers, error documents included (mod_headers). The check proves that Apache accepts the configuration; once it is live, Google’s guide suggests checking the answers with a tool such as curl, curl -I and the address, before the work starts. Step through the pause from Googlebot’s side:

Every code of this lesson answers one question: what happened to the content at this address? Answer it, and two more, for an address on your own site:

In short: 301, or 308 when forms post to it, for a move that stays; 302 or 307 for a move that ends; 404 or 410 for a page that is gone; and 503 with Retry-After for a pause of a day or two.

Practise: clean up 40 legacy redirects

Spice Box moved twice, and its server still holds the redirects of both moves. Write clean() so that each of the 40 old addresses reaches its live page in one hop or answers 410, and give the status code for six special cases; the sample tests follow every address hop by hop:

"""Spice Box's redirects after two moves: /shop/*.html became /products/*, then /products/* became /spices/*."""

RULES = {  # every redirect the server has today: old address -> where it points
    "/shop/kashmiri-chilli.html": "/products/kashmiri-chilli",
    "/shop/black-pepper.html": "/products/black-pepper",
    "/shop/cardamom.html": "/products/cardamom",
    "/shop/turmeric.html": "/products/turmeric",
    "/shop/cinnamon.html": "/products/cinnamon",
    "/shop/cloves.html": "/products/cloves",
    "/shop/saffron.html": "/products/saffron",
    "/shop/cumin.html": "/products/cumin",
    "/shop/coriander.html": "/products/coriander",
    "/shop/fenugreek.html": "/products/fenugreek",
    "/shop/mustard-seeds.html": "/products/mustard-seeds",
    "/shop/nutmeg.html": "/products/nutmeg",
    "/shop/star-anise.html": "/products/star-anise",
    "/shop/bay-leaves.html": "/products/bay-leaves",
    "/shop/fennel.html": "/products/fennel",
    "/shop/ajwain.html": "/products/ajwain",
    "/products/kashmiri-chilli": "/spices/kashmiri-chilli",
    "/products/black-pepper": "/spices/black-pepper",
    "/products/cardamom": "/spices/cardamom",
    "/products/turmeric": "/spices/turmeric",
    "/products/cinnamon": "/spices/cinnamon",
    "/products/cloves": "/spices/cloves",
    "/products/saffron": "/spices/saffron",
    "/products/cumin": "/spices/cumin",
    "/products/coriander": "/spices/coriander",
    "/products/fenugreek": "/spices/fenugreek",
    "/products/mustard-seeds": "/spices/mustard-seeds",
    "/products/nutmeg": "/spices/nutmeg",
    "/products/star-anise": "/spices/star-anise",
    "/products/bay-leaves": "/spices/bay-leaves",
    "/products/fennel": "/spices/fennel",
    "/products/ajwain": "/spices/ajwain",
    "/shop/index.html": "/products/",
    "/products/": "/spices/",
    "/shop/garam-masala-blend.html": "/products/garam-masala",
    "/products/garam-masala": "/blends/garam-masala",
    "/products/chai-masala": "/blends/chai-masala",
    "/shop/long-pepper.html": "/products/long-pepper",
    "/products/holi-box": "/offers/holi-box",
    "/offers/holi-box": "/products/holi-box",
}
LIVE = {  # the addresses that answer 200
    "/spices/kashmiri-chilli", "/spices/black-pepper", "/spices/cardamom",
    "/spices/turmeric", "/spices/cinnamon", "/spices/cloves",
    "/spices/saffron", "/spices/cumin", "/spices/coriander",
    "/spices/fenugreek", "/spices/mustard-seeds", "/spices/nutmeg",
    "/spices/star-anise", "/spices/bay-leaves", "/spices/fennel",
    "/spices/ajwain", "/spices/", "/blends/garam-masala",
    "/blends/chai-masala",
}
LEGACY = [  # 40 old addresses that links, bookmarks and other sites still use
    "/shop/kashmiri-chilli.html", "/shop/black-pepper.html", "/shop/cardamom.html",
    "/shop/turmeric.html", "/shop/cinnamon.html", "/shop/cloves.html",
    "/shop/saffron.html", "/shop/cumin.html", "/shop/coriander.html",
    "/shop/fenugreek.html", "/shop/mustard-seeds.html", "/shop/nutmeg.html",
    "/shop/star-anise.html", "/shop/bay-leaves.html", "/shop/fennel.html",
    "/shop/ajwain.html", "/products/kashmiri-chilli", "/products/black-pepper",
    "/products/cardamom", "/products/turmeric", "/products/cinnamon",
    "/products/cloves", "/products/saffron", "/products/cumin",
    "/products/coriander", "/products/fenugreek", "/products/mustard-seeds",
    "/products/nutmeg", "/products/star-anise", "/products/bay-leaves",
    "/products/fennel", "/products/ajwain", "/shop/index.html",
    "/shop/garam-masala-blend.html", "/products/garam-masala", "/products/chai-masala",
    "/shop/long-pepper.html", "/products/long-pepper", "/products/holi-box",
    "/offers/holi-box",
]


def clean(rules, live, legacy):
    """Return {old address: (status, target)} for every legacy address: (301, a live address) or (410, None)."""
    return {old: (301, rules[old]) for old in legacy if old in rules}


# The status code each case needs: one number per case.
SPECIAL = {
    "checkout": None,  # the checkout is down for one hour of maintenance
    "discontinued": None,  # a spice blend that will not come back, with no replacement
    "form-moved": None,  # the order form moved for good, and other sites' forms still POST to the old address
    "festival-offer": None,  # /offers/ points to /offers/holi/ for the seven days of the festival
    "too-many-requests": None,  # a crawler sends 50 requests a second to the search page
    "http-home": None,  # the http:// address of the home page, now on https
}

from redirect_plan import LEGACY, LIVE, RULES, SPECIAL, clean


def plan():
    return clean(RULES, LIVE, LEGACY)


def test_every_legacy_address_planned():
    """plans all 40 legacy addresses, each as a 301 or a 410"""
    result = plan()
    assert set(result) == set(LEGACY)
    assert {status for status, _ in result.values()} <= {301, 410}


def test_redirects_end_on_live_pages_in_one_hop():
    """every 301 goes straight to an address that answers 200, so no redirect leads to another"""
    for old, (status, target) in plan().items():
        if status == 301:
            assert target in LIVE, f"{old} goes to {target}, which is not a live page"


def test_chains_collapsed():
    """the old shop's pages skip /products/ and go straight to /spices/ and /blends/"""
    result = plan()
    assert result["/shop/cardamom.html"] == (301, "/spices/cardamom")
    assert result["/shop/garam-masala-blend.html"] == (301, "/blends/garam-masala")
    assert result["/shop/index.html"] == (301, "/spices/")


def test_dead_ends_and_loops_answer_410():
    """long pepper is gone and the Holi box's two addresses point at each other: all four answer 410"""
    result = plan()
    for old in ["/shop/long-pepper.html", "/products/long-pepper", "/products/holi-box", "/offers/holi-box"]:
        assert result[old] == (410, None), old


def test_special_cases():
    """the status codes of the six special cases"""
    assert SPECIAL["checkout"] == 503
    assert SPECIAL["discontinued"] in (404, 410)
    assert SPECIAL["form-moved"] == 308
    assert SPECIAL["festival-offer"] in (302, 307)
    assert SPECIAL["too-many-requests"] == 429
    assert SPECIAL["http-home"] in (301, 308)

Then practise the five ideas of this lesson, starting where your answers have been weakest:

Interview questions

Warm-up (fresher to 2–5 years): how does Google treat a 301 redirect differently from a 302?

Both send visitors and Googlebot to another address, and Google follows both. The difference is what Google shows: a 301, like a 308, is permanent, so Google takes it as a strong signal that the target should be canonical and shows the target in results; a 302, like a 307, is temporary, a weak signal, so Google usually keeps showing the source address. Use a 301 when a page has moved for good, after a rename or a move to HTTPS, and a 302 when it will come back, such as an offer page pointing to this week’s sale. If forms post to the address, use 308 or 307, because browsers may turn a POST into a GET when they follow a 301 or a 302. Whichever you use, point old addresses straight at the final page, not through a chain.

Two more questions follow, about a migration that redirected every old product to the home page and a week of maintenance. Make the first decision of each before you open it:

Key takeaways

• A status code tells Google what happened before it reads the page: 2xx goes on to indexing, 3xx is followed, 4xx but 429 means the content is missing, and 429 or 5xx slows crawling.
• A permanent redirect, 301 or 308, makes Google show the target; a temporary one, 302 or 307, keeps the source; 308 and 307 keep a form’s POST a POST.
• Every old address should reach its final page in one hop: clean chains and loops from the redirect map and write exact rules with full targets.
• A page that is gone answers 404 or 410; a “not found” page with status 200, an empty filter and old pages sent in bulk to the home page are soft 404s.
• A pause of a day or two answers 503 with Retry-After, robots.txt keeps answering 200, and a longer closure keeps a page that answers 200.