JavaScript SEO: rendering and its traps

Pune Bakes rebuilt its shop as a single-page app, and the HTML its server now sends for the mango mousse cake holds one word of text, no links, and a canonical that names the home page. Google sees the cake only after it has run the page’s script, and a crawler that runs no scripts never sees it at all.

Scripts are not a problem for Google Search in themselves, but they move a page’s content out of the server’s answer and into a second step. By the end of this page you can explain how Googlebot crawls, renders and indexes a JavaScript page, compare a page’s server HTML with its rendered HTML, fix the traps that hide content or send the wrong signal, make a single-page app answer 404 for a missing page, and choose how to build the pages that must be found.

Google crawls, then renders, then indexes

Google processes a JavaScript page in three phases: crawling, renderingrendering: Running a page's scripts and building the page they produce, as a browser does. Google renders pages that answer 200 in a step after crawling, with an evergreen Chromium, and indexes the rendered HTML. and indexing (JavaScript SEO basics). Googlebot reads robots.txt, fetches the page and reads the links in the HTML it received. Every page that answers 200 then waits in a render queue, unless a robots meta tag or header says not to index it, and a headless, evergreen version of Chromium runs its scripts; Google reads the links of the rendered HTML again and indexes what that HTML shows. Two cases change the path: for a page that answers 404 or another code than 200, “rendering might be skipped”, and when the HTML Google received says noindex, it “may skip rendering and JavaScript execution”. Step through the bakery’s app shell:

The renderer is not a person at a browser. It keeps no cookies or local storage between pages, cannot use WebSockets or WebRTC, and declines permission requests such as the camera (Fix Search-related JavaScript problems), and Google Search does not click or scroll (lazy-loaded content). Google has also removed what it calls “outdated information about JavaScript and accessibility” from its guides (documentation updates): its renderer handles modern scripts, and the cost of a JavaScript page is the wait and the second step, not a refusal.

Compare the server’s HTML with the rendered HTML

Google’s own guide says that server-side or pre-rendering is “still a great idea”, partly because “not all bots can run JavaScript” (JavaScript SEO basics), and other search engines “may choose to ignore JavaScript” (dynamic rendering). So what a page must say to every reader of its HTML, its title, description, canonical, robots rules, the Open Graph tags that link previews read, its main text and its links, belongs in the HTML the server sends. The bakery’s two versions of the cake page show the difference. The app shell, as the server sends it:

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Pune Bakes</title>
<link rel="canonical" href="https://punebakes.example/">
<script type="module" src="/app.js"></script>
</head>
<body>
<div id="app">Loading…</div>
</body>
</html>

and the script that draws the cake into it in the browser:

// The bakery's single-page app: it reads the address, fetches the cake and draws the page in the browser.
const slug = location.pathname.split("/").pop();
const response = await fetch(`/api/cakes/${slug}.json`);
const app = document.querySelector("#app");
if (!response.ok) {
  app.innerHTML = "<h1>Cake not found</h1><p>Try one of our other cakes.</p>";
} else {
  const cake = await response.json();
  document.title = `${cake.name} | Pune Bakes`;
  document.head.insertAdjacentHTML(
    "beforeend",
    `<meta name="description" content="${cake.summary}">` +
      `<link rel="canonical" href="https://punebakes.example/cakes/${slug}">` +
      `<meta property="og:title" content="${cake.name}">` +
      `<script type="application/ld+json">${JSON.stringify({ "@context": "https://schema.org", "@type": "Product", name: cake.name })}</script>`,
  );
  app.innerHTML =
    `<h1>${cake.name}</h1><p>${cake.description}</p>` +
    cake.related.map((r) => `<a href="/cakes/${r.slug}">${r.name}</a>`).join("") +
    `<a href="/cakes/">All cakes</a>` +
    `<button onclick="showReviews()">Show ${cake.reviews} reviews</button>`;
}

Rendering runs a page’s scripts and keeps the HTML they leave. The program below does that for app.js, with a small stand-in for the browser that supports exactly the calls the script makes, then reads the same nine items from the HTML the server sends and from the HTML after the script, for the shell and for cake.html, the version rendered on the server:

// What each crawler sees of two versions of Pune Bakes' mango mousse page: cake.html, which the server renders, and
// shell.html, an app shell whose script app.js fetches the cake and draws the page in the browser.
//
// Rendering runs a page's scripts and keeps the HTML they leave. This program does that for app.js with a small
// stand-in for the browser: a document that supports exactly the calls the script makes (document.title,
// head.insertAdjacentHTML, querySelector("#app").innerHTML) and a fetch that answers from mango-mousse.json. It then
// reads the same items from the HTML the server sends and from the HTML after the script, and compares them (the
// canonicals without their origin, https://punebakes.example).
import { readFileSync } from "node:fs";

const read = (file) => readFileSync(new URL(file, import.meta.url), "utf8");

async function render(shell, script, path) {
  let title = /<title>(.*?)<\/title>/s.exec(shell)[1];
  let head = "";
  let app = /<div id="app">(.*?)<\/div>/s.exec(shell)[1];
  const document = {
    get title() { return title; },
    set title(value) { title = value; },
    head: { insertAdjacentHTML: (where, html) => { head += html; } },
    querySelector: (selector) => (selector === "#app" ? { set innerHTML(html) { app = html; } } : null),
  };
  const fetch = async (url) => {
    const found = url === "/api/cakes/mango-mousse.json";
    return { ok: found, status: found ? 200 : 404, json: async () => JSON.parse(read("./mango-mousse.json")) };
  };
  const AsyncFunction = (async () => {}).constructor;
  await new AsyncFunction("document", "location", "fetch", script)(document, { pathname: path }, fetch);
  return shell
    .replace(/<title>.*?<\/title>/s, `<title>${title}</title>`)
    .replace("</head>", `${head}</head>`)
    .replace(/<div id="app">.*?<\/div>/s, `<div id="app">${app}</div>`);
}

function items(html) {
  const all = (re) => [...html.matchAll(re)].map((m) => m[1]);
  const body = /<body>(.*)<\/body>/s.exec(html)[1];
  return {
    title: all(/<title>(.*?)<\/title>/gs)[0],
    description: all(/<meta name="description" content="(.*?)">/g)[0] ?? "none",
    canonical: all(/<link rel="canonical" href="https:\/\/punebakes\.example(.*?)">/g).join(" and ") || "none",
    "og:title": all(/<meta property="og:title" content="(.*?)">/g)[0] ?? "none",
    h1: all(/<h1>(.*?)<\/h1>/g)[0] ?? "none",
    links: all(/<a href="(.*?)">/g).length,
    words: body.replace(/<[^>]*>/g, " ").split(/\s+/).filter(Boolean).length,
    "structured data": all(/"@type": ?"(\w+)"/g)[0] ?? "none",
    "behind a click": all(/<button[^>]*>(.*?)<\/button>/g)[0] ?? "nothing",
  };
}

const short = (value) => (String(value).split(" ").length > 7 ? `${String(value).split(" ").slice(0, 6).join(" ")} …` : value);

function compare(name, before, after) {
  console.log(name);
  for (const [key, value] of Object.entries(before)) {
    console.log(`  ${key}: ${short(value)}`);
    if (after[key] !== value) console.log(`    after the script: ${short(after[key])}`);
  }
}

const cake = read("./cake.html");
compare("cake.html, rendered on the server: the same before and after scripts", items(cake), items(cake));
const shell = read("./shell.html");
const before = items(shell);
const after = items(await render(shell, read("./app.js"), "/cakes/mango-mousse"));
console.log();
compare("shell.html with app.js: what the server sends, then what the script adds", before, after);
const unseen = Object.keys(before).filter((key) => key !== "behind a click" && before[key] !== after[key]);
const reviews = JSON.parse(read("./mango-mousse.json")).reviews;
console.log(`\nA crawler that runs no scripts gets none of the ${unseen.length} items the script adds or changes,`);
console.log(`and reads ${before.canonical} as the canonical. The ${reviews} reviews load only after a click,`);
console.log("so no crawler reads them.");

cake.html, rendered on the server: the same before and after scripts
  title: Mango mousse cake | Pune Bakes
  description: Eggless mango mousse cake made with …
  canonical: /cakes/mango-mousse
  og:title: Mango mousse cake
  h1: Mango mousse cake
  links: 3
  words: 31
  structured data: Product
  behind a click: nothing

shell.html with app.js: what the server sends, then what the script adds
  title: Pune Bakes
    after the script: Mango mousse cake | Pune Bakes
  description: none
    after the script: Eggless mango mousse cake made with …
  canonical: /
    after the script: / and /cakes/mango-mousse
  og:title: none
    after the script: Mango mousse cake
  h1: none
    after the script: Mango mousse cake
  links: 0
    after the script: 3
  words: 1
    after the script: 34
  structured data: none
    after the script: Product
  behind a click: nothing
    after the script: Show 48 reviews

A crawler that runs no scripts gets none of the 8 items the script adds or changes,
and reads / as the canonical. The 48 reviews load only after a click,
so no crawler reads them.

The server-rendered page says everything before any script runs. The shell says almost nothing: its title is “Pune Bakes”, it has one word of text and no links, and its canonical names the home page, so a crawler that runs no scripts reads a wrong canonical as well as missing content. After the script, the shell has all eight items, but now two canonicals, the home page’s and the cake’s, and the 48 reviews still wait behind a button that no crawler presses.

On your own site, compare the same two versions. The raw HTML is what the server sends, as a browser’s view source or a fetch shows it. The rendered HTML is in URL Inspection’s live test: “View tested page” shows the HTML, a screenshot, the JavaScript console output and the resources the page loaded; the screenshot exists only in a live test that succeeded (URL Inspection tool).

Five traps that hide content or send the wrong signal

Most JavaScript SEO problems are one of five traps, and each looks fine in a browser:

The first three come from Google’s guides to links, lazy loading and JavaScript (link best practices); the noindex trap was the subject of a section of the lesson on indexing controls; the canonical trap is the one the render comparison found. Google’s rule for it is that HTML is the best place for the canonical, and a script that sets it must use “the same value as the original HTML” (JavaScript SEO basics; specify a canonical).

Three more rules round it off. Shadow DOM is flattened in rendering, so only what reaches the rendered HTML, through a slot for example, can be indexed. Scripts and styles should carry a fingerprint of their content in their names, such as main.2bb85551.js, because Google caches them aggressively. And a paywall that sends the whole article and hides it with JavaScript “isn’t a reliable way to limit access to the content”: the article is in the HTML anyone downloads, so the server should send the full text only once the subscription is confirmed (Fix Search-related JavaScript problems), and the paywalled parts are marked up so that Google can tell them from cloaking (paywalled content).

A single-page app must still answer 404 for a missing page

A single-page app usually sends the same app shellapp shell: The near-empty HTML that a client-side app sends for every address: a title, a script and an empty element that the script fills in the browser. for every address, so its server answers 200 for every address, real or not, and the script draws “Cake not found” in the browser. Google asks for “a meaningful status code”, 404 for a page that does not exist (JavaScript SEO basics), and warns that apps which handle errors in the browser often report 200 instead, which can get error pages indexed and shown in results (Fix Search-related JavaScript problems). Step through a visit to a cake the bakery no longer makes:

The fix on the server is a lookup before the answer. The program below calls two versions of the bakery’s request handler for three addresses, the way the web server calls it for each request:

// Two versions of the request handler behind Pune Bakes' single-page app, called directly for three addresses, the
// way the web server calls it for each request. Before: every address gets the app shell with status 200, and the
// script then draws "Cake not found" in the browser. After: the handler looks the cake up first, and answers 404 with
// a small page of its own when the cake does not exist, as Google's JavaScript guide asks of a missing page.
const CAKES = new Set(["mango-mousse", "eggless-truffle", "rasmalai-cake"]); // the cakes the shop sells today
const SHELL = '<!doctype html><title>Pune Bakes</title><div id="app">Loading…</div>';
const NOT_FOUND = '<!doctype html><title>Cake not found | Pune Bakes</title><h1>Cake not found</h1>';

function before(path) {
  return { status: 200, body: SHELL };
}

function after(path) {
  const slug = /^\/cakes\/([a-z0-9-]+)$/.exec(path)?.[1];
  if (slug && !CAKES.has(slug)) return { status: 404, body: NOT_FOUND };
  return { status: 200, body: SHELL };
}

function shown(path, answer) {
  if (answer.status === 404) return "the not-found page";
  const known = CAKES.has(path.split("/").pop());
  return known ? "the script draws the cake" : 'the script draws "Cake not found": a soft 404';
}

const PATHS = ["/cakes/mango-mousse", "/cakes/blueberry-cheesecake", "/cakes/plum-pudding"];
for (const [name, handle] of [["Before: the app shell for every address", before], ["After: a lookup first", after]]) {
  console.log(name);
  for (const path of PATHS) {
    const answer = handle(path);
    console.log(`  ${path.padEnd(28)} ${answer.status}  ${shown(path, answer)}`);
  }
}

Before: the app shell for every address
  /cakes/mango-mousse          200  the script draws the cake
  /cakes/blueberry-cheesecake  200  the script draws "Cake not found": a soft 404
  /cakes/plum-pudding          200  the script draws "Cake not found": a soft 404
After: a lookup first
  /cakes/mango-mousse          200  the script draws the cake
  /cakes/blueberry-cheesecake  404  the not-found page
  /cakes/plum-pudding          404  the not-found page

Before, all three addresses answer 200 and two of them are soft 404s. After, the handler answers 404 for the two cakes the shop does not sell, and the mango mousse cake is unchanged. When only the script can tell whether a page exists, Google offers two ways out: the script sends the browser to an address that answers 404, such as /not-found, or it adds a robots meta tag with noindex to the error view.

Render on the server what must be found

Four ways of building a page come up in every choice, in web.dev’s words (Rendering on the Web):

• Server-side renderingserver-side rendering: Building a page's HTML on the server for each request, so that every visitor and crawler receives the content without having to run scripts. builds the page on the server for each request, so that the browser receives “HTML, rather than JavaScript”.
• Static generationstatic generation: Building the HTML of every address ahead of time, when the site is built, one file per address, so that the server only has to send files. builds the HTML of every address ahead of time, at build time, one file per address.
• Client-side renderingclient-side rendering: Building a page in the browser with JavaScript, from an app shell and data the script fetches. Only crawlers that run scripts see its content. builds the page “in a browser, using JavaScript to modify the DOM”, from an app shell.
• Hydrationhydration: Running scripts in the browser to make HTML that the server built interactive, without building the page again. runs scripts in the browser “to add application state and interactivity to server-rendered HTML”.

A fifth way, dynamic renderingdynamic rendering: Sending crawlers a server-rendered copy of a page and people its client-side version. Google calls it a workaround and not a recommended solution., sends crawlers a server-rendered copy and people the app; Google calls it “a workaround and not a recommended solution”, and recommends server-side rendering, static rendering or hydration instead (dynamic rendering). Two questions decide most cases. Answer them for a page of your own:

In short: build product, article and category pages ahead of time or on the server, so that every crawler gets their content, and add scripts on top for what people do on the page; keep client-side rendering for what is never searched for, such as an account’s dashboard.

Practise: compare five pages before and after their scripts

Spice Box has five pages, each as the HTML its server sends and as the HTML after its script ran. Write seo_view() with Python’s HTML parser, then list what only the scripts add and say which version Google can be expected to index; the sample tests run your code on all five:

"""Five Spice Box pages, each as the HTML its server sends (raw) and as the HTML after its script ran (rendered)."""

from html.parser import HTMLParser

SHELL = '<html><head><title>Spice Box</title><link rel="canonical" href="https://spicebox.example/"></head>' \
        '<body><div id="app"></div></body></html>'
PAGES = {
    "saffron": {  # rendered on the server: the script changes nothing
        "status": 200,
        "raw": '<html><head><title>Saffron threads | Spice Box</title>'
               '<meta name="description" content="Kashmiri saffron threads, one gram.">'
               '<link rel="stylesheet" href="/styles.2bb85551.css">'
               '<link rel="canonical" href="https://spicebox.example/spices/saffron"></head>'
               '<body><h1>Saffron threads</h1><a href="/spices/">Spices</a> <a href="/recipes/saffron-kheer">'
               'Saffron kheer</a></body></html>',
    },
    "cardamom": {  # an app shell: the script draws the whole page
        "status": 200,
        "raw": SHELL,
        "rendered": '<html><head><title>Green cardamom | Spice Box</title>'
                    '<link rel="canonical" href="https://spicebox.example/">'
                    '<meta name="description" content="Green cardamom pods from Idukki.">'
                    '<link rel="canonical" href="https://spicebox.example/spices/green-cardamom"></head>'
                    '<body><div id="app"><h1>Green cardamom</h1><a href="/spices/">Spices</a> '
                    '<a href="/recipes/masala-chai">Masala chai</a> <a href="/spices/black-cardamom">Black cardamom</a>'
                    '</div></body></html>',
    },
    "offers": {  # noindex in the HTML the server sends, removed by the script once the offers load
        "status": 200,
        "raw": '<html><head><title>Offers | Spice Box</title><meta name="robots" content="noindex"></head>'
               '<body><div id="app"></div></body></html>',
        "rendered": '<html><head><title>Offers | Spice Box</title></head>'
                    '<body><div id="app"><h1>This week\'s offers</h1><a href="/blends/garam-masala">Garam masala</a>'
                    '</div></body></html>',
    },
    "long-pepper": {  # a product that no longer exists, answered with the app shell and status 200
        "status": 200,
        "raw": SHELL,
        "rendered": '<html><head><title>Spice Box</title><link rel="canonical" href="https://spicebox.example/"></head>'
                    '<body><div id="app"><h1>Spice not found</h1><a href="/spices/">Spices</a></div></body></html>',
    },
    "blends": {  # a listing whose script draws its items as spans with onclick
        "status": 200,
        "raw": SHELL,
        "rendered": '<html><head><title>Blends | Spice Box</title>'
                    '<link rel="canonical" href="https://spicebox.example/"></head>'
                    '<body><div id="app"><h1>Blends</h1>'
                    '<span onclick="go(\'/blends/garam-masala\')">Garam masala</span> '
                    '<span onclick="go(\'/blends/chai-masala\')">Chai masala</span> '
                    '<a href="/">Home</a></div></body></html>',
    },
}
PAGES["saffron"]["rendered"] = PAGES["saffron"]["raw"]


def seo_view(html):
    """What one version of a page says to a crawler: a dict with the keys title, description, canonicals (a list),
    noindex (True or False), h1 and links (the href of every a element, in order; javascript: hrefs left out)."""
    return {"title": None, "description": None, "canonicals": [], "noindex": False, "h1": None, "links": []}


def only_after_scripts(name):
    """The keys of seo_view whose values differ between the page's raw and rendered HTML, sorted."""
    return []


def google_indexes(name):
    """Which version Google can be expected to index: "rendered" for a 200 page without noindex in its raw HTML;
    otherwise "raw", because Google may skip rendering a page whose status is not 200 or whose raw HTML says noindex."""
    return "rendered"

from seo_view import PAGES, google_indexes, only_after_scripts, seo_view


def test_server_rendered_page():
    """reads the saffron page: title, description, one canonical (not the stylesheet), no noindex, h1, two links"""
    assert seo_view(PAGES["saffron"]["raw"]) == {
        "title": "Saffron threads | Spice Box",
        "description": "Kashmiri saffron threads, one gram.",
        "canonicals": ["https://spicebox.example/spices/saffron"],
        "noindex": False,
        "h1": "Saffron threads",
        "links": ["/spices/", "/recipes/saffron-kheer"],
    }


def test_what_the_app_shell_adds():
    """the cardamom page's script adds everything but the home canonical, which stays as a second canonical"""
    assert only_after_scripts("cardamom") == ["canonicals", "description", "h1", "links", "title"]
    assert seo_view(PAGES["cardamom"]["rendered"])["canonicals"] == [
        "https://spicebox.example/", "https://spicebox.example/spices/green-cardamom"]
    assert only_after_scripts("saffron") == []


def test_noindex_in_the_raw_html():
    """the offers page carries noindex in its raw HTML, so Google may index the raw version, with its noindex"""
    assert seo_view(PAGES["offers"]["raw"])["noindex"] is True
    assert seo_view(PAGES["offers"]["rendered"])["noindex"] is False
    assert google_indexes("offers") == "raw"
    assert google_indexes("cardamom") == "rendered"


def test_spans_are_not_links():
    """the blends listing's items are spans with onclick: after its script, the only link is Home"""
    assert seo_view(PAGES["blends"]["rendered"])["links"] == ["/"]


def test_missing_spice_answers_200():
    """the long pepper page says Spice not found while its status says 200: a soft 404 for Google"""
    page = PAGES["long-pepper"]
    assert page["status"] == 200 and seo_view(page["rendered"])["h1"] == "Spice not found"
    assert "h1" in only_after_scripts("long-pepper")

Then practise the five ideas of this lesson, starting with your weakest:

Interview questions

Warm-up (fresher to 2–5 years): does Google index content that JavaScript adds to a page?

Yes, usually, but later and with conditions. Google crawls the page, queues it for rendering, runs its scripts in an evergreen version of Chromium and indexes the rendered HTML, so text, links and tags that a script adds can be indexed. Three conditions apply. The page must answer 200 and must not say noindex in the HTML the server sends, or Google may skip rendering. The content must appear without a click or a scroll, because Google does not interact with the page. And the scripts must not be blocked by robots.txt. Other crawlers, some search engines among them, may not run scripts at all, so what every reader of the page needs, the title, the description, the canonical, the main text and the links, is safest in the HTML the server sends.

Two more questions follow, about a shop rebuilt as an app whose pages stopped being indexed and a paywall that hides articles with a script. Make the first decision of each before you open it:

Key takeaways

• Google crawls, renders and indexes: a page that answers 200 without a noindex is rendered in an evergreen Chromium and indexed as rendered, while other pages, and crawlers that run no scripts, get only the server’s HTML.
• Compare the HTML the server sends with the rendered HTML from URL Inspection, and put what every reader needs, the title, description, canonical, main text and links, in the server’s HTML.
• The traps are links without href, content behind a click or a scroll, fragment routes, a noindex a script removes and a canonical a script changes.
• A single-page app answers 404 for a missing page from the server, or sends the browser to an address that answers 404, or adds noindex to its error view.
• Pages that must be found are built ahead of time or on the server, with scripts on top; dynamic rendering is a workaround, and client-side rendering suits pages nobody searches for.