Website SEO Crawler & Site Audit

About the Website SEO Crawler & Site Audit

Enter a start URL and MySmartCoPilot crawls the site the way a search engine does: it reads robots.txt, requests the start page, follows the links it finds, nearest pages first, and checks every URL it reaches. The audit lists what needs fixing — broken internal links, error pages, redirects and redirect chains, missing, duplicate and overlong titles and meta descriptions, missing <h1> headings, canonical problems, noindex pages, pages deep in the site, sitemap URLs that no link reaches, thin and duplicate pages — each with the affected URLs, why it matters and how to fix it.

The crawler is polite by design: it identifies itself as MySmartCoPilotBot, never requests what robots.txt disallows, honours the site’s Crawl-delay and waits at least a second between requests. With a Premium pass a run covers up to 1,000 URLs and the pages, issues and links export as CSV files; without one, the note above the tool says what the free preview covers.

Common uses

• A technical SEO health check before or after a redesign or a migration: broken links, redirects and canonicals in one pass.
• Finding duplicate and missing titles and meta descriptions across a site, with a CSV to fix them page by page.
• Checking a sitemap against the site: listed URLs that redirect, answer with an error, are noindex, or that no page links to.
• Auditing one section of a large site, such as the blog or the documentation folder, within one run.
• Preparing an audit of a client’s site with the issues and their URLs ready to hand over.

What the audit checks

• Status codes: pages that answer 4xx, server errors (5xx, 429) and URLs that did not answer, and broken internal links with the pages that link to them. Google treats every 4xx except 429 as “the content does not exist” (how HTTP status codes affect Google’s crawlers).
• Redirects: internal links that point to redirects, redirect chains (Google’s crawlers follow up to 10 redirect hops, HTTP status codes), redirect loops, temporary 302/307 redirects and meta refresh redirects (redirects and Google Search).
• Titles and descriptions: missing ones, duplicates among indexable pages, and ones likely to be cut off. Google publishes no length limit and shortens them to fit the screen (title links, snippets); the crawler estimates the width in pixels and lists titles wider than about 600 px and descriptions wider than about 920 px.
• Headings: pages without an <h1>, and pages with several (listed for review: HTML allows more than one).
• Canonicals: missing (Google recommends a self-referencing canonical), relative, conflicting, outside the <head>, or pointing to an error, a noindex page or a redirect (consolidate duplicate URLs), and pages that name another URL as canonical.
• Indexing: noindex in a robots meta tag or the X-Robots-Tag header (block indexing with noindex), pages your robots.txt blocks for Googlebot, and invalid elements that end the <head> early, after which Google ignores the metadata (valid page metadata).
• Site structure: click depth from the start page, sitemap URLs that no crawled page links to, sitemap URLs that are not indexable, pages with exactly the same text, pages with few words, internal nofollow links, http:// links on https:// pages, HTML larger than 2 MB (Googlebot reads only the first 2 MB, Googlebot) and slow responses.

How MySmartCoPilotBot crawls

• robots.txt first, read the way RFC 9309 describes: the group for MySmartCoPilotBot if there is one, otherwise the User-agent: * group; the longest matching rule wins and Allow wins a tie. A robots.txt that answers 4xx means there are no rules; one that answers 5xx or 429, or cannot be reached, means the whole site must be treated as disallowed, so nothing is crawled. MySmartCoPilot’s server applies robots.txt to every request as well, so it never fetches a disallowed URL.
• A gentle pace: one URL at a time, at least a second apart (two seconds without a pass, to stay within the per-minute allowance), or the site’s Crawl-delay when that is longer, up to 60 seconds; a site that asks for more is not crawled. When the site answers 429 or 503 the crawler slows down (up to 30 seconds between requests), goes back towards the normal pace only after several clean answers, and stops altogether after five errors in a row.
• One request per URL: a GET with redirects not followed (each redirect target is crawled as its own URL), reading at most the first 2 MB of HTML, as Googlebot does. Files such as PDFs and images get their status code only; private and internal network addresses are never requested.
• Links it follows: <a href> links in the HTML — the links Google can crawl, link best practices — and the <area href> links of image maps, breadth-first, so each page’s click depth is the fewest links from the start page. Links marked nofollow, ugc or sponsored, and links on pages whose robots meta says nofollow, are recorded but not followed unless you tick the option. Canonical targets and URLs found only in the sitemap are crawled last, while the URL limit lasts.
• To keep MySmartCoPilotBot out of your site, add User-agent: MySmartCoPilotBot and Disallow: / to your robots.txt.

Limitations

• Without a pass, the free preview crawls half of the URLs you ask for, at most 10, with every summary figure; with a Premium pass a run covers up to 1,000 URLs. Redirects and linked files count as URLs, and the server limits how many requests one connection makes in an hour. Large sites need several runs, for example one per section.
• The crawler reads the HTML as the server sends it and does not run JavaScript, so links and text that scripts add after the page loads are not seen.
• Orphan pages can only be found among sitemap URLs: a page that is neither linked nor listed in a sitemap cannot be found by any crawler.
• External links are counted but not checked; the Broken Link Checker checks the external links of a page.
• Sites can answer a crawler differently from a browser (bot protection, logins, country rules), and response times are measured from MySmartCoPilot’s server, not from your location.
• Nothing is saved: reloading or closing the page loses the report, so export the CSV files you need (with a pass). Keep the tab open and visible while it runs, because browsers slow down timers in background tabs.

Privacy

The start URL and the URLs the crawl reaches are sent to MySmartCoPilot’s server, which requests them from the website and returns the answers. MySmartCoPilot does not store the URLs or the pages; the server log records only the host name, status code and time of each request. The audit is built in your browser and is not saved anywhere.