AI Crawler Checker
Compare crawler and content-use tokens against the robots.txt text you supply.
Try before you buy.
- Free preview: counts and a share of token/path decisions; complete rules, vendor notes and exports open with a pass.
- Locked until you unlock it: download and copy.
- Unlock: Pro pass, ₹179 for 30 days or ₹79 for 1 week, a one-time payment that never renews. Unlock: Pro pass, $5.99 for 30 days or $2.99 for 1 week, a one-time payment that never renews.
Ways to unlock shows how to get the full result.
Printing this result is locked in the free preview.
Local source findings
Locked in the free preview. Opens the ways to unlock this result.
Locked in the free preview. Batch runs unlock with a pass.
Locked in the free preview. Query results unlock with a pass.
About the AI Crawler Checker
Read a robots.txt file as local text and compare its rules with documented training, corpus collection, search and user-request tokens. The checker evaluates the home page and an optional chosen path using exact product groups, merged matching groups, wildcard fallback, case-sensitive paths, percent handling and the longest matching rule. An allow tie wins.
The free preview shows counts and a share of real token/path decisions. A Pro pass opens the full table and exports. Content-use control tokens are labeled separately from request user agents; a rule result is never presented as proof of actual crawler compliance.
How to use it
- Paste the site's robots.txt text; an empty file is a valid file with no restrictions.
- Enter an optional site address for origin context and a path or same-origin URL. No address is requested.
- Analyze the home page and chosen path, then inspect purpose, token role, applied group lines and deciding rule.
- With a pass, copy or download the complete report. Open each token's official documentation to review vendor-specific behavior.
Examples
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
The supplied training rule disallows the home page; the supplied search rule allows it. User-request fetchers remain separate.
Rules used
RFC 9309 defines product groups, merging, wildcard fallback and path comparisons. Matching begins at the path's first octet. The checker retains encoded reserved octets, decodes unreserved ASCII escapes and encodes Unicode as UTF-8. The implicit /robots.txt allowance is preserved.
Documented bot purposes
GPTBot, OAI-SearchBot and ChatGPT-User, ClaudeBot, Claude-SearchBot and Claude-User, Google-Extended, Applebot-Extended, CCBot and Perplexity tokens have separate documented uses. Google-Extended also controls documented AI grounding uses; CCBot collects an open corpus rather than being labeled training-only.
Limitations
- No site URL, robots.txt file or /llms.txt resource is fetched. Paste the actual text yourself.
- The result is a rule interpretation, not authentication of a crawler, proof of enforcement, indexing eligibility, legal permission or removal of existing corpus copies.
- User-request fetchers differ: some vendors say robots.txt may not apply or is generally ignored. A disallow row does not promise that every user-triggered request is blocked.
- The local profile accepts 1 MiB of robots text and target addresses up to 8,192 characters. Crawl-delay, Sitemap and other extension records are outside Allow/Disallow path decisions.
Privacy
On this device; its files download once
Robots text and addresses stay on this device. Nothing is uploaded or requested from the supplied site. Editing, reset, cancel and leaving invalidate the current result.
Frequently asked questions
What do I get without a pass?
Without a pass, AI Crawler Checker shows counts and a share of token/path decisions; complete rules, vendor notes and exports open with a pass. Until you unlock it, the result can’t be downloaded or copied. A Pro, Premium or Ultimate pass, a one-time payment that never renews, unlocks the full result. The pricing page lists the passes and their prices.
Is a control token an actual crawler?
Not always. Google-Extended and Applebot-Extended control documented uses of already collected content and do not have their own page-fetching user agent.
Can blocking training keep search available?
The documented training and search controls are separate. Review the exact token's vendor scope and the supplied rule, rather than assuming one token controls every use.
Does an allowed row prove a bot can access the site?
No. Authentication, network protections, bot identity and vendor behavior are separate. This checker interprets only the supplied robots text.