HTTP Header Checker
See what a server sends back, and what each header means for search.
Check the headers of a URL
Requesting the URL from MySmartCoPilot’s server… slow sites can take up to a minute.
Headers
What this means
Responses
About the HTTP Header Checker
Enter a URL and MySmartCoPilot’s server requests it with GET or HEAD, follows redirects if you want, and shows the status code and every response header of each hop. The headers are explained in plain words and checked for what matters to search engines: X-Robots-Tag rules, Link headers that name a canonical URL or language versions, Content-Type and charset, compression, caching (Cache-Control, ETag, Last-Modified, Vary) and a few short notes on security headers.
Already have the headers? Switch to Paste headers and paste the output of curl -I, or a copy from your browser’s network panel. They are explained in your browser without a request to anyone.
How to use it
- Choose Fetch a URL and enter the address, or choose Paste headers to explain headers you already have.
- Pick GET (what browsers and crawlers send) or HEAD (headers only). Leave Follow redirects on to see the headers of the final page, or turn it off to see the redirect response itself.
- Press Check headers. The summary shows the status, number of headers, redirects and the time to the first response.
- Read What this means for problems and tips, each with the fix and the standard or Google page behind it; filter it by Search, Caching, Security or Content.
- Open a hop to see its headers in a table with a description of each one. Copy the report, or download it as text or CSV to share or compare.
Examples
https://www.example.com/private/price-list.pdf
200 OK content-type: application/pdf x-robots-tag: noindex, nofollow cache-control: max-age=600 X-Robots-Tag: noindex — Google will not show this URL in search results.
Files that are not HTML cannot carry a robots meta tag, so X-Robots-Tag is the way to give them robots rules. Remove it if the file should be found.
curl -sI https://example.com/ | pbcopy (then paste them into “Paste headers”)
HTTP/2 200 content-type: text/html; charset=UTF-8 content-encoding: br vary: Accept-Encoding etag: "3147526947"
Pasted headers are read in your browser. A status line such as HTTP/2 200 is used when present; without one, status 200 is assumed.
Common uses
- Checking that a PDF, image or feed carries the
X-Robots-Tagor canonicalLinkheader you set in the server. - Seeing whether pages are compressed (gzip or Brotli) and whether they can be revalidated with
ETagorLast-Modified. - Debugging caching: what the browser, a CDN and Google’s crawlers are told about how long a page stays fresh.
- Comparing what the server sends to a mobile or desktop browser user agent, for example with
Vary: User-Agentor dynamic serving.
Headers that matter for search
X-Robots-Tag: robots rules such asnoindexornosnippetfor any file type. Rules can name a crawler (googlebot: noindex). See Google’s robots meta tag, data-nosnippet and X-Robots-Tag specifications.Link: <url>; rel="canonical": a canonical URL for files that cannot carry a<link>tag, andrel="alternate"; hreflangfor language versions (canonicalization, localized versions). The syntax is RFC 8288.ETagandLast-Modified: Google’s crawlers use them to ask “has it changed?” and get304 Not Modified; when both are sent they useETag(Google’s crawlers and fetchers).Content-Encoding: Google’s crawlers accept gzip, deflate and Brotli, so compressing text responses is safe and makes pages faster.Locationand the status code: how redirects and errors are treated is explained in Google’s HTTP status codes and network errors.
What the checker sends
Requests come from Cloudflare’s network with the user agent MySmartCoPilotBot/1.0, or a current Chrome desktop or mobile string if you choose one so you can see what browsers get. Search-engine crawlers are never imitated. Each request has Accept-Encoding: gzip, br, no cookies and an 8 second timeout. Only http:// and https:// on ports 80 and 443 are allowed; before each new host is contacted its DNS records are looked up over HTTPS and every address must be public, and every redirect target is checked again.
Limitations
- A website can answer MySmartCoPilot’s server differently from your browser: bot protection, country or device rules, cookies and logged-in states all change the headers. Google Search Console’s URL Inspection shows what Google itself received.
- The fetch API lists headers in alphabetical order and in lower case, not in the order the server sent them. The HTTP version and the TLS details are not available; use the SSL Certificate Checker for the certificate.
Content-Lengthis the size on the wire when the server sends one, so for a compressed response it is the compressed size.- Time is measured from Cloudflare’s network to the first response bytes of each hop and does not include your own connection.
- Only public websites can be checked: not localhost, private networks or pages behind a login. For those, run
curl -Iyourself and use Paste headers. - Each visitor can run a limited number of checks per hour (and the whole site a limited number per day) to stay within free hosting limits.
Privacy
In Fetch mode the URL you enter is sent to MySmartCoPilot’s server, which requests it from the website. MySmartCoPilot does not store the URL, the headers or the result. For abuse prevention, the server log records only the host name, status code and time of each request, never the full URL or your IP address. In Paste mode the headers never leave your browser.
Frequently asked questions
How do I check the X-Robots-Tag of a page or file?
Enter its URL and look at the Search findings and the x-robots-tag row in the headers table. Rules that apply only to one crawler are shown separately (googlebot: noindex), and rules Google ignores (such as noarchive) or does not know are called out.
Why does the checker say my page is not compressed when my browser shows gzip?
MySmartCoPilot asks for gzip and Brotli, like Google’s crawlers, and reports what comes back. Some servers only compress for certain content types or above a size, and some CDNs compress differently for browsers and bots. Check the Content-Type, and compare with a request from your own device using curl -I -H "Accept-Encoding: gzip, br" URL.
What is the difference between GET and HEAD?
GET asks for the whole response, HEAD only for the headers. HEAD is lighter, but some servers answer it differently (another status, missing headers), so use GET when you want to see what browsers and crawlers receive.
Can I check a staging site or a page behind a login?
Not with Fetch mode: it only reaches public websites, with no credentials or cookies. Run curl -I on your own machine, paste the output into Paste headers, and the same explanations and checks run in your browser.
Which headers does Google actually use?
For indexing: the status code, Location, X-Robots-Tag, the canonical and hreflang Link headers, and Content-Type. For crawling efficiency: ETag and Last-Modified. Google says it works out a page’s language from its visible content, not from code-level information such as lang attributes, and that its crawling infrastructure supports the ETag and Last-Modified validators but no other HTTP caching directives.