SEO glossary
What is X-Robots-Tag?
Learn what the X-Robots-Tag HTTP response header is—how it applies noindex and snippet directives to HTML and non-HTML resources, and when to prefer it over meta robots tags.
Definition
X-Robots-Tag is an HTTP response header that carries robots directives—such as noindex, nofollow, nosnippet, and noarchive—to search engines for any URL response, including non-HTML files where HTML meta robots tags cannot be used.
X-Robots-Tag: robots directives in HTTP headers
X-Robots-Tag extends robots instructions beyond HTML. Any HTTP response—HTML document, PDF, video, font file, API error page—can include:
X-Robots-Tag: noindex, nofollow
Search engines that support the header read it when processing the URL. This is essential for indexation control on non-HTML resources and for applying consistent rules at CDN or web-server layers without editing CMS templates.
X-Robots-Tag syntax
Single directive:
X-Robots-Tag: noindex
Multiple directives comma-separated:
X-Robots-Tag: noindex, nosnippet, noarchive
Bot-specific values (same naming pattern as meta robots):
X-Robots-Tag: googlebot: noindex, nofollow
Snippet controls:
X-Robots-Tag: max-snippet:160, max-image-preview:large
Headers must be on the same URL response you want to control—not only on parent frames or linking pages.
When to use X-Robots-Tag
Non-HTML files
PDF datasheets, whitepapers, and downloadable assets lack <head> sections. Header noindex prevents file URLs from appearing as separate search results when HTML landing pages are preferred.
Server-wide template policies
Apache Header set, Nginx add_header, or CDN workers apply noindex to /staging/ paths without CMS access—useful during migrations.
Binary and media URLs
Image or video URLs indexed independently may cannibalize HTML pages. noindex on asset responses when HTML page is canonical experience.
API and error responses
Accidentally indexable JSON or verbose HTTP status code error pages with thin content—block with header noindex after fixing root cause.
Example: PDF ranking above landing page
/files/guide.pdf ranks for branded queries while /resources/guide/ lags. Add X-Robots-Tag: noindex on PDF responses; strengthen internal linking to HTML version.
Example: CDN layer noindex
Edge adds noindex to all ?preview=true URLs via header rule—cleaner than template forks.
X-Robots-Tag vs robots meta tag
| Factor | X-Robots-Tag | Robots meta tag |
|---|---|---|
| Location | HTTP response header | HTML <head> |
| Non-HTML support | Yes | No |
| CMS dependency | Low (server/CDN) | High (template) |
| Visibility in View Source | No (check Network tab) | Yes |
| JS rendering risk | Lower if set at origin | Higher if client-only |
Prefer one source of truth. Duplicating matching directives is okay; contradicting them is not.
X-Robots-Tag vs robots.txt
robots.txt blocks crawling paths. X-Robots-Tag noindex requires crawl (usually) to be processed. Workflow to remove indexed URLs:
- Remove disallow that prevents fetch (if present)
- Deploy
X-Robots-Tag: noindexor meta robots noindex - Wait for recrawl
- Optionally re-disallow after deindex if crawl waste remains
Never rely on robots.txt noindex—unsupported.
Implementation examples
Nginx
location /private-reports/ {
add_header X-Robots-Tag "noindex, nofollow" always;
}
Apache
<Location "/drafts/">
Header set X-Robots-Tag "noindex, nofollow"
</Location>
Cloud CDN worker (conceptual)
Apply header when request.url matches preview pattern—test thoroughly in staging.
Debugging X-Robots-Tag
Tools:
curl -I https://example.com/file.pdf— inspect response headers- Browser DevTools Network panel — Headers tab
- Search Console URL Inspection — Google-selected indexing state
- Crawlox crawl — stores headers per URL
Common bugs:
- Header only on HTML shell, not on PDF download redirect target
alwaysflag missing in Nginx—no header on 404 responses you still want noindex- CDN caching strips or overwrites headers
- Conflicting
indexin HTML andnoindexin header
Example: redirect drops header
/old.pdf 302 to /new.pdf without noindex; only first response had header. Set policy on final 200 asset URL.
X-Robots-Tag and caching
CDNs cache responses with headers. Purge cache after changing X-Robots-Tag rules or bots may see stale indexable copies. Document header changes in release notes alongside HTTP status code redirect updates.
Security and privacy note
X-Robots-Tag is not authentication. Private files need access controls; noindex only asks search engines not to list URLs—leaked links still work for humans.
Measuring impact
Track Search Console exclusions tagged noindex, PDF URL impressions declining, and crawl stats on /assets/ after header deployment. Pair with XML sitemap cleanup removing noindexed URLs.
X-Robots-Tag mistakes
| Mistake | Consequence |
|---|---|
| Sitewide accidental header | Mass deindex |
| Header on 301 only, not destination | Index persists on final URL |
| Fighting with sitemap inclusion | Processing delays, warnings |
| noindex while disallowing crawl | Slow removal |
| Assuming header affects third-party embeds | Scope is per URL response |
X-Robots-Tag myths
- Myth: "Headers are stronger than meta tags always." Reality: both work when seen; avoid conflicts instead of ranking strength debates.
- Myth: "X-Robots-Tag hides pages from hackers." Reality: not access control.
- Myth: "PDFs cannot be noindexed." Reality: header noindex works when crawled.
- Myth: "Only Google reads X-Robots-Tag." Reality: other engines may support; verify per bot docs.
How Crawlox helps with X-Robots-Tag
Crawlox records HTTP response headers during crawls, flagging URLs where X-Robots-Tag noindex conflicts with indexable templates, where PDFs lack headers while HTML alternates exist, and where CDN layers inject unexpected directives. Correlate header policy with robots meta tag HTML and robots.txt crawl permission in one audit—so indexation rules stay consistent from origin to edge.
Related terms
Frequently asked questions
What is the difference between X-Robots-Tag and the robots meta tag?
They accept the same directive values, but X-Robots-Tag is sent as an HTTP response header while meta robots lives in HTML head. Use X-Robots-Tag for PDFs, images, JSON responses, and server-wide rules; use meta robots for page-level CMS control in HTML templates.
Which wins if X-Robots-Tag conflicts with meta robots?
Google merges signals and the more restrictive directive typically applies for indexing. Avoid conflicts—align CDN, origin, and CMS outputs to the same policy.
Can X-Robots-Tag noindex block PDFs in Google?
Yes, when Google crawls the PDF and processes the header. Ensure the header is on the PDF response itself, not only on HTML pages linking to it.
Should I set X-Robots-Tag on all responses?
Only when needed. Default is indexable for public content. Apply noindex headers deliberately to non-public assets, not sitewide without strategy.
Does X-Robots-Tag work without allowing crawl?
Google must fetch the URL to read the header—similar to meta robots. If robots.txt disallows crawl, Google may not see noindex and might still list bare URLs found elsewhere. Allow crawl for URLs you want removed via noindex.
References
Explore authoritative guidance and frameworks related to x-robots-tag.
Explore every glossary definition
Return to the glossary to search by term, alias, starting letter, or category.