SEO glossary
What is a Robots Meta Tag?
Learn what the robots meta tag is—how HTML meta robots directives control indexing and link following, and when to use them instead of robots.txt or X-Robots-Tag.
Definition
The robots meta tag is an HTML meta element—typically <meta name="robots" content="...">—that instructs search engines whether to index a page and whether to follow links on it, using values such as noindex, nofollow, index, and follow.
Robots meta tag: index instructions in HTML
The robots meta tag places crawler directives inside HTML documents—most famously noindex to keep pages out of search results while still allowing fetch. It is the page-level counterpart to host-wide robots.txt rules and sibling to the X-Robots-Tag HTTP header.
<meta name="robots" content="noindex, follow">
Search engines read this tag during crawl/render. Directives apply to the URL that returned the HTML, not to embedded resources unless separately specified.
Common robots meta values
| Directive | Meaning |
|---|---|
| index | Default; URL may be indexed |
| noindex | URL should not appear in search results |
| follow | Default; links on page may be crawled |
| nofollow | Do not follow links on this page (rare page-level use) |
| none | Equivalent to noindex, nofollow |
| noarchive | Do not show cached link |
| nosnippet | Do not show text snippet |
| **max-snippet:**n | Limit snippet length |
| max-image-preview | Control image preview size |
| max-video-preview | Limit video preview seconds |
Combine with commas: noindex, follow is typical for pages you want crawled for link discovery but not indexed—though allowing indexable alternatives is usually cleaner.
Robots meta tag vs robots.txt vs X-Robots-Tag
| Tool | Layer | Best for |
|---|---|---|
| robots.txt | Host/path crawl permission | Block fetch to save budget |
| robots meta tag | HTML page index/follow | CMS templates, marketing pages |
| X-Robots-Tag | HTTP response header | PDFs, images, non-HTML, global template headers |
Example: staging noindex forgotten
Staging subdomain uses <meta name="robots" content="noindex"> in template—correct. Launch to production forgets to remove tag. Production pages crawl but drop from index. Automated deploy checks should assert absence of noindex on production hosts.
Example: noindex on paginated archives
Tag archive pages with two posts each carry noindex to prevent thin URL indexation. Category hubs remain index with strong internal linking.
When to use noindex
Appropriate:
- Site search result pages (
?q=) - Thank-you and confirmation pages
- Login and account dashboards
- Faceted filter URLs without unique value
- Printer-friendly duplicates when canonicals are insufficient
- Internal policy pages not for public search
Avoid noindex on:
- Revenue landing pages you want to rank
- Canonical product URLs
- "Quick fixes" instead of fixing duplicate URL structure
Implementation patterns
CMS template rules
Ecommerce platforms set noindex on cart, checkout, and account templates automatically—verify custom themes preserve tags.
Conditional noindex
<!-- Pseudologic: thin tag archive -->
<meta name="robots" content="noindex, follow" v-if="postCount < 3">
Document business rules; avoid accidental noindex on growing archives when post count crosses thresholds.
Conflicts with canonical
<link rel="canonical" href="https://example.com/winner/">
<meta name="robots" content="noindex">
Self-referencing noindex with canonical to another URL sends mixed signals. Pick one policy: consolidate to winner with canonical, or noindex duplicates explicitly.
Robots meta and JavaScript SEO
SPAs that inject robots tags only after client boot create windows where Googlebot sees index, follow default. Use SSR, SSG, or server headers for critical directives.
Test with:
- View rendered source
- URL Inspection live tests
- Crawlox rendered HTML capture
Snippet and preview controls
nosnippet and max-snippet influence SERP display, not whether URL is indexed (unless paired with noindex). Use when licensed content or spoilers require limited previews—expect CTR impact.
noimageindex (non-standard broadly) — prefer controlling images via on-page rules and Search Console removals when necessary.
Measuring robots meta impact
Search Console Page indexing report categories:
- "Excluded by noindex tag"
- Validates tag deployment at scale
Crawl audits should extract meta robots per URL and flag:
- Unexpected noindex on money templates
- Missing noindex on known junk templates
- Mismatch between HTML meta and X-Robots-Tag header
Example: header overrides HTML
CDN adds X-Robots-Tag: noindex while HTML says index. Header often wins. Align CDN rules with CMS output.
Robots meta tag mistakes
| Mistake | Result |
|---|---|
| Sitewide noindex in dev theme | Production deindex |
| noindex + disallow together | Slow or incomplete removal |
| noindex important paginated series | Deep content never ranks |
| Relying on noindex in robots.txt | Unsupported; ignored |
| Duplicate conflicting tags | Unpredictable processing |
Robots meta vs rel=nofollow on links
- Page-level meta nofollow — rare; affects all links on page
- Link rel=nofollow — per-anchor; standard for UGC and paid links
Do not confuse external linking rel attributes with page-level robots meta.
Robots meta tag myths
- Myth: "noindex passes equity through to linked pages magically." Reality: noindex pages are dropped; links may still be followed if follow is set.
- Myth: "noindex is instant." Reality: recrawl and processing lag, especially on large sites.
- Myth: "Meta robots block AI training." Reality: use appropriate controls and policies; standard robots meta targets search indexing features documented by Google.
- Myth: "Only one robots meta tag allowed." Reality: googlebot-specific tags possible but rarely needed.
How Crawlox helps with robots meta tags
Crawlox extracts robots meta directives from rendered HTML across your crawl, highlighting templates where noindex appears on high-traffic URLs, where tags are missing on faceted junk, and where meta conflicts with X-Robots-Tag or XML sitemap inclusion. Validate indexation policy before launches—not after Search Console shows thousands of "Excluded by noindex tag" on product pages.
Related terms
Frequently asked questions
What is the difference between noindex and Disallow in robots.txt?
noindex in the robots meta tag tells search engines not to index a page after they crawl it. robots.txt Disallow tries to prevent crawling entirely. If Google cannot crawl a page, it may not see noindex—so blocking crawl while wanting removal is counterproductive. Allow crawl + noindex to remove indexed URLs.
Should I use noindex or canonical for duplicates?
Use canonical when you want one version indexed and duplicates consolidated. Use noindex when a URL should not appear in search at all—thank-you pages, internal search results, thin tag archives.
Does nofollow on meta robots block link equity?
nofollow on a page-level meta robots applies to all links on that page (uncommon). Usually you set rel=nofollow on individual anchors. Page-level nofollow is rare; confirm you mean link-level attributes.
Can robots meta tags be ignored?
Google generally respects standard directives when crawled. Conflicting signals—noindex plus sitemap inclusion, or noindex with strong external links—may delay processing but noindex usually wins when seen.
Where should robots meta tags appear?
In the HTML <head> early enough to be parsed. For JavaScript sites, ensure SSR or prerender includes tags in initial HTML response—not only after delayed client render.
References
Explore authoritative guidance and frameworks related to robots meta tag.
Explore every glossary definition
Return to the glossary to search by term, alias, starting letter, or category.