SEO glossary
What is Noindex?
Learn what noindex means in SEO, how robots meta tags and HTTP headers work, when to use noindex vs block crawling, and how to audit accidental index exclusions.
Definition
Noindex is a directive telling search engines not to include a URL in their search index, even if the page is crawled—typically implemented via a robots meta tag or X-Robots-Tag HTTP header.
Noindex in plain language
Noindex is how you tell Google and other search engines: crawl this page if you want, but do not list it in search results. It is an indexing control—not a crawl blocker, not a ranking boost, and not a substitute for fixing duplicate architecture.
When a URL carries a valid noindex signal, the engine may still fetch it (unless robots.txt prevents that), read content, and follow links—but the URL should drop out of the public index over time. For SEO teams, noindex is the scalpel for URLs that must exist for users or operations but should not compete in organic search.
How noindex is implemented
Robots meta tag (HTML)
Placed in the <head> of a document:
<meta name="robots" content="noindex">
Common variants:
| Directive | Meaning |
|---|---|
noindex | Do not index this URL |
noindex, nofollow | Do not index; do not follow outbound links |
noindex, follow | Do not index; still follow links for discovery |
Google supports noindex in the robots meta tag and treats it as a strong hint once seen on a recrawl.
X-Robots-Tag (HTTP header)
Useful for non-HTML resources and server-wide policies:
X-Robots-Tag: noindex
PDFs, images, and API-driven templates often rely on headers because they lack a <head> section. Headers can also apply noindex to entire path prefixes at the CDN or origin.
Conflicting signals
If robots.txt disallows crawling, Google may not see an on-page noindex tag—creating stale indexed URLs that cannot be updated. Best practice for removal: allow crawl, serve noindex, then optionally tighten robots after deindexing.
Noindex vs disallow vs canonical
| Approach | Crawling | Indexing | Best when |
|---|---|---|---|
| Noindex | Usually allowed | Prevented | Page must exist; should not rank |
| Disallow (robots.txt) | Blocked | May remain if URL is known | Hide paths from crawl; not reliable deindex |
| Canonical | Allowed | Consolidates to preferred URL | Duplicates with one winner |
Example: faceted filter pages with zero search value → noindex, follow on combinations you cannot 404. Near-identical product color variants → canonicalization to the parent SKU instead of noindex on every variant.
When noindex is the right tool
Intentional exclusions
- Thank-you and confirmation pages after conversions
- Internal search result pages (
/search?q=) - Staging mirrors (still protect with auth + robots)
- Printable or PDF alternate views with no unique query intent
- User-generated profile shells with thin content
- Paginated archives where only page 1 should rank (sometimes—test against canonical strategies)
Pruning index bloat
Large catalogs accrue parameterized URLs, legacy CMS paths, and A/B test leftovers. After analytics and Search Console review, noindex removes clutter from the index without deleting customer-facing functionality.
Temporary campaigns
Short-lived landing pages may ship with noindex until content is finalized—then remove the directive. Document toggles; forgotten noindex on revenue pages is a common post-launch incident.
When noindex is the wrong tool
- Primary money pages — fix content and internal linking instead
- True duplicates — prefer canonical URL consolidation and redirects
- Soft 404s — return proper 404/410, not 200 + noindex masquerading as gone content
- Penalty recovery — noindex does not erase manual actions on sibling URLs
- Hiding paid link schemes — policy violations are not solved by meta tags
Auditing noindex across a site
Crawl + extract
Run a full crawl and collect:
<meta name="robots">valuesX-Robots-Tagresponse headers- Conflicts between HTML and header layers
Flag templates where noindex is inherited unintentionally (global CMS default, plugin misconfiguration).
Search Console cross-check
In the Pages report, inspect "Excluded by 'noindex' tag." Sudden spikes after deploys point to template regressions. Compare excluded URLs to analytics landing pages—any overlap is revenue risk.
Staging vs production drift
CI pipelines sometimes copy robots policies from staging. Automate checks that production allowlists exclude noindex on indexable templates.
Common noindex mistakes
| Mistake | Consequence |
|---|---|
| Noindex on paginated series incorrectly | Hides deep inventory from index entirely |
| Noindex + blocked by robots | Stale indexed snippets persist |
| JavaScript-injected noindex only | Google may index pre-render HTML without tag |
| Noindex on hreflang alternates | Breaks locale clustering |
| Leaving noindex after A/B test ends | Winner variant never indexes |
For JavaScript sites, verify noindex in rendered HTML—the same URL Googlebot Smartphone uses for indexing.
Noindex and link equity
noindex, follow allows crawlers to traverse outbound links—useful for faceted pages that should not rank but still expose paths to core categories. noindex, nofollow is stricter; employ when outbound links should not be associated with the site graph (rare on public marketing sites).
Noindex does not "concentrate PageRank" on sibling pages the way a 301 might. Think in terms of index cleanliness and crawl focus, not magical ranking transfers.
Noindex myths
- Myth: "Noindex instantly removes URLs." Reality: removal waits on recrawl and processing.
- Myth: "Disallow is faster for deindexing." Reality: disallow can prevent Google from seeing noindex.
- Myth: "Noindex hurts domain reputation." Reality: prudent noindex on junk URLs improves overall quality signals.
- Myth: "Google ignores noindex on important pages." Reality: Google honors valid noindex when crawled—importance does not override explicit directives.
How Crawlox helps with noindex
Crawlox inventories every URL's robots meta and response headers alongside status codes and internal links—surfacing accidental noindex on templates that should rank, and highlighting indexed URLs that should be excluded. Compare crawl-time directives to Search Console coverage so indexing policy matches intent: fewer surprise exclusions, faster cleanup of index bloat, and confidence that staging rules never shipped to production.
Related terms
Index Bloat
Low-value URLs that noindex can help prune from the index.
Duplicate Content
Duplicates you may noindex when consolidation is impractical.
Canonicalization
Preferred alternative to noindex when one URL should represent a set.
Near-Duplicate Content
Thin variants often candidates for noindex after audit.
Frequently asked questions
Does noindex stop Google from crawling a page?
No. Noindex asks Google not to show the URL in search results after crawling. To block crawling entirely, use robots.txt disallow—but Google may still know the URL exists from external links. Noindex and disallow serve different purposes.
How long does it take for noindex to remove a page from Google?
After Google recrawls the URL and processes the noindex directive, removal typically happens within days to a few weeks depending on crawl frequency. Use the Removals tool in Search Console only for urgent temporary hides—not as a substitute for proper noindex.
Can I use noindex and canonical on the same page?
Technically yes, but the combination is usually contradictory. Canonical says 'index another URL instead'; noindex says 'do not index this URL.' Pick one strategy per URL—consolidate with canonicals when a preferred version exists, noindex when the page should not appear at all.
What is the difference between noindex and nofollow?
Noindex controls indexing of the current page. Nofollow tells crawlers not to pass link signals through outbound links on that page (nofollow on links is a separate mechanism). They can appear together as noindex, nofollow when you want neither indexing nor followed outbound links.
Will noindex free crawl budget on large sites?
Indirectly. Noindex does not block crawls, but once Google learns a URL is consistently non-indexable, crawl demand often drops. Pair noindex with internal link cleanup and blocking low-value paths when appropriate to reduce wasted fetches.
References
Explore authoritative guidance and frameworks related to noindex.
Explore every glossary definition
Return to the glossary to search by term, alias, starting letter, or category.