What is Robots.txt?

Robots.txt: crawl permission at the host level

Robots.txt is a text file at https://example.com/robots.txt that tells compliant crawlers which paths they should not request. It operates at the host level before individual page HTML is fetched. It is part of the Robots Exclusion Protocol (REP)—a convention, not a security mechanism.

Robots.txt controls crawl, not index directly. Misunderstanding that distinction causes accidental deindexing scenarios and "why is this still in Google?" support tickets.

Basic robots.txt syntax

User-agent: *
Disallow: /private/
Allow: /private/public-report/

User-agent: Googlebot
Disallow: /tmp/

Sitemap: https://example.com/sitemap.xml

Key directives:

• User-agent — which crawler the rules target (* = all)
• Disallow — path prefix crawlers should avoid requesting
• Allow — exceptions within disallowed paths (longest match wins in Google)
• Sitemap — optional URL to XML sitemap (discovery hint)

Paths are relative to the host root. Disallow: /blog blocks /blog and /blog/post unless Allow overrides match more specifically.

Robots.txt vs noindex

Example: disallow-only mistake

Team disallows /promo/ in robots.txt hoping to hide launch pages. External blog links to promo URLs. Google never crawls them but lists URLs with snippetless results generated from anchor text. Fix: allow crawl + noindex until launch, or authenticate.

Example: blocking assets

Disallow: /static/

/static/ contains sitewide JavaScript required to render product copy. Smartphone Googlebot fetches blocked JS; rendered page looks empty. Fix: allow public assets; disallow only admin bundles.

Common robots.txt use cases

Save crawl budget

Block infinite calendars, internal search result pages, printer-friendly duplicates, and low-value session URLs.

Hide non-public environments

Staging should use authentication plus noindex—not only robots.txt on production copies.

Separate crawler policies

Rarely, different rules per bot:

User-agent: AdsBot-Google
Allow: /landing/

Most sites use one policy for * unless ad landing verification requires exceptions.

Robots.txt mistakes that break SEO

Example: CMS default robots

WordPress default Disallow: /wp-admin/ is fine; plugins sometimes add Disallow: / during maintenance mode and forget to remove it.

Example: international subdirectory block

Disallow: /de/

German market invisible to crawlers—traffic cliff in /de/ property. Intentional geo-blocking requires broader strategy than accidental robots paste.

Robots.txt and URL discovery

Disallowed URLs may still appear in search results without content snippets if linked externally. They also will not pass internal link equity through crawled paths if bots cannot fetch them—internal linking to disallowed URLs creates dead ends for Googlebot.

Sitemap entries pointing to disallowed URLs produce Search Console warnings. Align sitemap and robots policies.

Testing and monitoring robots.txt

Practices:

1. Version-control robots.txt like application code
2. Diff robots on deploy pipelines
3. Alert when production Disallow: / appears
4. Crawl with robots enabled vs disabled to measure impact
5. Check both http and https and www variants

Google Search Console URL Inspection shows robots.txt fetch status for tested URLs.

Robots.txt limitations

• Not access control — malicious bots ignore it
• Not index control alone — use robots meta tag or X-Robots-Tag
• Not cross-domain — each host has its own file (blog.example.com separate from www)
• Not retroactive magic — removing disallow does not instantly restore indexation

Robots.txt and JavaScript rendering

Google needs to crawl JS/CSS to render many pages. Overly broad disallow on /assets/ or CDN paths harms rendering quality. Use targeted disallows on secret paths only.

Enterprise robots.txt governance

Large organizations maintain:

• Central registry of path prefixes with owners
• Review board for new disallow requests
• Automated tests that critical templates remain allowed
• Documentation linking robots rules to indexation policy

Ad-hoc disallows by engineering during incidents often linger for years.

Robots.txt myths

• Myth: "Robots.txt removes pages from Google." Reality: use noindex or remove URLs publicly.
• Myth: "Blocking CSS improves SEO." Reality: harms rendering.
• Myth: "All bots must obey robots.txt." Reality: only polite crawlers; scrapers may not.
• Myth: "Robots.txt errors always block the whole site." Reality: depends which lines are wrong—partial blocks are common and subtle.

How Crawlox helps with robots.txt

Crawlox fetches and applies your live robots.txt during authorized crawls, showing which indexable URLs are accidentally disallowed, which assets block rendering, and where sitemap URLs conflict with disallow rules. Compare crawl coverage with robots enabled against your intended policy—catch production deploy regressions before crawlability collapses across entire template directories.