SEO glossary
What is a Fetch?
Learn what fetch means for SEO—the HTTP request and response step crawlers perform before rendering and indexing—and how status codes, headers, and timing affect bot retrieval.
Definition
Fetch is the HTTP retrieval step in which a search crawler requests a URL and receives the response—including status code, headers, and body—before any rendering or indexing decisions are made.
Fetch as the atomic unit of crawling
Every crawl story boils down to repeated fetches: one URL, one HTTP conversation. The crawler sends a request; your server (or CDN edge) returns a response. That payload—HTML, headers, redirects—is raw material for rendering and indexing.
Without a successful fetch, there is nothing to parse. Crawl errors are failed fetches. Crawlability determines whether fetches are allowed. URL discovery decides which URLs eventually get fetched.
Anatomy of a crawler fetch
Typical sequence:
- DNS resolution — hostname to IP.
- TCP/TLS handshake — secure connection on HTTPS.
- HTTP request — method (usually GET), path, headers (
User-Agent,Accept-Encoding, conditionalIf-Modified-Since). - HTTP response — status line, headers (
Content-Type,Cache-Control,Locationfor redirects), body bytes. - Follow-up — additional fetches for linked resources during rendering (CSS, JS, images).
Example request/response (simplified)
GET /pricing HTTP/2
Host: example.com
User-Agent: Googlebot/2.1
Accept-Encoding: gzip, deflate, br
HTTP/2 200
Content-Type: text/html; charset=utf-8
Cache-Control: max-age=3600
<!DOCTYPE html>...
The fetch completes when the response body is received (or fails mid-stream).
What fetch delivers to search systems
| Response element | SEO relevance |
|---|---|
| Status code | Success vs crawl error classification |
Location header | Redirect discovery and chain length |
Content-Type | HTML vs PDF vs binary—indexing eligibility |
X-Robots-Tag | Indexing directives at HTTP layer |
| HTML body | Links for further discovery, meta robots, canonical |
Last-Modified / ETag | Conditional fetch efficiency (304 Not Modified) |
Fetch does not interpret JavaScript by itself in all pipelines—that is rendering. But the initial fetch must retrieve HTML and referenced asset URLs.
Fetch timing and crawl efficiency
Slow fetches tax crawl budget:
- High time-to-first-byte (TTFB) on list pages reduces how many URLs Googlebot dequeues per day.
- Timeouts look like reliability problems—bots back off.
- Oversized HTML bodies increase transfer cost without adding indexable text.
Example: CDN cache miss storm
Origin serves 200 but takes 8 seconds on cache miss. Googlebot still fetches successfully—no crawl error—yet effective crawl rate drops. Monitoring fetch latency in logs matters as much as error counts.
Fetch vs render in diagnostics
Search Console distinguishes:
- Crawled as — what the initial fetch retrieved.
- Rendered HTML — DOM after JavaScript execution (when applicable).
A page can fetch static HTML with little text, then render rich content client-side. SEO troubleshooting requires checking both stages—not assuming the first fetch tells the whole story.
Conditional and duplicate fetches
Crawlers reuse connections and may send validators:
- 304 Not Modified — body skipped; saves bandwidth when content unchanged.
- Repeated fetches — same URL recrawled on schedules driven by change signals and queue priority.
Accidental infinite freshness (always Cache-Control: no-store on static assets) forces full refetches every time—wasteful but not always wrong for dynamic apps.
Security and fetch integrity
- Block only malicious bots at the edge; verify Googlebot via DNS before allowlisting.
- Do not cloak—returning different HTML to bots vs users violates guidelines.
- Ensure TLS certificates cover all host variants bots resolve.
CDN, edge, and origin fetch behavior
Most fetches hit a CDN before origin. Each layer can change outcomes:
| Layer | What bots experience | Common SEO mistake |
|---|---|---|
| CDN cache HIT | Fast 200 from edge | Stale noindex or old redirects cached |
| CDN cache MISS | Slower origin round trip | Timeouts counted against crawl efficiency |
| Edge WAF | 403 challenge or block | Aggressive bot rules without Googlebot allowlist |
| Origin | Authoritative content | 5xx under load during crawl spikes |
Vary headers and cache keys matter when mobile and desktop responses differ—smartphone Googlebot must receive the mobile-intended payload, not a cached desktop variant.
Example: cache poisoning after deploy
A deploy briefly returns 500 for /products/*. CDN caches error responses for 60 seconds. Googlebot fetches during the window and logs crawl errors even after origin recovers. Purge cache or shorten error TTLs on HTML routes.
HTTP versions and connection reuse
Modern crawlers reuse connections:
- HTTP/2 multiplexing reduces handshake overhead across many fetches to the same host.
- Keep-alive avoids repeated TLS negotiation—broken keep-alive configs inflate latency.
- HTTP/3 (QUIC) appears in some bot infrastructure; servers should not assume HTTP/1.1 only.
Fetch efficiency at the protocol layer raises how many URLs drain from the crawl queue per hour—especially on asset-heavy templates.
Request headers that shape fetches
Beyond User-Agent, crawlers send hints your stack should handle correctly:
| Header | Typical crawler use | Misconfiguration risk |
|---|---|---|
Accept-Encoding | gzip, br | Double compression or broken encoding |
If-Modified-Since / If-None-Match | Conditional GET → 304 | Always 200 with full body wastes bandwidth |
Accept | text/html preference | Wrong Content-Type on HTML routes |
From / contact (some bots) | Abuse reporting | Irrelevant to SEO but used in verification |
Log full request headers for a sample of verified Googlebot hits during audits—surprises often hide in content negotiation.
Fetch debugging workflow
When URL Inspection disagrees with your browser:
curl -A "Googlebot" -I https://example.com/path— status and headers only.- Follow redirects manually (
-L) and count hops. - Compare response from origin vs CDN (bypass headers if safe in staging).
- Check robots and auth middleware order—WAF may run before your app logs the request.
- Measure TTFB and total download time for HTML document only.
- Diff response body hash before/after deploy.
Document findings per template—one fixed product URL does not prove category pagination fetches cleanly.
Fetch efficiency checklist
Before blaming indexing for visibility gaps:
- HTML document TTFB under team target (often <500ms for bots)
- No redirect chains longer than one hop for canonical URLs
-
Content-Typematches actual body (text/htmlfor pages) - Compression enabled without corruption
- 304 responses work for unchanged templates
- Error pages return correct status codes (not 200 soft errors)
- TLS valid on all host aliases (
www, apex, alternate TLDs)
How Crawlox helps with fetch analysis
Crawlox records HTTP status, response times, redirect hops, and payload characteristics during authorized crawls. You compare fetch outcomes across templates, spot slow or bloated responses, and align what your server returns with what search crawlers need before rendering even begins.
Related terms
Frequently asked questions
Is fetch the same as crawl?
Crawl is the broader process of discovering and retrieving site URLs. Fetch is the specific HTTP transaction for one URL—request out, response back.
What does Google Search Console Live Test fetch?
URL Inspection’s live test performs a real-time fetch (and often render) to show what Google can retrieve now—not a guarantee of indexed state.
Do crawlers always use GET?
Search crawlers primarily use HTTP GET for page retrieval. They do not submit forms or execute purchases during standard fetches.
Can fetch succeed but indexing fail?
Yes. A 200 fetch can still lead to noindex, canonical consolidation, or quality exclusion during indexing.
Does gzip compression affect fetch?
Bots request compressed responses when supported. Compression reduces transfer time—a positive for crawl efficiency if the server handles it reliably.
References
Explore authoritative guidance and frameworks related to fetch.
Explore every glossary definition
Return to the glossary to search by term, alias, starting letter, or category.