SEO glossary
What is HTTP?
Learn what HTTP is, how request-response cycles power crawling, why unencrypted HTTP is legacy for SEO, and how to handle HTTP URLs without duplicate content or redirect loops.
Definition
HTTP (Hypertext Transfer Protocol) is the application-layer protocol governing how web clients and servers exchange requests and responses—forming the basis of URL fetching by browsers and search engine crawlers before TLS encryption in HTTPS.
HTTP: the protocol every crawl starts with
HTTP (Hypertext Transfer Protocol) defines how clients ask for resources and servers answer with status codes, headers, and bodies. Every Googlebot fetch is an HTTP conversation—whether the wire is encrypted (HTTPS) or plaintext (HTTP on port 80).
SEO practitioners rarely configure HTTP directly—they configure URLs, redirects, and headers that HTTP carries. Understanding HTTP semantics clarifies why 301 Redirect beats meta refresh, why 404 Error differs from 500 Error, and why redirect loops block indexing entirely.
Basic HTTP request-response
GET /blog/technical-seo HTTP/1.1
Host: example.com
User-Agent: Googlebot/2.1
Accept: text/html
Accept-Encoding: gzip, br
HTTP/1.1 200 OK
Date: Fri, 14 Aug 2026 12:00:00 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 28400
Link: <https://example.com/blog/technical-seo>; rel="canonical"
Cache-Control: public, max-age=600
<!DOCTYPE html>...
| Part | Role in SEO |
|---|---|
Request method (GET) | Standard crawl retrieval |
| Request path | URL discovery target |
Status code (200) | 200 OK vs error classification |
Headers (Link, Cache-Control) | Canonical, caching, robots |
| Body (HTML) | Content for indexing |
HTTP methods crawlers encounter
| Method | Crawler usage | SEO note |
|---|---|---|
| GET | Primary page fetch | Default crawl |
| HEAD | Lightweight existence check | Same status as GET |
| POST | Rare for HTML index | APIs, forms—not typical SERP URLs |
| PUT/DELETE | Rare | Method-sensitive redirects use 307/308 |
HTTP status codes SEO depends on
HTTP defines the status vocabulary glossary entries map to:
2xx success → 200 OK (indexable baseline)
3xx redirect → 301, 302, 307, 308 + Location header
4xx client → 404 Error, 410 Gone
5xx server → 500 Error, 502 Bad Gateway, 503 Service Unavailable
Search engines implement policies on these codes—honest status accelerates correct indexation or deindexing.
HTTP vs HTTPS for public SEO
| Public web SEO | Recommendation |
|---|---|
http:// indexable pages | Migrate to HTTPS |
| HTTP with 301 to HTTPS | Correct legacy handling |
| HTTP and HTTPS both 200 | Duplicate URLs—fix redirects |
| HTTP only internal backhaul | OK behind CDN |
Google's HTTPS signal rewards secure sites; browser UX punishes HTTP pages with "Not Secure" warnings—affecting engagement metrics indirectly.
Correct HTTP legacy pattern
http://example.com/page → 301 → https://example.com/page → 200 OK
Single-hop 301 Redirect on canonical host—avoid redirect chains stacking host and protocol fixes separately over years.
HTTP headers with direct SEO impact
Link header
Link: <https://example.com/page>; rel="canonical"
HTTP-layer canonical—works for non-HTML types.
X-Robots-Tag
X-Robots-Tag: noindex, nofollow
Controls indexing without HTML access—must return on successful fetch (not only 500 Error).
Location (with 3xx)
HTTP/1.1 301 Moved Permanently
Location: https://example.com/new-url
Drives redirect chains and redirect loops when misconfigured.
Retry-After (with 503)
HTTP/1.1 503 Service Unavailable
Retry-After: 1800
Guides crawler backoff—503 Service Unavailable semantics.
Cache-Control
Over-caching 404 Error or redirect responses poisons CDN behavior post-fix.
HTTP in the crawl pipeline
URL discovery → HTTP GET → Status evaluation → Render → Index
Crawl budget counts HTTP transactions—including redirect hops and failed attempts.
Common HTTP mistakes in SEO audits
Both schemes live
Both http:// and https:// returning 200 OK creates duplicate indexing risk—consolidate with 301 Redirect.
Internal links use HTTP on HTTPS site
Forces redirect hop every crawl—fix templates and CMS "site URL" settings.
Redirect loops between schemes
Break by aligning origin and edge HTTPS policy—see Redirect Loop.
Example: legacy HTTP backlinks
http://example.com retained 200 OK for years "for old links" while https:// also 200 OK.
Problems:
- Split indexed URLs in Search Console
- External HTTP links never consolidated without redirect
- Analytics session fragmentation
Fix: HTTP 301 Redirect to HTTPS for all paths; HSTS header; internal link sweep; sitemap HTTPS-only. Duplicate indexed URLs dropped 31% over two months.
Testing HTTP behavior
curl -I http://example.com/
curl -v -L --max-redirs 10 -o /dev/null http://www.example.com/ 2>&1 | grep '< HTTP'
How Crawlox helps with HTTP URL hygiene
Crawlox identifies HTTP URLs still returning 200 OK, maps redirect chains from http:// entry points to final HTTPS destinations, and flags internal links that unnecessarily start crawls on HTTP. Audit scheme consistency across sitemaps, canonicals, and hrefs—so plaintext HTTP legacy does not split indexing signals or trap Googlebot in redirect loops between HTTP and HTTPS policies.
Related terms
Frequently asked questions
Is HTTP bad for SEO?
Public indexable pages on HTTP are deprecated practice. Browsers warn users, Google prefers HTTPS, and duplicate HTTP/HTTPS URLs split signals if both return 200 OK.
Should I block HTTP with robots.txt?
Usually no—redirect HTTP to HTTPS with 301 instead. Blocking may leave HTTP URLs uncrawled but still discovered via links, creating unclear states.
Can Googlebot crawl HTTP sites?
Yes. Googlebot still fetches http:// URLs when discovered. Best practice is immediate redirect to HTTPS equivalents for public content.
What HTTP methods do crawlers use?
Primarily GET for page retrieval. HEAD occasionally. POST/PUT are rare in standard web crawling—see 307/308 for method-preserving redirects on APIs.
Is HTTP still used internally?
Origins behind CDNs may use HTTP on private backhaul—but public URLs in sitemaps and links should be HTTPS.
References
Explore authoritative guidance and frameworks related to http.
Explore every glossary definition
Return to the glossary to search by term, alias, starting letter, or category.