What is Compression?

Compression: shrinking text on the wire

When a browser or crawler requests your homepage, the server can send raw HTML—or the same HTML compressed so fewer bytes cross the network. HTTP compression (content encoding) is the umbrella mechanism: the client advertises supported algorithms in Accept-Encoding; the server picks one, compresses the response body, and sets Content-Encoding so the client decompresses before parsing.

For Technical SEO, compression is foundational hygiene. Uncompressed 180 KB JavaScript bundles slow JavaScript rendering, hurt LCP on mobile, and waste crawl budget bandwidth that could fetch additional URLs.

Compression complements—but does not replace—code minification, image optimization, and efficient HTML architecture.

The negotiation handshake

Request:

GET /assets/app.js HTTP/2
Accept-Encoding: br, gzip, deflate, zstd

Response:

HTTP/2 200
Content-Type: application/javascript
Content-Encoding: br
Vary: Accept-Encoding

The browser decompresses br body bytes before the JS engine parses them. Crawlers including Googlebot send Accept-Encoding: gzip, deflate, br—they benefit identically.

Algorithms at a glance

This page covers the general HTTP compression layer. For algorithm-specific tuning, CPU tradeoffs, and server config snippets, read Gzip and Brotli glossary entries—they dive deeper without repeating the full HTTP primer here.

What to compress (and what to skip)

Compress:

• HTML documents
• CSS stylesheets
• JavaScript bundles
• JSON API responses (when crawl-relevant)
• SVG graphics
• XML sitemaps

Skip (already compressed):

• JPEG, PNG, WebP, AVIF, GIF
• Video (MP4, WebM)
• WOFF2 fonts
• PDF, ZIP archives

Rule: if the format is a compression codec, HTTP compression adds little value.

Compression + minification stack

Source code → Minify (Terser/esbuild) → HTTP Brotli → Browser decompress → Parse

Minification removes semantic redundancy; Brotli exploits byte patterns. Together a 500 KB dev bundle might become 120 KB minified and 35 KB over the wire—versus 500 KB uncompressed disaster.

Dynamic vs precompressed static assets

Precompressing at build time avoids server CPU spikes during traffic surges—stable TTFB helps crawlers.

Caching compressed responses

Caches must key on Accept-Encoding or store separate variants. Missing Vary: Accept-Encoding causes browsers to receive wrong encoding—garbled pages or cache poisoning.

Pair with Cache-Control and browser caching policies:

Cache-Control: public, max-age=31536000, immutable
Content-Encoding: br
Vary: Accept-Encoding

Immutable hashed assets compress once, cache forever.

Compression and Core Web Vitals

• LCP: Faster HTML and critical CSS/JS download → earlier paint.
• INP: Less main-thread time waiting on network for parser-blocking scripts.
• CLS: Indirect—faster CSS arrival stabilizes layout rules sooner.

Field data improvements may lag deploy 28 days—lab tests (WebPageTest, Lighthouse) confirm immediately.

Crawl efficiency angle

Search engines operate global fetch infrastructure with practical throughput limits per host. Compressing sitemaps, RSS feeds, and HTML leaves headroom for deeper crawl depth on large catalogs—especially on shared hosting with bandwidth caps.

Common misconfigurations

Testing compression

curl -H "Accept-Encoding: br" -I https://example.com/
# Expect: Content-Encoding: br

curl -H "Accept-Encoding: gzip" -I https://example.com/app.js
# Expect: Content-Encoding: gzip (or br if prioritized)

Chrome DevTools Network panel shows “content download” size vs uncompressed resource size.

Compression vs TLS

HTTPS encrypts the compressed body on the wire—compression happens before encryption inside the TLS stack. No conflict. HTTP/2 multiplexing benefits remain.

Security note: BREACH

Compression side channels (CRIME/BREACH) historically targeted secrets in responses co-compressed with user input. Mitigate by not reflecting secrets in compressible pages with user-controlled fragments—rare in marketing SEO contexts.

Choosing between Gzip and Brotli (summary)

• Default policy: br preferred, gzip fallback, identity last resort.
• Static assets: precompute both; CDN serves best match.
• Dynamic HTML: Brotli level 4–6 on-the-fly or gzip level 6 if CPU constrained.

Detailed tuning lives in algorithm-specific glossary pages.

How Crawlox helps with compression awareness

Crawlox measures response sizes during crawls—highlighting hosts serving unusually large HTML or JS without corresponding fast TTFB. While Crawlox does not inspect Content-Encoding headers alone, bloated transfer sizes signal missing compression or minification worth fixing before crawl render queues stall on obese bundles.