Cloudflare prototyped Zstandard cache transcoding and could save petabytes — here's what that means for edge infrastructure

Cloudflare published a prototype last week that compresses cached assets with Zstandard on-the-fly inside Pingora, their Rust HTTP proxy. The goal: get more cache capacity from the same SSDs. The result: petabytes of savings if it ships, but tradeoffs that aren't obvious from the headline.

The problem is simple. Cloudflare caches origin responses across 330+ data centers. Most assets arrive compressed with gzip (zlib level 6, the nginx default). Those assets sit on SSD. When a cache fill happens, Cloudflare stores what the origin sent. If the origin sent gzip, the cache stores gzip. If the origin sent uncompressed, the cache stores uncompressed.

Zstandard compresses better than gzip at comparable speeds. Cloudflare's prototype intercepts cache writes, decompresses gzip payloads, recompresses them with Zstandard at level 9, and writes the smaller blob to SSD. On cache reads, it decompresses Zstandard and serves gzip (because most clients still send Accept-Encoding: gzip first).

The compression win is real. Cloudflare tested 50 million cached objects. Median size reduction: 8%. 90th percentile: 22%. For a network storing exabytes, 8% is petabytes.

The CPU cost

The post is honest about the tradeoff. Decompressing and recompressing on every cache write costs CPU. Decompressing on every cache read costs CPU. Cloudflare's edge servers already run near capacity during traffic spikes. Adding compression to the hot path means either provisioning more cores or accepting higher tail latencies.

They tested Zstandard level 9 (high compression, slow) and level 3 (faster, less compression). Level 9 gave the best size reduction but the worst CPU profile. Level 3 was faster but only saved 4% median. The sweet spot isn't obvious yet.

The other tradeoff: Zstandard isn't universally supported. Browsers shipped Accept-Encoding: zstd support in 2023–2024, but older clients and some enterprise proxies still only advertise gzip or Brotli. If Cloudflare stores everything as Zstandard and a client requests gzip, the edge has to decompress and recompress on every read. That's more CPU than just serving the gzip blob that was already on disk.

What this means for edge caching

Cloudflare's post is a prototype, not a launch. They're exploring whether the SSD savings justify the CPU cost. The answer depends on their traffic mix, hardware refresh cycles, and whether they can offload transcoding to dedicated compression cores (Intel QAT, ARM SVE2, etc.).

For everyone else building edge infrastructure, the lesson is: cache storage isn't free, but neither is CPU. The gzip-everywhere default made sense in 2010 when SSD was expensive and CPUs were idle. In 2026, SSD is cheaper per gigabyte and CPUs are running agentic workloads. The calculus shifted.

If you're running your own edge cache (Varnish, nginx, Caddy, whatever), the Zstandard tradeoff is worth testing. Measure your cache hit ratio, your median object size, and your CPU headroom during peak traffic. If you have spare cycles and limited SSD, transcoding might be a win. If you're CPU-bound and SSD is cheap, it's not.

Cloudflare's advantage is scale. They can test this on 1% of traffic, measure the impact, and roll it out globally if it works. Most teams don't have that luxury. If you're prototyping this yourself, start with cache writes only (compress on fill, serve as-is). That gives you the SSD savings without the read-path CPU hit. Then measure whether the 8% size reduction justifies the complexity.

The other takeaway: Pingora is Cloudflare's internal Rust rewrite of nginx, and they're using it as a platform for this kind of experimentation. Cache transcoding isn't possible in nginx without forking or writing a custom module. In Pingora, it's a prototype they can test in production. If you're building infrastructure that needs to evolve quickly, the lesson is: own your proxy layer or accept the constraints of whatever you're running.