Saving another 100TB of RAM with math (and Rust)

Cloudflare reclaimed 100TB of RAM across its global fleet by deleting 90% of its load-balancer routing points and fixing a subtle memory alignment issue in Rust. 👇

Cloudflare's internal load balancer, Pingora Backend Router, was consuming far more memory than expected. The router relies on consistent hashing to route cacheable requests to backend servers. Because different compliance rules and features require isolated rings, servers had to maintain thousands of distinct consistent hash rings in memory simultaneously.

To trim the fat, Cloudflare redesigned its open-source `pingora-ketama` library. They eliminated struct padding and mathematically proved that generating far fewer hash points per server maintains a balanced load without introducing error.

The team tackled the problem in three steps:

* Bypassing Rust struct padding: Rust aligns structs to multiples of their largest field. Storing a 4-byte hash integer forced each point struct to pad out to 8 bytes. By replacing the struct fields with raw byte arrays and custom getters, Cloudflare bypassed padding rules and immediately cut point memory by 25%. * Reducing hashes per server: Consistent hashing creates multiple virtual points per server to distribute traffic evenly, defaulting to 160 points per node. Cloudflare derived the exact mathematical error bounds and discovered that on a 32-bit hash space, high point counts suffer from the birthday paradox—causing hash collisions that drop points unpredictably. They proved they could cut the points per server by 90% with zero noticeable impact on load distribution. * Migrating without blowing up caches: Changing hash points alters request routing. Switching instantly would invalidate cache across the network and crush origin servers. Cloudflare kept both old and new rings in memory side-by-side, gradually migrating traffic data-center-by-data-center. Once fully migrated, they deleted the old rings.

Decommissioning the legacy hash rings instantly freed 100TB of RAM worldwide. The memory-compacted ring implementation is now available as a feature flag in the open-source `pingora-ketama` Rust crate.