ZGateway: Learnings from Putting a Proxy in Front of ZippyDB

Meta cut its database connection counts by 98% while processing over 1 billion operations per second—all by putting a proxy layer in front of its key-value store for just 6% CPU overhead. 👇

Before the proxy, over a million client hosts talked directly to Meta's ZippyDB key-value storage fleet. A single client held tens of thousands of outbound TLS connections across distinct database shards. When client fleets deployed updates or restarted services, those connections dropped and reconnected all at once. The resulting connection storms regularly exhausted file descriptors, triggered out-of-memory crashes, and drove entire database clusters into reboot loops.

To fix this, Meta built ZGateway, a stateless proxy tier that sits between client applications and database servers. It collapses the chaotic, many-to-many connection mesh into two tightly controlled hops.

Because ZGateway sits in the path of every caller, it can optimize traffic in ways no individual client library could:

Sticky regional routing: Clients connect only to a small pool of local ZGateway hosts. The proxy handles TLS termination, tenant authorization, and key-to-shard mapping.

Cross-client batching: A shared batcher holds incoming requests for a brief linger window. It merges reads and writes from completely unrelated client processes headed to the same shard into a single backend RPC, amortizing serialization and network overhead.

Request coalescing: If thousands of clients query the exact same key at the same moment, ZGateway issues a single read to the database and fans the result back out to every caller, preventing hot-key stampedes.

Discriminant load shedding: Incoming work maps to per-tenant buckets that drain round-robin. When a noisy neighbor spikes, its specific bucket fills and sheds excess requests, leaving all other tenants completely unaffected.

Adaptive load balancing: A control plane regularly measures host CPU utilization and nudges weighted routing tables to steer traffic away from hot nodes without causing aggressive cache-thrashing reshuffles.

ZGateway reduced total persistent database connections by 19x and per-host connection counts by 97–98%. It currently handles 40% of Meta's total ZippyDB workload at over 1 billion operations per second with only 6% CPU overhead. During overload tests past 90% CPU utilization, the system isolated and shed load for 6 noisy tenants while 1,344 other tenants ran smoothly with 99.9% request success rates.