Zalando's 1M RPS Client-Side Load Balancer

In-Process Routing at Extreme Scale

Zalando's development of an in-process client-side load balancer for their Product Read API is a testament to sophisticated engineering addressing a very specific, high-scale problem. The key insight is the strategic decision to move routing logic from a shared edge load balancer (Skipper) into the application itself for high fan-out internal traffic. This move enabled them to decouple their service's performance from external infrastructure latency spikes, leading to more predictable latency, significant infrastructure cost savings (reducing Skipper fleet size and daily costs), and improved visibility into failure origins. The meticulous reimplementation of Skipper's algorithm (xxHash64 with virtual nodes) during migration is particularly noteworthy, ensuring cache consistency and a seamless transition. The adoption of a watch-based Kubernetes informer over polling and the development of fast, reversible deploys highlight a commitment to operational efficiency. The article effectively communicates the trade-offs and the rationale behind building a custom solution only when existing tools, even internal ones like Skipper, proved insufficient for their extreme edge case. The emphasis on 'lessons learned' and the caution against building custom solutions for typical scenarios are crucial takeaways, positioning this as an advanced architectural pattern rather than a general recommendation.

However, the decision to build an entirely new, internal-only load balancing module, even while Skipper is open-source and developed internally, raises questions about internal collaboration and potential for code reuse. While the goal was to address a specific pain point, the effort involved in rebuilding and maintaining a complex component like a load balancer is substantial. The initial attempt at Availability Zone (AZ)-aware routing, which fragmented caches and caused read explosions, serves as a cautionary tale about the complexities of distributed systems optimization. The reliance on a single jittered retry and FIFO overload shedding, while effective, might be considered a minimal set of resilience patterns for such a critical path. Furthermore, the fact that the new client-side load balancer remains internal and not open-sourced might limit its broader adoption and community-driven hardening, despite the impressive results achieved by Zalando. This approach is a prime example of solving an extreme problem with a bespoke solution, but it underscores the importance of thoroughly evaluating existing, mature solutions before embarking on custom development.

This architecture would primarily benefit organizations operating at a similar scale and facing similar high-fan-out internal traffic patterns. E-commerce platforms, large-scale microservice architectures, and any system with a critical internal API handling millions of requests per second with strict latency requirements could learn from Zalando's approach. Developers working on performance-critical systems, distributed systems design, and load balancing strategies will find the technical details on consistent hashing, virtual nodes, cache-aware scaling, and rollout strategies particularly valuable. The article implicitly suggests that for most, leveraging mature proxy solutions like Envoy or even their own internal Skipper would be sufficient. However, for those at the bleeding edge of scale, understanding how Zalando tackled the limitations of shared infrastructure and chose to embed intelligence within the client offers a compelling blueprint for pushing performance boundaries.

Key Points

• Zalando developed an in-process client-side load balancer to handle 1 million requests per second for their Product Read API.
• The primary motivation was to overcome latency spikes from shared infrastructure (Skipper) and improve predictability for high fan-out internal traffic.
• Key benefits include more predictable latency, a significant drop in infrastructure costs (e.g., reducing Skipper fleet from 50+ to 8 pods), and better failure origin visibility.
• The solution reimplemented Skipper's exact algorithm (xxHash64, 100 virtual nodes) for seamless migration and cache consistency.
• Engineering decisions involved consistent hashing, occupancy-based load balancing, cache-aware scaling, and a phased rollout strategy.
• Observability was improved with a watch-based Kubernetes informer and richer logging.
• The article emphasizes that building a custom client-side load balancer is an extreme edge case solution, and for most, mature proxy solutions like Skipper or Envoy are recommended.
• Challenges included initial attempts at AZ-aware routing causing cache fragmentation and read explosions.

---

📖 Source: How Zalando Built an In-Process Client-Side Load Balancer for One Million Requests per Second