Agoda's Rust Proxy Boosts S3 Load Balancing

Beyond DNS: Smarter Storage Routing

Agoda's Storefront initiative is a compelling example of how internal engineering teams can innovate to overcome systemic limitations inherent in off-the-shelf solutions, particularly in the realm of large-scale data infrastructure. The core problem of DNS-based load balancing leading to uneven distribution and hotspots is a well-understood challenge in distributed systems, and their solution, Storefront, directly addresses this. The choice of Rust and Pingora is noteworthy, indicating a preference for performance, memory safety, and efficient I/O handling – critical factors for a high-throughput reverse proxy. The evolution from a simple proxy to a feature-rich component incorporating latency-aware routing, operational safeguards like IO timeouts, and sophisticated credential-less authentication via Kubernetes pod identity highlights a mature engineering process and a deep understanding of their operational needs. The integration with OpenTelemetry for telemetry is also a best practice, enabling crucial observability for such a critical component.

While the article provides a good overview, deeper technical dives into the latency-aware scoring algorithm and the specifics of the credential-less authentication mechanism would be highly valuable for other organizations facing similar challenges. Understanding the performance gains (e.g., latency reduction percentages, throughput improvements) and the operational overhead of maintaining Storefront would further solidify its impact. The reliance on VAST Data's S3-compatible endpoints implies that organizations using different storage backends might need to adapt the solution. Nevertheless, the architectural patterns and the Rust implementation are highly transferable. Storefront demonstrates a pragmatic approach to optimizing object storage performance and security in a cloud-native environment, offering a blueprint for others to consider.

Key Points

• Agoda developed Storefront, an S3-compatible reverse proxy in Rust, to improve load distribution for object storage.
• It addresses limitations of DNS round-robin, which causes uneven traffic distribution.
• Storefront uses latency-aware scoring for real-time backend selection, moving beyond simple algorithms.
• Implemented operational safeguards like IO timeouts and optimized HTTP header handling.
• Features credential-less authentication by identifying Kubernetes pods for centralized access control.
• Exposes telemetry via OpenTelemetry for performance and traffic monitoring.

---

📖 Source: Inside Agoda’s Storefront: A Latency-Aware Reverse Proxy for Improving DNS Based Load Distribution