Agoda's DragonflyDB Leap: Outperforming 72 SQL Shards

Sharding SQL's Limits with DragonflyDB

Agoda's migration from a 72-shard SQL Server price cache to DragonflyDB is a compelling case study in modernizing data infrastructure for high-throughput, low-latency scenarios. The key takeaway is the clear demonstration of DragonflyDB's ability to handle massive read and write volumes (300k reads, 1.5M writes/sec) with significantly improved latency, directly addressing the limitations of their previous SQL Server setup. The incremental migration strategy, including dual reads and A/B testing, is a best practice that minimizes risk. Furthermore, the decentralized failure detection mechanism for high availability is an elegant solution that avoids single points of failure common in traditional architectures. This move highlights a broader trend of organizations moving away from complex, manually managed RDBMS sharding for caching layers towards more scalable, purpose-built in-memory solutions like DragonflyDB.

The innovation lies not just in the choice of DragonflyDB but in the systematic approach to validation and migration. Agoda's decision to evaluate DragonflyDB against their specific workload using memtier_benchmark rather than relying solely on vendor claims is commendable. The architecture's success in reducing P99 read latency by approximately eightfold and handling the demanding workload with sub-10ms P99 latency for writes underscores the performance gains. The inherent advantages of DragonflyDB, such as its shared-nothing, multithreaded architecture and Redis compatibility, made it a natural fit. The article implicitly points to the operational burden of managing 72 SQL Server shards, including complex scaling, remapping, and data migration, which DragonflyDB's cluster-based scaling aims to simplify.

While the article is overwhelmingly positive, potential limitations or concerns might include the learning curve associated with a new datastore for the engineering team, long-term operational costs comparison (though explicitly stated as more cost-effective than scaling SQL Server), and potential vendor lock-in if DragonflyDB's ecosystem is not as mature as SQL Server's. The article could also benefit from more granular details on the specific challenges encountered during the dual-read phase or the A/B testing rollout. However, for companies facing similar scaling bottlenecks with relational databases for caching, especially those with high write volumes, this case study offers a strong blueprint and a compelling alternative.

Key Points

• Agoda replaced a 72-shard SQL Server price cache with DragonflyDB for improved scalability and performance.
• The new system handles ~300,000 reads and 1.5 million writes per second with ~8ms P99 read latency.
• DragonflyDB's shared-nothing, multithreaded architecture and Redis compatibility were key factors.
• Incremental migration, dual reads, and A/B testing were used to ensure a smooth transition.
• Decentralized failure detection provides high availability without a central coordinator.
• The migration significantly reduces operational complexity and improves cost-effectiveness compared to scaling SQL Server.

---

📖 Source: Agoda Replaces 72-Shard SQL Server Price Cache with DragonflyDB