iFood's ClickHouse Leap: Agentic Security at Scale

ClickHouse Powers Agentic Security

iFood's case study offers a compelling narrative of overcoming significant data scaling and cost challenges by migrating their security platform to ClickHouse Cloud. The most striking takeaway is the enablement of 'agentic threat hunting,' transforming a week-long manual process into a two-hour automated workflow. This highlights the critical role of performant databases in modern AI-driven security operations, where rapid data access and massive parallel processing are no longer luxuries but necessities. The reported 9-16x query speed improvement at 40-50% cost reduction, coupled with near real-time data freshness, underscores the tangible benefits of choosing the right analytical database for specialized workloads. The integration with tools like ClickPipes for ingestion and Querybook for investigation, alongside the strategic use of Langfuse for AI observability, paints a picture of a mature and forward-thinking technical implementation.

However, a deeper dive into the 'agentic' aspect would be beneficial. While the article mentions AI agents sweeping logs and testing hypotheses, the specifics of how these agents are built, orchestrated, and how ClickHouse specifically facilitates their parallel execution and learning loop (beyond just fast queries) remain somewhat high-level. Understanding the architecture of these agents and their interaction patterns with ClickHouse would add significant technical depth. Furthermore, while the cost savings are impressive, a more detailed breakdown of the cost comparison between Databricks and ClickHouse Cloud (considering factors like data egress, compute, and storage) would provide a more robust financial justification. The reliance on AWS S3 as a staging area is standard, but how ClickHouse Cloud's native ingestion layer (ClickPipes) optimizes this flow and manages potential bottlenecks could be elaborated upon. The long-term retention goal of 6-12 months for such high-volume data (1.6 TB/day raw EDR logs) is ambitious and a testament to ClickHouse's capabilities, but potential challenges in managing, querying, and the associated storage costs at this scale should be acknowledged, even if the current solution is cost-effective.

Key Points

• iFood built its agentic security platform on ClickHouse Cloud, replacing a Databricks-based solution.
• The migration resulted in 9-16x faster queries and 40-50% cost reduction.
• Near real-time data freshness was achieved, moving from hourly batch updates.
• Enabled 'agentic threat hunting,' reducing investigation time from a week to approximately 2 hours.
• ClickHouse Cloud supports handling over 30 TB of data in parallel for threat hunting.
• The architecture uses AWS S3 for log storage, ingested via ClickPipes into ClickHouse Cloud.
• SQL is the primary query language, with Querybook used for investigation notebooks.
• Langfuse is employed for AI observability to tune threat hunting agents.
• Plans to migrate CDN and API logs from OpenSearch to ClickHouse, expecting significant cost savings and expanded analysis capabilities.

---

📖 Source: How iFood built its agentic security platform on ClickHouse Cloud