Cogent Security's AI Platform: ClickHouse Fuels Sub-Second Vulnerability Management

ClickHouse: The Engine of AI-Native Security

Cogent Security's successful migration from Postgres to ClickHouse for their AI-native vulnerability management platform highlights a critical trend: the necessity of high-performance data infrastructure to support modern AI applications. The dramatic reduction in query latency, from seconds to sub-second, is particularly impressive, especially without relying on traditional caching layers. This demonstrates ClickHouse's inherent capabilities for handling massive datasets and complex analytical queries at speed, which is paramount in the fast-paced cybersecurity landscape where milliseconds can mean the difference between a breach and a secure system. The article effectively showcases how query-driven design, denormalization via dbt projections, and the strategic use of ClickHouse's specialized features like projections and compression codecs are foundational to achieving such performance gains. This approach is not merely an optimization; it's an architectural shift that directly enables the 'machine speed' required to counter AI-enabled attackers.

The integration of an agentic loop architecture with ClickHouse is another standout innovation. The improvement in the Chart Agent's accuracy from 40% to 94% by leveraging ClickHouse projections and enabling live SQL execution is a testament to the synergy between sophisticated AI logic and a robust, responsive data backend. The concept of 'interleaved thinking' in the agent, combined with real-time data access, allows for dynamic adaptation and validation, which is crucial for generating accurate insights from vast and complex security data. The development of the Ontology Service, providing a 'one source of truth' for data models and semantics, further addresses a common pain point in complex data environments, making the data model 'agent-friendly' and adaptable to unique customer environments. This holistic approach, from data ingestion to AI-driven analysis and reporting, is a compelling blueprint for building scalable and intelligent enterprise solutions.

While the article celebrates significant achievements, a potential limitation could be the inherent complexity of setting up and managing a ClickHouse instance, especially when employing advanced features like projections and integrating with tools like dbt and Iceberg. The article touches upon the expertise required for 'query-driven design' and optimizing compression codecs, suggesting a steep learning curve for teams without deep database engineering experience. Furthermore, the 'per-tenant overlays' in the Ontology Service, while powerful, imply a significant engineering effort to maintain and scale. The success of Cogent Security is clearly tied to their specialized engineering team and their deliberate architectural choices. For organizations looking to replicate this, the initial investment in expertise and infrastructure setup will be substantial. The reliance on Iceberg as the source of truth also introduces another layer of complexity and potential points of failure to manage.

Key Points

• Cogent Security migrated from Postgres to ClickHouse for its AI-native vulnerability management platform, achieving sub-second query latency.
• The platform handles billions of security findings, with significant performance gains demonstrated at 100M and 500M rows.
• ClickHouse's speed is crucial for enabling fast, iterative agentic loops in AI applications.
• Innovations include query-driven design, denormalization via dbt projections, and specialized ClickHouse features like projections and compression.
• The Chart Agent's accuracy improved from 40% to 94% by integrating an agentic loop with ClickHouse, including live SQL execution and interleaved thinking.
• The Ontology Service provides a unified source of truth for data models and semantics, making data more 'agent-friendly'.

---

📖 Source: Intelligent security at ClickHouse speed: How Cogent Security built an AI-native vulnerability management platform