ClickStack MCP: AI Agents Investigate Incidents Faster

Bridging AI and Observability

The ClickStack MCP server represents a compelling step forward in making AI agents more effective for production incident investigation. By abstracting raw SQL into higher-level, purpose-built tools, ClickHouse addresses the core challenges of schema discovery, query construction, and multi-step workflow orchestration that plague direct SQL access for LLMs. The introduction of hdx-evals is particularly noteworthy, offering a transparent and reproducible benchmarking framework. The deterministic data seeding, blind LLM judging, and isolation of agent execution environments are crucial for ensuring the validity and trustworthiness of the reported metrics (18% more accurate, 26% fewer tool calls, 2.4x more consistent). This detailed explanation of their methodology lends significant credibility to their claims.

However, a key limitation, though perhaps inherent to the current state of LLM interaction with complex systems, is the reliance on LLM reasoning for final answer scoring. While the blind LLM judge is an improvement, the inherent subjectivity and potential for bias in LLM evaluations, even with clear rubrics, remains a concern. The article acknowledges this by combining LLM judging with programmatic checks and tool-error penalties, but the weight given to the LLM judge (60%) means its performance is paramount. Furthermore, the focus is exclusively on Claude agents, and while this might be due to current partnerships or testing focus, evaluating performance with a broader range of LLMs would provide a more comprehensive picture of the MCP's general applicability. The article is light on details regarding the specific LLM prompts and constraints used beyond the general SRE role and tool call budget, which could offer more insight into how agents are guided to utilize the MCP effectively.

Key Points

• ClickStack MCP server provides purpose-built observability tools for AI agents, abstracting raw SQL into higher-level primitives.
• Benchmarking framework hdx-evals ensures reproducibility through identical synthetic telemetry, blind scoring, and isolated agent execution.
• Reported metrics show ClickStack MCP is 18% more accurate, uses 26% fewer tool calls, and is 2.4x more consistent than direct ClickHouse SQL access.
• Synthetic data generation is deterministic and designed to avoid pre-existing knowledge of failure modes in LLMs.
• hdx-evals allows for evaluating new LLM models against existing scenarios before production deployment.
• The benchmarking process includes multiple stages: provisioning, schema creation mirroring production, and deterministic data generation.
• Agent execution is sandboxed with strict filesystem restrictions and a deny-list of non-investigation tools to prevent cheating.
• Scoring combines programmatic checks, a blind LLM judge (weighted 60%), and a tool-error penalty.

---

📖 Source: Benchmarking the ClickStack MCP Server with hdx-evals