AI Falls Short in Replacing Human Site Reliability Engineers
Artificial intelligence companies are increasingly promoting autonomous site reliability engineer (SRE) agents – AI designed to investigate production incidents in place of human engineers. However, a recent benchmark conducted by Datadog reveals that current AI models are not yet capable of outperforming the engineers they are intended to replace.
The benchmark, known as ARFBench (Anomaly Reasoning Framework Benchmark), was developed jointly by Datadog and Carnegie Mellon University. It is based on 63 real production incidents, drawing data from engineers’ Slack threads during live emergencies. The benchmark comprises 750 multiple-choice questions covering 142 monitoring metrics and 5.38 million data points, with each question verified manually. The data used is entirely real-world, avoiding synthetic data or textbook scenarios.
Researchers note that system outages result in trillions of dollars in losses annually, and the benchmark aims to assess whether AI can effectively mitigate these losses.
The study’s authors highlight the uncertainty surrounding the ability of modern foundation models to reliably answer the time series questions that engineers routinely face during incident response.
The questions in ARFBench are categorized into three tiers. Tier I assesses the existence of an anomaly in a chart. Tier II focuses on identifying the start time, severity, and type of the anomaly. Tier III, the most challenging, requires cross-metric reasoning to determine if one chart is causing a problem in another. It is in Tier III that AI models struggle. For example, GPT-5 achieved only a 47.5% F1 score on Tier III questions, a metric that penalizes models for selecting the most common answer.
GPT-5 led the existing models with an accuracy of 62.7%, compared to a 24.5% success rate for random guessing. Gemini 3 Pro scored 58.1%, Claude Opus 4.6 scored 54.8%, and Claude Sonnet 4.5 scored 47.2%.
In contrast, domain experts achieved 72.7% accuracy, while non-domain experts – time series researchers at Datadog without extensive observability experience – scored 69.7%.
No AI model surpassed either human baseline.
The top-performing model overall was Datadog’s hybrid approach, Toto-1.0-QA-Experimental, which combines their internal time series forecasting model, Toto, with Qwen3-VL 32B. Toto-1.0-QA-Experimental achieved 63.9% accuracy, slightly exceeding GPT-5 while utilizing fewer parameters. Specifically, it outperformed all other models by at least 8.8 percentage points in F1 when identifying anomalies.
The study’s findings support the expectation that a domain-specific model, trained on observability data, would outperform a general-purpose system in this specific task.
The most significant finding of the research is not which model scored highest.
Researchers observed that leading models and human experts exhibit substantially different error profiles, suggesting their strengths are complementary. AI models are prone to hallucination, missing metadata, and losing domain context, while humans may misinterpret timestamps or struggle with complex instructions. The errors made by each group rarely overlap.
By creating a theoretical “Model-Expert Oracle” – a perfect judge that always selects the correct answer between the AI and the human – the researchers achieved 87.2% accuracy and an 82.8% F1 score, significantly higher than either approach alone.
This research provides a documented target, based on real-world emergencies, that quantifies the potential performance gains of human-AI collaboration. The leaderboard is available on Hugging Face, with GPT-5 currently at 62.7% accuracy, and the potential ceiling at 87.2%.
Leave a Reply