HuggingFace Papers
★ 9/10 signal
AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
agenticevalmulti-agent
Summary
This paper introduces AgentDebugX, an open-source toolkit for debugging LLM agents that organizes the process into a closed loop of Detect, Attribute, Recover, and Rerun. At its core is DeepDebug, a multi-turn diagnostic agent that performs root-cause analysis through global trajectory understanding, structure-guided investigation, and cross-examination. On the Who&When benchmark, DeepDebug achieves 28.8% strict agent-and-step attribution accuracy with a qwen3.5-9b backbone, compared to 21.7% for the strongest single-pass baseline. On GAIA, it repairs 13 of 73 failed tasks in a single rerun, improving overall accuracy from 55.8% to 63.6%, more than doubling the repairs of decoupled self-correction baselines.
Problem
Debugging LLM agents is difficult because the step where a failure becomes visible is often not the step that caused it. Existing observability tools replay execution traces but do not help identify the root cause, while self-correction methods are more reliable when the error's location is already known. This creates a gap between observing a failure, attributing its root cause, proposing a repair, and verifying that repair through a rerun.
Method
The authors developed AgentDebugX, a framework that processes agent executions through a four-stage loop: Detect, Attribute, Recover, and Rerun. For complex failures, it uses DeepDebug, a multi-turn diagnostic agent that performs a global read of the execution trace, a structure-guided investigation (e.g., bisecting the trace), and cross-examination of candidate error steps to produce a diagnosis and a suggested fix. The system's attribution accuracy was evaluated on the Who&When benchmark, and its end-to-end recovery capability was tested on the GAIA benchmark.
Details
Failure Attribution on Who&When:
DeepDebug was evaluated on all 184 traces from the Who&When benchmark using a qwen3.5-9b policy. It outperformed all single-strategy baselines across five metrics.
| Method | Agent Acc. | Step (exact) | Step (±1) | A+S (exact) | A+S (±1) |
|---|---|---|---|---|---|
| Rule heuristic | 14.1% | 1.6% | 6.5% | 1.6% | 2.7% |
| All-at-Once | 47.8% | 22.3% | 35.3% | 21.7% | 23.9% |
| Step-by-Step | 41.8% | 18.5% | 38.0% | 17.9% | 19.6% |
| Binary-Search | 41.8% | 17.4% | 38.6% | 17.4% | 20.1% |
| DeepDebug | 56.0% | 28.8% | 44.0% | 28.8% | 32.1% |
- With a qwen3.6-27b backbone, DeepDebug achieved 38.0% strict agent-and-step accuracy versus 36.4% for the best baseline.
- The performance gain is concentrated in traces longer than 40 events.
End-to-End Recovery on GAIA:
The system was tested on 73 tasks from the GAIA validation set that a vanilla qwen3.5-9b agent failed. DeepDebug's diagnosis and recovery proposal were compared against three decoupled self-correction baselines after a single rerun.
| Method | Repaired Cases | L1 Acc. | L2 Acc. | L3 Acc. | Overall Acc. |
|---|---|---|---|---|---|
| Vanilla qwen3.5-9b | — | 77.4% | 48.8% | 34.6% | 55.8% |
| + CRITIC | 4/73 | 81.1% | 51.2% | 34.6% | 58.2% |
| + AutoManual | 5/73 | 79.2% | 53.5% | 34.6% | 58.8% |
| + Reflexion | 6/73 | 77.4% | 55.8% | 34.6% | 59.4% |
| + DeepDebug (ours) | 13/73 | 81.1% | 61.6% | 34.6% | 63.6% |
Cost and Ablations:
- On a 25-trace sample from Who&When, DeepDebug averaged 12.8K tokens per diagnosis, compared to 8.1K for a single whole-trace pass (a 1.6x increase).
- An ablation on gpt-5.4-mini showed that replacing DeepDebug's structure-guided investigation with a second global search dropped strict accuracy by 4.8 points.
- The benefit of DeepDebug's multi-turn process is model-dependent; on some hosted backbones (e.g., gemini-3.5-flash), a single global reading was already stronger and adjudication did not improve it.
What's new
The primary contribution is AgentDebugX, a unified, open-source toolkit that connects the full agent debugging lifecycle from failure detection and root-cause attribution to verified recovery. It introduces DeepDebug, a novel multi-turn diagnostic agent for root-cause analysis. The system also proposes a framework for a shareable 'Error Hub' to create a corpus of diagnosed failures.
Limitations
The evaluation does not measure developer debugging time or the console's usability. The performance gains from DeepDebug are model-dependent and not uniformly beneficial across all backbones. The GAIA recovery experiment evaluates the complete system recipe on a fixed set of failed tasks, rather than isolating the effect of attribution alone, and is not a blind single-shot evaluation. The Error Hub's retrieval and taxonomy induction features are implemented but not yet evaluated. The default data scrubber redacts known patterns but cannot guarantee removal of all sensitive content from shared traces.
Conclusion
The authors present AgentDebugX, a closed-loop debugging framework that connects failure detection, root-cause attribution, recovery, and rerunning for LLM agents. The core diagnostic method, DeepDebug, demonstrates that improving fault attribution can translate into measurable gains in downstream task recovery. The authors hope AgentDebugX will provide a practical foundation for debugging LLM agents and facilitate future research into reliable agent development.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.