Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction

Based on the paper, here is the summary:

This paper introduces DARC (Diagnosis-guided Agent Recovery and Correction), a framework that recasts agent self-correction from indiscriminate context expansion to diagnosis-guided recovery. The paper states: Our insight is that development-set failures can recover part of the missing diagnostic substrate by deciding which recovery interventions are admissible before test-time correction.

The motivation is that coding agents benefit from compiler-like feedback (type errors, failing assertions, timeouts) that makes failures actionable, but broad language-agent tasks often expose only a coarse task failure. The paper identifies three concrete problems with generic recovery playbooks: "1 Intervention Mismatch: a recovery signal chosen without diagnosis often cannot address the failure at hand... 2 Recovery Interference: irrelevant procedures can pull an embodied agent away from state-compatible actions... 3 Uncontrolled Recovery Cost: triggering every recovery signal on every failure inflates environment steps, retrieval budget, and inference tokens."

DARC's approach is described as: a diagnosis-guided recovery harness that profiles task-family failure modes, prunes mismatched interventions from a shared recovery library, and freezes a verifier-selected success-cost policy for deployment. The paper explains: This causal order makes correction selective: the harness first determines what kind of failure can be repaired, then decides how much recovery evidence to spend.

The framework constructs a recovery harness defined as a tuple H m = (m, R m, π m), where R m is the set of admissible interventions for a diagnosed failure mode and π m is the ordered deployment policy. The recovery objective is: max E x∼X [s(x, π)] − λ max(0, E x∼X [c(x, π)] − τ free).

DARC works in two stages: "The diagnosis stage turns development failures into a task-family failure mode and restricts the recovery interventions that may be used. The policy-distillation stage freezes a cost-aware fallback policy over that restricted intervention set. The diagnosis is frozen at the task-family level: This frozen, task-family-level diagnosis makes the harness auditable, avoids test-time updates, and prevents the policy from observing test labels."

The paper evaluates DARC in three settings: "ALFWorld tests action validity, where recovery should constrain state-incompatible actions rather than expose unrelated manuals. AppWorld tests procedural breadth, where recovery should reveal richer API workflow knowledge only when cheaper procedural sources fail. XBRL Finance tests format precision, where recovery should tune demonstration evidence without expanding into a broad playbook."

Key results include: On ALFWorld with DeepSeek-V4-Flash, valid-unseen success increases from 39.55% for the base agent and 54.48% for ACE to 90.30%. On AppWorld, the procedural recovery policy gives the strongest gains on Test-Normal and improves average TGC for all three base models. On Finance, the format-precision policy reaches 94.50% macro accuracy with DeepSeek-V4-Flash, compared with 80.50% for ACE and 74.00% for MIPROv2.

The paper addresses a fairness concern about ALFWorld gains through a matched-information fairness evaluation: Granting the base agent the same parsed metadata (task type, target object, movable and goal containers) improves success only marginally (39.55%→43.28%), so information access alone is not the driver. The paper concludes: The ALFWorld improvement is therefore attributable to the quality of the diagnosis-guided ranking of already-available admissible actions, not to privileged information or restriction alone.

A factorial decomposition shows the gain is an interaction effect: "Neither component works alone: the recovery prompt on the full action set is worth −0.75 pp, the ranked action view without the recovery prompt is worth +3.73 pp... The two components together are worth +49.26 pp, an interaction of +46.27 pp."

The ablation study comparing correct, generic, and mismatched policies shows: the correct recovery policy (aligned with the diagnosed failure mode) outperforms both a generic playbook and a mismatched intervention set across all evaluated settings. For AppWorld Test-Normal, correct diagnosis reaches 70.24% TGC compared to 61.90% (generic) and 64.88% (mismatched).

Cost efficiency results show: Relative to the base agent, DARC reduces average environment steps by 54.2% and invalid actions by 92.9%. It also improves success per 100 steps from 1.16 to 6.02.

Cross-task transfer results show: Formula-to-FiNER reaches 89.50% versus 90.00%, and FiNER-to-Formula reaches 96.00% versus 99.00%. For ALFWorld, the same action-validity policy transfers from valid seen to valid unseen and reaches 90.30%.

The paper also includes a weight-space training extension: DARC-GRPO improves macro success by 4.74 percentage points, while DARC-OPSD yields a smaller 2.19 pp gain, though this is treated as a proof of concept rather than a fully verified training recipe.

The paper concludes: "By profiling task-family failure modes to restrict admissible interventions and distilling a cost-aware policy, DARC improves success across ALFWorld, AppWorld, and XBRL Finance while mitigating the overhead and interference of generic playbooks. Future work includes extending our static diagnosis to dynamic, instance-level routing for mixed-mode failures, developing more sample-efficient policy distillation methods, and expanding full-library cascade comparisons to broader domains."

Improvements to AI systems:

1. Diagnosis-gated recovery selection: Instead of applying generic recovery strategies on every failure, the AI system first classifies the failure mode (e.g., action-invalid, procedural-knowledge-gap, format-mismatch) using a frozen, task-family-level profile. It then restricts recovery to only admissible interventions for that mode, pruning mismatched ones. This prevents wasted steps, tokens, and retrieval calls on irrelevant fixes.

2. Cost-aware fallback policy distillation: The system learns an ordered deployment policy over the restricted intervention set, balancing success against a free-cost threshold (τ free). It stops recovery once marginal cost exceeds expected benefit, reducing average environment steps by 54.2% and invalid actions by 92.9% in the paper’s evaluations.

3. Matched-information action ranking: For embodied or action-validity tasks, the system ranks already-available admissible actions by diagnosis-guided relevance (e.g., task type, target object, movable/goal containers) rather than expanding context with external manuals. This yields a +46.27 pp interaction effect when combined with a recovery prompt, far exceeding either component alone.

4. Cross-task transferable recovery policies: The system freezes a diagnosis at the task-family level (e.g., action-validity for ALFWorld) and reuses it across similar instances or domains (valid seen → valid unseen), achieving 90.30% success without test-time updates. This makes recovery auditable and avoids label leakage.

5. Selective procedural knowledge activation: For tasks requiring broad API workflows (e.g., AppWorld), the system only reveals richer procedural knowledge when cheaper sources (e.g., direct tool calls) fail, preventing recovery interference and preserving state-compatible actions.

What the improved AI system can do:

• Reduce recovery overhead: It can cut environment steps by over half and invalid actions by over 90% while improving success per 100 steps from 1.16 to 6.02, by pruning mismatched interventions and enforcing a cost budget.

• Achieve high accuracy on precision-critical tasks: It can reach 94.50% macro accuracy on XBRL Finance (vs. 74–80% for baselines) by tuning demonstration evidence to the diagnosed format-precision failure mode, without expanding into a broad playbook.

• Maintain robust performance across diverse benchmarks: It can improve valid-unseen success from 40% to 90.30% on ALFWorld, and raise Test-Normal TGC to 70.24% on AppWorld (vs. 62–65% for generic/mismatched policies), all while using fewer resources.

• Transfer recovery knowledge safely: It can reuse a frozen diagnosis across task variants (e.g., Formula→FiNER, FiNER→Formula) with minimal performance loss (89.5–96% vs. 90–99% for same-task), enabling deployment in new but related environments without retraining.

• Provide auditable, label-safe corrections: Since diagnosis is frozen at the task-family level and never observes test labels, the system can be inspected for which interventions are admissible and why, making it suitable for regulated or safety-critical applications.