“LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction.” [5]
The agent fails.
The operator says try again.
The second attempt still has no evidence.
In The Agent Was Right. The Evidence Wasn’t. we showed that an MCP-Atlas answer can look finished while the recorded tool trace does not support the claims, and that the reverse also happens: the trace can already contain the evidence while the answer leaves it out. Those were not only scoring disagreements. They were different engineering failures. A claim present in the answer but absent from the trace needs evidence recovery. A claim present in the trace but missing from the answer needs synthesis, not another tool call. A claim missing from both needs a new plan.
The next question was whether the execution trace could be used to choose the right recovery, instead of asking the model to try again.
Previous work has shown that language models are weak intrinsic self-correctors when they are given no reliable external signal [5]. Toolformer and ReAct established that models can decide which APIs to call and can interleave reasoning with those calls [1, 2]. AgentBoard noted that most evaluation still focuses on the final success rate, revealing little about the process [3]. MCP-Atlas moved the target further downstream: from valid API calls alone toward claim-aware stopping rules and trajectory-grounded answer verification [4]. If the trace tells us which evidence is missing, which call failed and which claim remains unsupported, then a retry that ignores that trace is discarding the most useful signal the system has.
This experiment keeps the same product roles as the diagnostic batch. SourceryKit captures the live MCP tool-call evidence by interception. MASFaultsSimulator (MFS) compares the claims with that evidence and constructs a correction packet. The model executes the recovery. Provably issues an external receipt when the SDK returns a receipt identifier; where it does not, the experiment stores a labelled local hash attestation of the intercept summary. We therefore do not claim that SourceryKit diagnosed the logical error itself. SourceryKit preserved the run. MFS used that run to select the correction. The model tried to recover.
We again used single-agent systems (SAS) only. The point is to isolate one agent, one immutable execution trace and one bounded correction before introducing multi-agent systems (MAS) delegation, handoffs and coordination failures. We reused the same frozen 12-task MCP-Atlas panel from the previous article: four simple, four medium and four hard tasks, stratified by the reference trajectory. The original core batch contained 13 models and 156 runs; the published diagnostic article also reported two supplementary models, bringing its total to 180. Inline healing was disabled during the base runs. Agents saw only the task prompt. Ground-truth claims were not shown to them.
From those failures we created a correction cohort of 60 traces. The original locked cohort contained 24 traces, balanced across four failure strata. A prespecified precision rule required expansion if the primary comparison had fewer than 10 discordant pairs or if the Wald confidence interval was wider than 0.35 (a Wald interval gives a range of plausible values for the true percentage-point improvement; we report it because the experiment’s prespecified stopping rule used the width of that interval to decide whether more traces were needed). The expansion therefore added 36 additional missing-evidence/tool traces without reshuffling the original 24. The final cohort is intentionally enriched for the failure mode for which trace-guided recovery is designed. The aggregate 60-trace rate should not be read as an estimate of how often all agent failures can be healed in production.
Each of the 60 traces was replayed under three correction conditions, producing 180 correction episodes. The attempt budget was three. Attempt 1 could reuse existing intercepted evidence only. Attempt 2 could make new live MCP calls. Attempt 3 could re-plan dependencies if attempt 2 had made progress.
Condition
What the agent sees
Blind retry
The task plus an instruction to review the work and try again. No failed-claim IDs, claim-state, evidence excerpt or recovery route.
Claim-ID feedback
The task plus the identifiers of failed claims (the “Atlas-claim-only” condition in the experiment logs). No evidence excerpt, claim-state or recovery route.
SourceryKit-guided
A trace-informed MFS packet: failed claims, excerpts of intercepted evidence and a recommended recovery route.
Table 1. The three correction conditions. The model and task stay fixed; only the recovery information changes.
The primary endpoint is grounded-run repair: after correction, every expected claim must be backed by intercepted tool evidence. This is the trace-supported view from the first article, now applied after a recovery attempt. The secondary endpoint is the MCP-Atlas answer-only pass at the 0.75 claim-coverage threshold. We report both because they answer different questions. One asks whether the run contains the evidence. The other asks whether the final answer says enough of the right things.
180 of 180 correction episodes completed. There were no provider errors and no intercept-incomplete stops.
Condition
Grounded-run repair
Rate
Answer-only pass
Rate
Avg. attempts
Blind retry
18 / 60
30.0%
19 / 60
31.7%
2.33
Claim-ID feedback
19 / 60
31.7%
18 / 60
30.0%
2.27
SourceryKit-guided
32 / 60
53.3%
21 / 60
35.0%
2.27
Table 2. Repair under the two evaluation views. Each condition was applied to the same 60 traces. Source: SourceryKit/MFS healing experiment.
Figure 1. Full-run repair rate by correction condition under trace-supported evaluation (primary) and answer-only MCP-Atlas evaluation at the 0.75 threshold. Source: SourceryKit/MFS healing experiment.
On the primary endpoint, the trace-informed packet repaired 32 of 60 traces, or 53.3%. Blind retry repaired 18 of 60, or 30.0%. The difference is +23.3 percentage points. The preregistered 95% Wald interval is [+6.2%, +40.5%]. Against claim-ID feedback the difference is +21.7 points, with a 95% Wald interval of [+4.4%, +38.9%]. The precision rule is therefore satisfied and no further expansion is recommended.
The most relevant number is 14.
Those are the traces that were repaired by the SourceryKit-guided condition and not by blind retry. Blind-only recoveries on the same endpoint: zero. Eighteen traces were repaired under both conditions. Twenty-eight were repaired under neither. Because every condition is applied to the same underlying trace, the experiment is paired. On the guided-versus-blind comparison the 14-to-0 discordance gives an exact two-sided McNemar p-value of approximately 0.00012 (McNemar's test asks whether paired binary outcomes differ on the same traces, using only the cases where the two conditions disagree). The result is not only that two percentages differ; on the same failures, the guided condition repeatedly succeeded where blind retry did not.
Figure 2. Paired outcomes for SourceryKit-guided versus blind retry on grounded-run repair (n = 60). Source: SourceryKit/MFS healing experiment.
The 18 traces repaired under both guided and blind retry do not identify a guidance effect. Fifteen of those were also repaired by the claim-ID condition. The identifying evidence is in the discordant pairs. In the guided-only cases, recovery typically reaches attempt 2, where new live MCP calls are permitted. SourceryKit intercepts the call. MFS re-verifies the claims against the new intercept. Blind retry, given only “try again,” does not find the same route.
The lift is not a general property of “healing.” It is concentrated where the missing thing is evidence.
Figure 3. Trace-supported repair rate by failure stratum. The three conditions separate only on missing-evidence/tool recovery. Source: SourceryKit/MFS healing experiment.
In the enriched missing-evidence/tool stratum, guided recovery repaired 23 of 42 traces. Blind retry repaired 9. Claim-ID feedback repaired 10. The failed-claim identifiers alone are therefore almost indistinguishable from asking the model to retry. What changes the outcome is not simply telling the model that something is wrong. The treatment gives MFS enough execution context to construct a recovery route from what actually happened.
This distinction matters for the causal claim. The experiment does not isolate “the trace” as one atomic feature. The guided packet contains failed-claim state, evidence excerpts and a recommended route. What the data establish is narrower and more useful operationally: a recovery packet constructed from the grounded execution trace outperforms blind retry for missing-evidence/tool failures. SourceryKit supplies the evidence record. MFS turns that record into a correction decision. The model executes it.
The same pattern appears when the traces are grouped by the task tiers that were locked before correction. The correction cohort is not balanced across tiers because the precision expansions added missing-evidence traces, which fell mostly on simple and hard chains. Difficulty is therefore descriptive here, not another randomized treatment.
Figure 4. Repair by locked task tier. Simple: 3 required tools and 6 reference steps. Medium: 4 tools and 8 steps. Hard: 4 tools and 10–14 steps. Source: SourceryKit/MFS healing experiment.
Simple chains show the largest separation: 15 of 21 guided traces repair, against 7 of 21 under either control. Hard chains still show a guided lift, including the longest tool-recovery examples in the file. Medium does not separate: all three conditions repair 4 of 14. That should not be read as “medium tasks cannot be healed.” In this correction file, medium is dominated by derivation and dependency rows, which are condition-identical by design or in the observed data. It is a composition fact.
The traces also show why a single pass/fail label still hides the work. One hard Amazon Nova 2 Lite run has a reference trajectory of four required tools and ten steps. It has to go from Guggenheim Bilbao to Frank Gehry, from Gehry to Toronto, from Toronto to the Toronto Reference Library, then to the library coordinates and the nearest subway. Blind retry ends with the claims still unsupported. Guided recovery cannot make a new call on attempt 1, then on attempt 2 it uses live MCP tools. After that attempt, all five expected claims are supported in the intercepted run, including the library latitude and longitude. The answer-only scorer still marks every claim missing from the answer.
A simple Dolphin Mall chain shows the same split. The trajectory requires three tools and six reference steps: identify the building, find its opening date, then retrieve papers published that day. Guided recovery supports all three claims in the trace on attempt 2. The final answer still omits them.
The recovery system can fix the evidence chain without fixing the sentence the user reads.
On the answer-only endpoint, guided recovery passes 21 of 60 traces (35.0%), blind retry 19 of 60 (31.7%) and claim-ID feedback 18 of 60 (30.0%). The paired guided-versus-blind comparison is only 5 guided-only against 3 blind-only. Of the 32 guided grounded-run repairs, just 11 also pass the answer view; 21 are trace-only. That is not a result to paper over. It is the same dual-scoring contract from the diagnostic article, now applied after healing.
The first article changed the unit of trust from the response to the run. It showed 109 claims that answer-only evaluation accepted even though the recorded execution did not support them. This experiment takes the next step. The execution record is not only useful for explaining why an agent failed. For the failure mode in which tools and evidence are the bottleneck, that record can be turned into a better recovery than a blind retry.
For production agents, “try again” is not a healing strategy.
A system that already recorded which tool failed, which arguments were used, which evidence arrived and which claims remain unsupported should not throw that state away and sample another completion from scratch. The first generation of agent research demonstrated that models can use tools [1, 2]. MCP-Atlas showed that stronger agents increasingly fail later in the workflow, including early termination and faulty synthesis [4]. Our diagnostic batch showed that tool use is not the same as grounded tool use. This batch shows that a trace-informed recovery packet can materially improve evidence acquisition when the missing piece is the evidence itself.
It also draws a boundary around what SourceryKit does. SourceryKit does not magically make a model reason correctly. It preserves the requests, responses, failures and call order so that another layer can determine what the model actually did. In this experiment MFS performed that diagnosis and selected the recovery route. Without the execution record, the guided packet could not exist. The experiment also exposes the next failure mode. In 21 of the 32 guided grounded-run repairs, the execution contained enough intercepted evidence to support every expected claim, but the final answer still failed the answer-only check. The run had been repaired; the response had not.
The next problem is therefore answer synthesis: how to take verified evidence that is already present in the execution and reliably carry it into the words shown to the user.
That is another story that will be told another time (coming soon on provably.ai)
[2] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan and Y. Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” International Conference on Learning Representations (ICLR), 2023. https://arxiv.org/abs/2210.03629
[4] C. Bandi et al., “MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers,” arXiv:2602.00933v3, 2026. https://arxiv.org/abs/2602.00933
[5] J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song and D. Zhou, “Large Language Models Cannot Self-Correct Reasoning Yet,” International Conference on Learning Representations (ICLR), 2024. https://openreview.net/forum?id=IkmD3fKBPQ
