“Current evaluation frameworks mostly focus on the final success rate, revealing few insights during the process.” [3]
The agent runs.
The agent returns the right number.
The human thinks success.
But once we move away from the final answer and look at the execution that generated it, the situation can be very different. The relevant API call may have failed, the returned data may have been incomplete, or the agent may have called the wrong tool and nevertheless produced the expected answer. In that case, did the agent solve the task, or did it simply arrive at the right number without generating the evidence required to justify it? We think this problem can be captured with a simple message:
it is not only the answer, it is the trace too
In the last few years, research on AI agents has focused on moving language models from text generation to action. Toolformer explored how models can decide “which APIs to call, when to call them, what arguments to pass” [1]. ReAct combined reasoning and actions so that models could gather information from external sources instead of relying only on what was already encoded in their parameters. On ALFWorld and WebShop, ReAct improved the absolute success rate by 34 and 10 percentage points respectively [2]. These results demonstrated that external tools can materially improve the capability of language models. But the use of tools also creates a new problem: the final natural-language answer does not necessarily explain which tool was selected, which arguments were used, whether the call succeeded, or whether the returned information supports the claims presented to the user.
A conventional software system can be observed through logs, database records, traces and explicit error codes.
An agentic system is more difficult because a fluent answer can hide the execution that produced it.
The agent may call ten tools and still fail to retrieve the evidence required by the task. It may retrieve the correct evidence and then omit it from the final response. It may receive an error and continue answering as if the call had succeeded. An observability tool for agents must therefore record more than the existence of a tool call, it should construct the complete execution trace. That means preserving the request, the parameters, the response, the failure, the order of execution and the relationship between those events and the final claims.
AgentBoard described the limitation of outcome-only evaluation by observing that current frameworks “mostly focus on the final success rate, revealing few insights during the process” [3].
We think this is the point at which observability becomes part of agent reliability rather than only part of infrastructure operations. To achieve this, we built SourceryKit, which captures live MCP execution and turns each agent run into a durable evidence record that can be independently evaluated. SourceryKit does not create the benchmark claims and it does not need to copy claims from tool outputs. Its role is to preserve what the agent actually did, so that a separate evaluator can determine whether the work performed by the agent justifies the answer that it produced.
SourceryKit was one of the components used to build MASFaultsSimulator (MFS), the research platform from which this experiment was produced (coming soon Open Source). MFS can create and compare both Single-Agent Systems (SAS) and Multi-Agent Systems (MAS). A MAS is not simply one model receiving a longer prompt. It is a system in which several autonomous agents divide a task, exchange messages and evidence, hand off subtasks and combine their work into a final result. Depending on the design, the agents can work independently, through a central coordinator, through peer-to-peer coordination, or through a hybrid topology. MFS also allows the design of scenarios for critical AI agents use cases like payments, customer service, and knowledge retrieval.
MFS was created to test whether this collaboration actually improves the quality of the result. It runs SAS and MAS under controlled tasks and budgets, injects faults, preserves tool and message lineage, and measures verified success together with coordination overhead, redundancy and failure propagation. SourceryKit supplies the observability and execution-evidence layer required to see what each agent and each tool did. For this first batch, however, we deliberately disabled MAS coordination and used SAS only. The narrower design isolates the relationship between one agent, its live tool trace and its final answer before introducing delegation and handoff failures.
To test this problem under realistic conditions, we used MCP-Atlas. The benchmark contains 1,000 tasks across 36 production MCP servers and 220 tools. In 98.6% of the tasks, the agent must use two or more servers. Each task exposes between 6 and 37 tools, while only 2 to 8 of those tools are relevant. The best model in the original MCP-Atlas evaluation reached an 82.2% pass rate at the 0.75 claim-coverage threshold, but 63.3% of the diagnosed failures were cognitive rather than mechanical [4]. Models often entered the correct tool-use regime and then stopped too early or combined the retrieved information incorrectly. The MCP-Atlas authors conclude that the research target is moving from valid API calls alone towards “claim-aware stopping rules and trajectory-grounded answer verification” [4].
Our experiment focused on that last point. We ran 15 models on the same panel of 12 MCP-Atlas tasks, producing 180 canonical runs and 570 independently defined benchmark claims [5]. The first 13 models formed the core panel, while Amazon Nova 2 Lite and AI21 Jamba Large 1.7 were added as a supplementary non-Chinese extension. The complete model list and the full-coverage failures observed under the two evaluation views are reported in Table 1. The agents saw only the task prompt. The ground-truth claims were not shown to the agents and were not copied from the tools. SourceryKit intercepted the live MCP calls, while MFS compared the independently defined claims with two views of the same immutable run: the final answer and the recorded tool trace. No verification feedback was returned to the model during execution. This first batch was diagnostic only.
Table 1. Model-level full-coverage failures. Each model ran 12 tasks. Delta equals SourceryKit-grounded failures minus Atlas answer-only failures [5].
Model
Atlas failures
Grounded failures
Delta
Avg. tool calls
Claude Opus 5
9
12
+3
12.0
NVIDIA Nemotron 3 Ultra
10
10
+0
14.9
GPT-5.6 SOL
10
11
+1
8.8
Grok 4.5
8
10
+2
17.1
Gemini 3.6 Flash
7
9
+2
14.3
Mistral Medium 3.5
10
10
+0
14.3
MiniMax M3
6
10
+4
10.8
Kimi K3
8
10
+2
12.9
Qwen 3.7 Plus
9
8
-1
20.3
Xiaomi MiMo V2.5
12
10
-2
19.3
GLM 5.2
11
8
-3
17.6
* Supplementary extension. A positive delta means that some answers appeared correct even though the model’s tool trace did not support them. A negative delta means that the trace contained the required evidence, but the model failed to use or recognise it in its final answer. Because each model was tested on only 12 tasks, these results are descriptive rather than a definitive model ranking.
Figure 1. Claim-level comparison across 570 independently defined benchmark claims. Source: SourceryKit/MFS diagnostic experiment.
The most relevant number is 109. These were claims that the answer-only evaluation accepted, but that were not supported by the captured tool execution. They are the claims that a conventional benchmark is least able to explain. Some may be lucky guesses or facts remembered by the model. Others may be conclusions based on incomplete inputs, values asserted after a failed tool call, or derivations for which the execution does not preserve a complete evidence chain. The reverse situation also occurred. In 52 cases, the model’s tool calls found supporting evidence, but its final answer incorrectly omitted or rejected that evidence. In 74 cases, both the answer and the trace supported the claim. In 335 cases, neither the trace nor the answer was correct.
The four categories describe very different engineering problems. A claim that is present in the answer but absent from the trace requires evidence recovery, a corrected tool call, or the removal of an unsupported assertion. A claim that is present in the trace but missing from the answer does not normally require another tool call; it requires better synthesis of evidence that the agent has already collected. A claim that is missing from both requires a new plan. The trace is therefore not only useful for explaining failure. It provides the information required to select the appropriate correction.
The traces also showed how the final score can hide what happened. In one task, the agent explicitly reported that every relevant tool call had failed and returned null values for the requested fields. The answer-only evaluator nevertheless credited the benchmark claims, while the SourceryKit-grounded evaluation rejected them because no successful evidence had been retrieved. In another task, the agent returned three correct source facts and the correct difference of 36 years. The answer looked complete, but the trace contained no recorded evidence link for the derived result. We also observed the opposite case: the agent retrieved the requested values, but the final answer omitted them or presented them in a structure that the answer evaluator did not recognise. The same pass/fail label would mix a tool outage, a provenance failure and an answer-formatting problem.
Figure 2. Full-coverage run results under answer-only and trace-supported evaluation. Source: SourceryKit/MFS diagnostic experiment .
At full claim coverage, the answer-only evaluation accepted 39 of 180 runs. Requiring complete support in the recorded execution reduced that number to 27. But SourceryKit-grounded evaluation was not simply a more severe version of the same test. Twenty-nine runs passed the answer-only evaluation and failed complete trace support, while 17 runs had complete trace support and failed the answer-only evaluation. Only 10 runs passed both views simultaneously . The answer tells us what the agent said. The trace tells us whether the execution supports saying it.
The difference became larger as the tasks became more complex. On the hard tier, the answer-only evaluator accepted 7 of 60 runs at full coverage. Complete trace support was found in 0 of 60. The agents were not simply refusing to use tools: the hard-task runs averaged approximately 16.9 tool calls each. The problem is that tool use is not the same as grounded tool use. An agent can call many tools, collect partial information, lose a dependency, and still generate a polished final response.
SourceryKit changes the unit of trust from the response to the run. It creates the observability and evidence layer required to distinguish a wrong answer from a right-looking answer without provenance, a retrieval failure from a synthesis failure, and a tool outage from a model hallucination. In our experiment, MFS performed the claim comparison, but it could only do so because SourceryKit preserved the live MCP requests, responses, failures and ordering of calls. Without that execution record, the 109 unsupported claims would look like successes.
For production agents, “the output looks right” is not a sufficient reliability standard. A system that acts on financial data, executes code, queries private databases or changes the state of another service must be able to show how its answer was produced.
The first generation of agent research demonstrated that models can use tools. The next generation must demonstrate that the tools used by the agent support the claims and actions that follow.
There is also a practical consequence for correction. Previous work has shown that language models often struggle to improve their reasoning through intrinsic self-correction alone and can even degrade when they receive no reliable external feedback [5]. This creates the next question for our experiment:
If SourceryKit can distinguish a missing answer from missing evidence, a failed tool path or an incomplete derivation, can that evidence be used to select the right healing action?
That is another story that will be told another time (coming soon on provably.ai)
References
[1] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda and T. Scialom, “Toolformer: Language Models Can Teach Themselves to Use Tools,” arXiv:2302.04761, 2023. https://arxiv.org/abs/2302.04761
[2] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan and Y. Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” International Conference on Learning Representations, 2023. https://arxiv.org/abs/2210.03629
[3] C. Ma, J. Zhang, Z. Zhu, C. Yang, Y. Yang, Y. Jin, Z. Lan, L. Kong and J. He, “AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents,” arXiv:2401.13178, 2024. https://arxiv.org/abs/2401.13178
[4] C. Bandi et al., “MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers,” arXiv:2602.00933v3, 2026. https://arxiv.org/abs/2602.00933
[5] J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song and D. Zhou, “Large Language Models Cannot Self-Correct Reasoning Yet,” International Conference on Learning Representations, 2024. https://arxiv.org/abs/2310.01798
