01
What does a failed run tell me?
When an agent fails, “the model could not do it” leaves a lot unexplained. It may not know where it is, what goal it should pursue, how an action works, or what the feedback means. Those call for different changes to the system.
We studied this by removing or degrading World, Goal, Mechanics, and Feedback information separately. Instead of changing everything at once, the benchmark asks how behavior changes when a particular source of information is missing.
02
My part was in the runs and the traces
I owned the KA59 experimental lane: building and auditing the evaluation infrastructure, running model and random-baseline sweeps, and analyzing the resulting traces. I also contributed to the paper’s framing and integration. The public evaluation code is available for inspecting the setup.
For this work, the trace matters because it shows what an agent actually did with the information it had. A final score compresses exploration, repeated mistakes, and adaptation into one outcome. Looking at the sequence lets us ask more specific questions about the failure.
03
Knowing the rule and using it come apart
Across the studied environments, the bottlenecks differed. Degrading world information was consistently destructive; degrading mechanics could be fatal or recoverable depending on the environment and model. I would not expect a single description of “reasoning ability” to explain all of those cases.
Behavior and verbal explanation also separated. Agents could trigger or exploit a hidden mechanic without reliably stating its rule under the paper’s strict verbal filter. In the other direction, an explanation alone does not establish that the agent adapted its behavior.
This is why I want evaluations to retain both the intervention and the trajectory. A convincing account after the fact is another output to examine, not a substitute for observing what changed.
04
The question I would bring to a deployed agent
If a team adds memory, tools, or feedback and the score improves, I want to know which addition the improvement depends on. Removing one channel at a time offers a way to investigate that, provided the intervention itself is well defined.
That is a possible use of the method, rather than a result established across production systems. Our environments give us controlled comparisons. The harder next question is which diagnoses remain useful across tasks, and which are specific to the environment we built.