Research digest · COLM 2026 · Agent Behavior

Diagnosing Agent Capabilities

Through Information-Axis Knockouts

A benchmark score says whether an agent won. An information-axis knockout asks what the agent needed in order to win—and which absence changed its behavior.

Accepted poster

Edward’s role: Co-author
Status: Accepted · August 2026
Authors: Kaaustaaub Shankar · Edward Lue Chee Lip · Joshua Liu · Benjamin J. Smith
Artifacts: Code and active manuscript available
Reading time: 4 min read
Contribution: Co-author; owned the KA59 experimental lane, built and audited evaluation infrastructure, ran model and random-baseline sweeps, analyzed traces, and contributed to the paper framing and integration. Final wording is pending collaborator review.
Claim state: E2 · accepted; artifact-backed
Last checked: 13 Aug 2026
Provenance: Official OpenReview author-dashboard decision evidence plus the active public repository and manuscript. Model-specific draft numbers are intentionally excluded from this digest.
Citation: Canonical proceedings citation pending.

One result · visually

Evidence before interpretation.

The same zero decomposes into different information failures.

01World

02Goal

03Mechanics

04Feedback

Outcome → diagnosis

The same object · seven altitudes

From question
to falsifier.

A paper becomes easier to inspect when its claim, boundary, intervention, measurement, failure mode, and next disconfirming result are kept separate.

01Question
Which missing information actually causes an agent to fail inside a partially observed environment?
02Claim
Capability is a dependence profile across information channels, not a scalar attached to a model name.
03System boundary
Agent policy, task environment, prompt-visible information, feedback channel, and the behavioral trace they jointly produce.
04Intervention
Remove or degrade World, Goal, Mechanics, and Feedback information one axis at a time.
05Measurement
Behavioral change under each knockout: success, exploration, adaptation, and trajectory—not verbal explanation alone.
06Failure mode
A model can describe the missing mechanic correctly while its policy remains unchanged.
07Next falsifier
A cross-environment representation that predicts knockout behavior without erasing environment-specific dependencies.

From scores to diagnoses

Partially observed environments mix several capabilities into one outcome. The agent must see the world, infer the goal, discover hidden mechanics, and interpret feedback. A zero can come from failure at any one of those layers.

The benchmark removes or degrades World, Goal, Mechanics, and Feedback information one axis at a time. The resulting behavioral change becomes a diagnostic signature rather than another leaderboard number.

What survives across environments

Different environments expose different bottlenecks, and models can fail on the same axis in different ways. World degradation is consistently destructive; mechanics degradation ranges from fatal to recoverable depending on the environment and model capability.

Behavior and articulation also separate: agents can trigger or exploit a hidden mechanic without reliably stating the rule under the paper’s strict verbal filter. That is why behavioral traces matter more than confident post-hoc explanation alone.

Capability is multidimensional, not a scalar attached to a model name.
The environment and the model jointly determine the binding constraint.
Prompted reflection is not evidence of behavioral adaptation.

Why the knockout frame travels

The same method can examine tools, memory, system prompts, demonstrations, and feedback channels. Instead of asking only whether an agent is capable, we can ask which information substrate its capability depends on.

Still open

The next challenge is to connect these signatures across tasks without pretending that one environment’s axes are universal. A diagnostic should remain specific enough to explain failure and general enough to compare systems.