01
The appeal of splitting the work
A capable model can break a coding task into smaller pieces, then pass those pieces to other models. That makes it natural to ask whether the separation also helps with control: let an untrusted model plan, limit what the implementers can see, and have a monitor check the plan.
I led the protocol development, implementation, experiment design, adversarial evaluation, analysis, and writing with my co-authors. We examined where to put the monitor and what information to give it.
“Trusted” and “untrusted” describe the assumptions of this protocol. They do not mean that any model is infallible. The question is how the arrangement behaves under those assumptions.
02
The monitor needs something to inspect
The same separation that limits what a component can do can also remove information it needs. An abstract task description may look harmless even when its implementation has a harmful effect. I wanted to understand whether checking the plan was enough.
In the public paper, decomposition-only monitoring achieved AUROC 0.52, compared with 0.96 when the monitor could inspect concrete implementations. In these evaluated trajectories, the implementation gave the monitor much more useful evidence for distinguishing malicious from benign examples.
AUROC describes ranking across thresholds, with 0.5 corresponding to chance ranking. The 0.96 result is not a claim that the system is 96% safe or catches 96% of attacks at a chosen threshold. It tells us about discrimination in this experiment.
03
Why this changed the question for me
Prompt injection drew my attention to what an instruction could make a model do. This project made me look harder at the information passed between components. If the evidence of a problem only becomes visible in code, a monitor restricted to the plan may be checking at the wrong point.
For someone building a multi-agent coding system, I think that is worth testing directly. Splitting work into more agents does not, on its own, tell us whether oversight improved. We need to inspect which context the split preserves, which it discards, and where harmful behavior becomes observable.
04
What I still want to test
The result leaves a tradeoff I am still interested in: how much isolation can we introduce before it deprives the monitor of useful context? Could a plan retain enough relevant detail to support reliable checks earlier?
An adaptive planner makes that question harder. I would want a protocol tested against a model that understands how it will be monitored. The current result is a comparison of monitoring conditions, not a guarantee about every future decomposer.