A result exists, but its meaning is still underspecified.
System note · evaluation infrastructure
Eval Evidence
System boundary
A score becomes
an evidence envelope.
- model + provider state
- harness + budget
- task + environment
- verifier + artifacts
- field provenance
Matched · different · unavailable · integrity failure
01
The problem is not another leaderboard
Evaluation results are routinely compared after model identity, harness configuration, token budget, verifier, environment, and artifact lineage have been compressed into a row. The number survives; the conditions that make it interpretable do not.
Eval Evidence works after a run. It detects a supported run format, builds a deterministic bundle, validates its schema and digest, and can re-hash referenced local files. Missing historical evidence remains unavailable instead of being guessed.
02
Separate the kinds of evidence
The bundle records item-validity claims, the evaluation instrument, and verifier evidence as different layers. Operator assertions do not silently become observations. A reported reward remains separate from reward-independent evidence.
That distinction is small but consequential: artifact identity, signer identity, verifier correctness, and ground truth are different claims. A hash can identify bytes; it cannot make the bytes true.
03
Design the refusal boundary
The system states what it cannot prove. A retrospective bundle cannot establish that an archive was unchanged before the bundle existed. An unsigned digest does not identify who produced it. Coverage is not comparability, and no default completeness threshold can substitute for a field-specific policy.
I care about this negative space because evaluation infrastructure becomes dangerous when administrative neatness is mistaken for epistemic authority. The verifier must remain able to say unavailable, different, or out of scope.
04
Why it belongs in the loop
Eval Evidence is the remember stage made concrete. The run leaves an evidence envelope that another reviewer can inspect without executing the model or trusting the original narration.
The next layer is not to make the envelope grander. It is to add narrowly scoped attestation profiles, comparison policies, and—where the domain demands it—an external software or physical verifier with its own trust model.