Eval Evidence — Edward Lue Chee Lip

What I want to know after a run

In my evaluation work, I care about what happens after a score appears. Someone else should be able to ask which model ran, which tools and budget it had, how success was checked, and which files support the number. Those questions become harder when the answer is scattered across logs and summaries.

I built Eval Evidence to preserve that account in a form software can check. It is a narrower task than deciding whether an evaluation is scientifically sound, but it gives that later review something concrete to work from.

What goes in and what comes out

The input is a completed evaluation run in a supported format. The tool records the model, harness, budget, task, environment, verifier, and available artifacts in a deterministic evidence bundle. It validates the bundle and can compare referenced files with their recorded hashes.

The public repository contains the schemas, CLI, tests, and trust model. The tool works after the run; it does not need to call the model again.

Try a small example

The synthetic walkthrough starts with a score file and saves a bundle. It then changes the score file while leaving the bundle untouched. Checking only the bundle still passes; checking the referenced files detects the change.

That example demonstrates an integrity check. It does not establish that the original score measured the intended capability.

What I built

I designed and implemented the evidence model, schemas, command-line tool, adapters, integrity checks, documentation, and review workflow. A central design choice is to keep an operator’s assertion separate from a direct observation.

Missing historical evidence stays marked as unavailable. A tidy bundle should not quietly fill a gap with something plausible.

What the bundle cannot establish

A hash can identify bytes; it cannot make their contents true. An unsigned bundle does not establish who produced it, and a bundle made after a run cannot prove what happened before it was created.

For another team using this, I would want the bundle to make a review easier: find the relevant files, identify missing evidence, and check whether something changed. Verifier correctness and the interpretation of the score still require their own investigation. Preserving the record is a starting point for independent verification.

From the project’s own documentation

This source diagram separates the reported outcome, its execution context, and the bytes that can be checked again. It documents the implemented evidence model; it is not an experimental result or a certificate that a score is correct.

An evidence bundle keeps reported run results, execution context and provenance, and file byte identities separate. Missing evidence remains explicitly unavailable.

Eval Evidence project documentation · commit 6d4a25b5f288 · original figure, unchanged. Full-resolution figure ↗