Reproducibility CheckAgentic reproduction of published results tables
Six papers · three publishers

Can an AI rebuild a paper's results table from its Methods and its data alone?

The AI gets only a paper's Methods and the original data the authors deposited, never their code or their results. It rebuilds one results table, and a fixed script grades every number against the published one. Where a number misses, we find out why: usually the Methods left a choice unsaid.

1. Read the MethodsThe Methods are turned into a list of steps, each quoting the paper's own sentence, with the places the paper leaves open marked.
2. Rebuild blindAn AI model works in a closed folder with that list, a blank copy of the table and the authors' data. It writes code, runs it and records every choice it had to make.
3. Grade every cellA fixed script compares each number with the published one: within 2%, within 20%, or further off.
4. Explain the missesA second pass tests each miss against the data and names the cause, and the sentence the Methods would need.

The papers

Two papers per publisher, chosen because the authors deposited their data openly. Click a paper for its full check: the rebuilt table, every graded number and why each miss happened.

Claude Opus 5.5 vs Claude Sonnet 5.5

Each paper was rebuilt by both models from exactly the same inputs. Below: how close each got, then every open choice the Methods left and what each model decided.

Every choice the models had to make

Each row is a point where the Methods left something open. The models wrote down each choice as they worked; the authors' column comes from the follow-up analysis of the misses. Highlighted rows are where the two models chose differently.

Cost