Amir Reza Peimani
All research

AI-ASSISTED SCIENCE · REPRODUCIBILITY

Executable Scientific Contracts for Research Auditing

What should an executable check establish about a scientific claim?

Amir Reza Peimani2026Submitted / under review

Executable contracts test whether paper claims, code, and released artifacts agree, using fixtures that expose specific scientific errors.

Paper claims, code, and released artifacts can encode different scientific quantities. Executable audits locate these mismatches and measure their effect on research conclusions.

Approach

Connect each claim to its implementation, construct a fixture that distinguishes competing behaviors, and test where the discrepancy occurs in released artifacts and what it changes.

Main result

25executable checks across five public AI repositories

The study instantiated 25 contracts across AgentPRM, ContractBench, ToolACE, τ-bench, and AgentAbstain. All 30 study-specific mutations were detected. The external mutation set yielded 7/8 detections on its first execution and 8/8 after repairing the missed fixture.

Original Figure 1: check outcomes across five repositories and mutation sensitivity. AgentAbstain checks were specified prospectively. The external set includes results before and after correcting the missed fixture.
Original Figure 1: check outcomes across five repositories and mutation sensitivity. AgentAbstain checks were specified prospectively. The external set includes results before and after correcting the missed fixture. Open full resolution ↗
The complete audit across five repositories
Figure 3. All 25 executable checks, grouped by repository and outcome. Marker shapes identify contract classes; the final five AgentAbstain checks were specified before execution.
Figure 3. All 25 executable checks, grouped by repository and outcome. Marker shapes identify contract classes; the final five AgentAbstain checks were specified before execution. Open full resolution ↗

Related work

Training on Broken Contexts ↗State-Complete Replay ↗