Backtestspydanticrun-20261022-024030
pydantic/pydantic
Data validation library for Python, with a Rust core.
This run · 22 Oct 2026 · run-20261022-024030
80%judged better than the merged fix
24 of 30 tickets. sudo-solver 0.5.1 on the foreman lane, judged by the four-judge panel.
Runs on this sample
Click a run to see its tickets27 Oct 2026run-20261027-165153 sudo-solver 0.5.1 on the foreman lane, judged by the four-judge panel
22 Oct 2026run-20261022-024030 sudo-solver 0.5.1 on the foreman lane, judged by the four-judge panel
The four ties are tickets where the judges split two against two. Two pairs of tickets here were closed by the same commit (8998 and 13207, 13017 and 13089); future samples keep only one ticket per fix. Every ticket passed verification and audited clean.
13 Oct 2026run-20261013-104308 sudo-solver 0.5.1. Test suite changes: the tests related to each changed file run on the code before and after the patch, and a file no rule maps gets up to 10 nearby tests instead of none. No generated acceptance test
9 Oct 2026run-20261009-110034 Tests now run only the tests related to each change, on the code before and after the patch, plus a generated acceptance test. Ticket 13786 is shown as an error: its upstream fix commit is empty.
7 Oct 2026run-20261007-130130 sudo-solver 0.5.1 random sample
5 Oct 2026run-20261005-183002 sudo-solver 0.5.1
3 Oct 2026run-20261003-150343 sudo-solver 0.4.0 (foreman stripped to run); 20-minute shell timeout; 1 harness error (8998)
2 Oct 2026run-20261002-175417 Foreman 0.3.1, gate checked against the box; 3 timeouts from hung shell commands
30 Sep 2026run-20260930-183947 Foreman lane (the factory solver) on the same sample as the Opus 5.5 run
This run used a different solver, the foreman factory lane, on the same 30 tickets as the Opus 5.5 run of the same day. The three errors are all tooling, not judgement: on two Rust tickets (13621, 13661) the agent's test gate ran against a stale compiled extension, so it saw its own regression tests fail and declined to commit; on 13257 the agent hung itself on a huge number in a probe and was killed at the step time limit. Every other ticket passed verification and audited clean.
30 Sep 2026run-20260930-144322 Updated to Opus 5.5
This result is split across two harness runs on the same 30-ticket sample: the first (run-20260930-134012) judged 20 tickets before a harness fault on the host stopped it, and the second (run-20260930-144322) ran the remaining 10. Every ticket passed verification and audited clean.
29 Sep 2026run-20260929-204949 Every verification fix in place; 3 errors from a login-expiry bug, 1 no-patch
Three tickets (13453, 13460, 13465) errored because the sandbox's copy of the login expired mid-solve and could not be renewed; no patch was attempted on them. The bug was fixed in the harness the next day. One ticket (12978) is a deliberate no-patch: the agent found a test asserting the very behaviour the ticket reports and declined to change it. Both count as errors.