Backtestspydanticrun-20261009-110034

pydantic/pydantic

Data validation library for Python, with a Rust core.

This run · 9 Oct 2026 · run-20261009-110034

87%judged better than the merged fix

26 of 30 tickets. Tests now run only the tests related to each change, on the code before and after the patch, plus a generated acceptance test. Ticket 13786 is shown as an error: its upstream fix commit is empty..

26 better0 par3 worse1 error

Runs on this sample

Click a run to see its tickets
27 Oct 2026run-20261027-165153 27 better0 par3 worse0 error sudo-solver 0.5.1 on the foreman lane, judged by the four-judge panel 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
22 Oct 2026run-20261022-024030 24 better0 par4 tie2 worse0 error sudo-solver 0.5.1 on the foreman lane, judged by the four-judge panel 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite

The four ties are tickets where the judges split two against two. Two pairs of tickets here were closed by the same commit (8998 and 13207, 13017 and 13089); future samples keep only one ticket per fix. Every ticket passed verification and audited clean.

13 Oct 2026run-20261013-104308 27 better0 par3 worse0 error sudo-solver 0.5.1. Test suite changes: the tests related to each changed file run on the code before and after the patch, and a file no rule maps gets up to 10 nearby tests instead of none. No generated acceptance test 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
9 Oct 2026run-20261009-110034 26 better0 par3 worse1 error Tests now run only the tests related to each change, on the code before and after the patch, plus a generated acceptance test. Ticket 13786 is shown as an error: its upstream fix commit is empty. 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
7 Oct 2026run-20261007-130130 25 better0 par4 worse1 error sudo-solver 0.5.1 random sample 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
5 Oct 2026run-20261005-183002 28 better0 par2 worse0 error sudo-solver 0.5.1 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
3 Oct 2026run-20261003-150343 26 better0 par3 worse1 error sudo-solver 0.4.0 (foreman stripped to run); 20-minute shell timeout; 1 harness error (8998) 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
2 Oct 2026run-20261002-175417 25 better0 par2 worse3 error Foreman 0.3.1, gate checked against the box; 3 timeouts from hung shell commands 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
30 Sep 2026run-20260930-183947 25 better0 par2 worse3 error Foreman lane (the factory solver) on the same sample as the Opus 5.5 run 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite

This run used a different solver, the foreman factory lane, on the same 30 tickets as the Opus 5.5 run of the same day. The three errors are all tooling, not judgement: on two Rust tickets (13621, 13661) the agent's test gate ran against a stale compiled extension, so it saw its own regression tests fail and declined to commit; on 13257 the agent hung itself on a huge number in a probe and was killed at the step time limit. Every other ticket passed verification and audited clean.

30 Sep 2026run-20260930-144322 20 better2 par8 worse0 error Updated to Opus 5.5 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite

This result is split across two harness runs on the same 30-ticket sample: the first (run-20260930-134012) judged 20 tickets before a harness fault on the host stopped it, and the second (run-20260930-144322) ran the remaining 10. Every ticket passed verification and audited clean.

29 Sep 2026run-20260929-204949 12 better2 par12 worse4 error Every verification fix in place; 3 errors from a login-expiry bug, 1 no-patch 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite

Three tickets (13453, 13460, 13465) errored because the sandbox's copy of the login expired mid-solve and could not be renewed; no patch was attempted on them. The bug was fixed in the harness the next day. One ticket (12978) is a deliberate no-patch: the agent found a test asserting the very behaviour the ticket reports and declined to change it. Both count as errors.

23 Sep 2026run-20260923-170312 14 better0 par16 worse0 error First pydantic run 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite