How Our Software Factory Compares

We replay real, already-fixed GitHub issues from public repos. Our software factory gets the issue and the code as it was before the fix, works in an isolated container with the network locked down, and never sees the real fix. A blind judge then compares our patch to the one maintainers merged, labelled only A and B.

Our progress:

% marked better 0% 25% 50% 75% 100% 17 Sep 23 Sep 2 Oct 6 Oct 13 Oct 21 Oct 25 Oct 11 Nov 23 Sep 2026 · pydantic/pydantic · run-20260923-170312: 47% better (14 better, 0 par, 16 worse, 0 error). First pydantic run 29 Sep 2026 · pydantic/pydantic · run-20260929-204949: 40% better (12 better, 2 par, 12 worse, 4 error). Every verification fix in place; 3 errors from a login-expiry bug, 1 no-patch 30 Sep 2026 · pydantic/pydantic · run-20260930-144322: 67% better (20 better, 2 par, 8 worse, 0 error). Updated to Opus 5.5 30 Sep 2026 · pydantic/pydantic · run-20260930-183947: 83% better (25 better, 0 par, 2 worse, 3 error). Foreman lane (the factory solver) on the same sample as the Opus 5.5 run 2 Oct 2026 · pydantic/pydantic · run-20261002-175417: 83% better (25 better, 0 par, 2 worse, 3 error). Foreman 0.3.1, gate checked against the box; 3 timeouts from hung shell commands 3 Oct 2026 · pydantic/pydantic · run-20261003-150343: 87% better (26 better, 0 par, 3 worse, 1 error). sudo-solver 0.4.0 (foreman stripped to run); 20-minute shell timeout; 1 harness error (8998) 5 Oct 2026 · pydantic/pydantic · run-20261005-183002: 93% better (28 better, 0 par, 2 worse, 0 error). sudo-solver 0.5.1 7 Oct 2026 · pydantic/pydantic · run-20261007-130130: 83% better (25 better, 0 par, 4 worse, 1 error). sudo-solver 0.5.1 random sample 9 Oct 2026 · pydantic/pydantic · run-20261009-110034: 87% better (26 better, 0 par, 3 worse, 1 error). Tests now run only the tests related to each change, on the code before and after the patch, plus a generated acceptance test. Ticket 13786 is shown as an error: its upstream fix commit is empty. 13 Oct 2026 · pydantic/pydantic · run-20261013-104308: 90% better (27 better, 0 par, 3 worse, 0 error). sudo-solver 0.5.1. Test suite changes: the tests related to each changed file run on the code before and after the patch, and a file no rule maps gets up to 10 nearby tests instead of none. No generated acceptance test 22 Oct 2026 · pydantic/pydantic · run-20261022-024030: 80% better (24 better, 0 par, 4 tie, 2 worse, 0 error). sudo-solver 0.5.1 on the foreman lane, judged by the four-judge panel 27 Oct 2026 · pydantic/pydantic · run-20261027-165153: 90% better (27 better, 0 par, 3 worse, 0 error). sudo-solver 0.5.1 on the foreman lane, judged by the four-judge panel 18 Sep 2026 · pydantic/pydantic-ai · run-20260918-215936: 39% better (7 better, 0 par, 9 worse, 2 error). First pydantic-ai run 17 Sep 2026 · gohugoio/hugo · run-20260917-015623: 30% better (9 better, 3 par, 16 worse, 2 error). First Hugo run 22 Sep 2026 · gohugoio/hugo · run-20260922-141813: 43% better (13 better, 1 par, 14 worse, 2 error). First Hugo run to reach 30/30 with zero attrition 30 Sep 2026 · gohugoio/hugo · run-20260930-140149: 67% better (20 better, 2 par, 6 worse, 2 error). Opus 5.5 in the box, solver only; 1 error from a mid-run login exchange (since fixed) 2 Oct 2026 · gohugoio/hugo · run-20261002-200313: 83% better (25 better, 0 par, 4 worse, 1 error). Foreman lane (the factory solver) on the same sample as the 2026-09-30 solver-only run 5 Oct 2026 · gohugoio/hugo · run-20261005-133023: 87% better (26 better, 0 par, 3 worse, 1 error). sudo-solver 0.4.0; one sample split across three harness runs after two Docker engine outages 7 Oct 2026 · gohugoio/hugo · run-20261007-020926: 93% better (28 better, 0 par, 2 worse, 0 error). sudo-solver 0.5.1 random new sample of tickets 9 Oct 2026 · gohugoio/hugo · run-20261009-153657: 90% better (27 better, 0 par, 3 worse, 0 error). Tests now run only the tests related to each change, on the code before and after the patch, plus a generated acceptance test. Ticket 15174 was rerun after a harness fix. 13 Oct 2026 · gohugoio/hugo · run-20261013-011256: 93% better (28 better, 0 par, 2 worse, 0 error). sudo-solver 0.5.1. Test suite changes: the tests related to each changed file run on the code before and after the patch, and a file no rule maps gets up to 10 nearby tests instead of none. No generated acceptance test 24 Oct 2026 · gohugoio/hugo · run-20261024-162305: 90% better (27 better, 0 par, 2 worse, 1 error). sudo-solver 0.5.1 on the foreman lane, first full hugo run with the judge panel 30 Sep 2026 · vuejs/core · run-20261001-022054: 43% better (13 better, 0 par, 15 worse, 2 error). First vuejs/core run 2 Oct 2026 · vuejs/core · run-20261002-115633: 60% better (18 better, 0 par, 10 worse, 2 error). used the foreman lane (software factory) 3 Oct 2026 · vuejs/core · run-20261003-231210: 90% better (27 better, 0 par, 2 worse, 1 error). Foreman lane on sudo-solver 0.4.0, same sample as the first vuejs/core run 5 Oct 2026 · vuejs/core · run-20261005-124535: 83% better (25 better, 0 par, 4 worse, 1 error). sudo-solver 0.4.1 5 Oct 2026 · vuejs/core · run-20261005-155915: 83% better (25 better, 0 par, 5 worse, 0 error). sudo-solver 0.5.1 7 Oct 2026 · vuejs/core · run-20261007-145226: 83% better (25 better, 0 par, 5 worse, 0 error) 9 Oct 2026 · vuejs/core · run-20261009-161443: 80% better (24 better, 0 par, 5 worse, 1 error). Tests now run only the tests related to each change, on the code before and after the patch, plus a generated acceptance test. Ticket 15217 was rerun after a harness fix. 13 Oct 2026 · vuejs/core · run-20261013-093701: 80% better (24 better, 0 par, 6 worse, 0 error). sudo-solver 0.5.1. Test suite changes: the tests related to each changed file run on the code before and after the patch, and a file no rule maps gets up to 10 nearby tests instead of none. No generated acceptance test 21 Oct 2026 · vuejs/core · run-20261021-133314: 80% better (24 better, 0 par, 2 tie, 4 worse, 0 error). Added a test-writing agent to test that the fix fixes the bug; added 4 parallelized judges; cut the setup agent's prompt from 200+ lines to 94 6 Oct 2026 · jhy/jsoup · run-20261006-181938: 73% better (22 better, 0 par, 8 worse, 0 error). sudo-solver 0.5.1; first Java repo; 9 tickets re-run after a GitHub API rate limit 25 Oct 2026 · scrapy/scrapy · run-20261025-211112: 97% better (29 better, 0 par, 1 worse, 0 error). First run on GitHub Actions: 5 tickets in parallel, each change checked against its related tests; judged by the four-judge panel 24 Oct 2026 · openai/openai-agents-python · run-20261024-184640: 73% better (22 better, 0 par, 2 tie, 6 worse, 0 error). Run on GitHub Actions: 5 tickets in parallel, each change checked against its related tests; judged by the four-judge panel 6 Nov 2026 · pytest-dev/pytest · run-20261106-234554: 83% better (25 better, 0 par, 4 worse, 1 error). First pytest run; one sample split across two harness runs after a lane setup failure 18 Oct 2026 · BerriAI/litellm · run-20261018-182052: 87% better (26 better, 0 par, 4 worse, 0 error). sudo-solver 0.5.1 on the foreman lane; judge panel v2: four Fable 5.1 judges, with tests drafted by a test-writing agent 19 Oct 2026 · BerriAI/litellm · run-20261019-033420: 80% better (24 better, 0 par, 1 tie, 5 worse, 0 error). sudo-solver 0.5.1 on the foreman lane; judge panel v3: four Fable 5.1 judges, with tests drafted by a test-writing agent, which now skips fixes that change only data or config 11 Nov 2026 · BerriAI/litellm · run-20261111-220245: 87% better (26 better, 0 par, 2 tie, 2 worse, 0 error)
pydanticpydantic-aihugovuejsoupscrapyopenai-agents-pythonpytestlitellm

Each point is one run on that repo's fixed ticket sample, placed at the day it ran; two runs on one day sit side by side. Dotted lines and text label any changes we made.

How to read a verdict

Better

Our patch passes the full test suite and the judge preferred it to the merged fix.

Par

Both are equally good.

Tie

The judges split evenly: half preferred our patch, half the merged fix.

Worse

The merged fix was better, or our patch broke a test that passed before. We show every one.

Error

The harness couldn't finish the ticket. Counted in the total, never dropped.

Repos

RepoLanguageLatest runBetter · par · worse% better