How Our Software Factory Compares
We replay real, already-fixed GitHub issues from public repos. Our software factory gets the issue and the code as it was before the fix, works in an isolated container with the network locked down, and never sees the real fix. A blind judge then compares our patch to the one maintainers merged, labelled only A and B.
Our progress:
Each point is one run on that repo's fixed ticket sample, placed at the day it ran; two runs on one day sit side by side. Dotted lines and text label any changes we made.
How to read a verdict
Our patch passes the full test suite and the judge preferred it to the merged fix.
Both are equally good.
The judges split evenly: half preferred our patch, half the merged fix.
The merged fix was better, or our patch broke a test that passed before. We show every one.
The harness couldn't finish the ticket. Counted in the total, never dropped.
Repos
| Repo | Language | Latest run | Better · par · worse | % better |
|---|---|---|---|---|
| pydantic/pydantic | Python, Rust | 27 Oct 2026 | 27 better0 par3 worse | 90% |
| pydantic/pydantic-ai | Python | 18 Sep 2026 | 7 better0 par9 worse2 error | 39% |
| gohugoio/hugo | Go | 24 Oct 2026 | 27 better0 par2 worse1 error | 90% |
| vuejs/core | TypeScript | 21 Oct 2026 | 24 better0 par2 tie4 worse | 80% |
| jhy/jsoup | Java | 6 Oct 2026 | 22 better0 par8 worse | 73% |
| scrapy/scrapy | Python | 25 Oct 2026 | 29 better0 par1 worse | 97% |
| openai/openai-agents-python | Python | 24 Oct 2026 | 22 better0 par2 tie6 worse | 73% |
| pytest-dev/pytest | Python | 6 Nov 2026 | 25 better0 par4 worse1 error | 83% |
| BerriAI/litellm | Python | 11 Nov 2026 | 26 better0 par2 tie2 worse | 87% |