Backtestslitellmrun-20261018-182052
BerriAI/litellm
Python SDK and proxy server for calling 100+ LLM APIs in the OpenAI format.
This run · 18 Oct 2026 · run-20261018-182052
87%judged better than the merged fix
26 of 30 tickets. sudo-solver 0.5.1 on the foreman lane; judge panel v2: four Fable 5.1 judges, with tests drafted by a test-writing agent.
26 better0 par4 worse0 error
Runs on this sample
Click a run to see its tickets11 Nov 2026run-20261111-220245
19 Oct 2026run-20261019-033420 sudo-solver 0.5.1 on the foreman lane; judge panel v3: four Fable 5.1 judges, with tests drafted by a test-writing agent, which now skips fixes that change only data or config
Ticket 40477 counts as worse because of a harness fault, not the fix: the agent ran mypy, its patch picked up a binary .mypy_cache file, and the tests could not apply it. Its code change matches the human fix in approach and files. The fault is fixed for later runs.