Backtestslitellmrun-20261019-033420
BerriAI/litellm
Python SDK and proxy server for calling 100+ LLM APIs in the OpenAI format.
This run · 19 Oct 2026 · run-20261019-033420
80%judged better than the merged fix
24 of 30 tickets. sudo-solver 0.5.1 on the foreman lane; judge panel v3: four Fable 5.1 judges, with tests drafted by a test-writing agent, which now skips fixes that change only data or config.
24 better0 par1 tie5 worse0 error
Runs on this sample
Click a run to see its tickets11 Nov 2026run-20261111-220245
19 Oct 2026run-20261019-033420 sudo-solver 0.5.1 on the foreman lane; judge panel v3: four Fable 5.1 judges, with tests drafted by a test-writing agent, which now skips fixes that change only data or config
Ticket 40477 counts as worse because of a harness fault, not the fix: the agent ran mypy, its patch picked up a binary .mypy_cache file, and the tests could not apply it. Its code change matches the human fix in approach and files. The fault is fixed for later runs.