Backtestslitellm

BerriAI/litellm

Python SDK and proxy server for calling 100+ LLM APIs in the OpenAI format.

Latest run · 11 Nov 2026 · run-20261111-220245

87%judged better than the merged fix

26 of 30 tickets.

26 better0 par2 tie2 worse0 error

Runs on this sample

Click a run to see its tickets
11 Nov 2026run-20261111-220245 26 better0 par2 tie2 worse0 error 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
19 Oct 2026run-20261019-033420 24 better0 par1 tie5 worse0 error sudo-solver 0.5.1 on the foreman lane; judge panel v3: four Fable 5.1 judges, with tests drafted by a test-writing agent, which now skips fixes that change only data or config 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite

Ticket 40477 counts as worse because of a harness fault, not the fix: the agent ran mypy, its patch picked up a binary .mypy_cache file, and the tests could not apply it. Its code change matches the human fix in approach and files. The fault is fixed for later runs.

18 Oct 2026run-20261018-182052 26 better0 par4 worse0 error sudo-solver 0.5.1 on the foreman lane; judge panel v2: four Fable 5.1 judges, with tests drafted by a test-writing agent 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite