Backtestshugorun-20260930-140149

gohugoio/hugo

Static site generator written in Go.

This run · 30 Sep 2026 · run-20260930-140149

67%judged better than the merged fix

20 of 30 tickets. Opus 5.5 in the box, solver only; 1 error from a mid-run login exchange (since fixed).

20 better2 par6 worse2 error

Runs on this sample

Click a run to see its tickets
24 Oct 2026run-20261024-162305 27 better0 par2 worse1 error sudo-solver 0.5.1 on the foreman lane, first full hugo run with the judge panel 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite

The one error is a verdict we threw out, not a crash. Ticket 15228 (upgrading hugo to Go 1.27) was judged better, but the sandbox ran Go 1.27.3, which shipped seven weeks after the real fix. The agent copied in template security fixes that did not exist yet when the maintainer did the work. The harness now runs each ticket on the Go release that was current on the fix date. Two tickets here, 7515 and 15199, were closed by the same commit; future samples keep only one. Every ticket passed verification and audited clean.

13 Oct 2026run-20261013-011256 28 better0 par2 worse0 error sudo-solver 0.5.1. Test suite changes: the tests related to each changed file run on the code before and after the patch, and a file no rule maps gets up to 10 nearby tests instead of none. No generated acceptance test 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
9 Oct 2026run-20261009-153657 27 better0 par3 worse0 error Tests now run only the tests related to each change, on the code before and after the patch, plus a generated acceptance test. Ticket 15174 was rerun after a harness fix. 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
7 Oct 2026run-20261007-020926 28 better0 par2 worse0 error sudo-solver 0.5.1 random new sample of tickets 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
5 Oct 2026run-20261005-133023 26 better0 par3 worse1 error sudo-solver 0.4.0; one sample split across three harness runs after two Docker engine outages 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
2 Oct 2026run-20261002-200313 25 better0 par4 worse1 error Foreman lane (the factory solver) on the same sample as the 2026-09-30 solver-only run 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
30 Sep 2026run-20260930-140149 20 better2 par6 worse2 error Opus 5.5 in the box, solver only; 1 error from a mid-run login exchange (since fixed) 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
22 Sep 2026run-20260922-141813 13 better1 par14 worse2 error First Hugo run to reach 30/30 with zero attrition 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite
17 Sep 2026run-20260917-015623 9 better3 par16 worse2 error First Hugo run 30 tickets · permalink
IssueTitleVerdictLines ours / mergedTest suite