sudo ships merged PRs from your ticket queue and charges per PR that survives. Before that, we replay your last 90 days of closed tickets against your repo as it was, solve each one blind, and hand you an honest comparison against what your engineers actually shipped.
Pay per merged PR. The backtest comes first.
Report headline
Walkthrough
Scroll to follow three tickets out of Jira, into sealed containers, and through to a verdict. The tickets, code and results below are illustrative.
01 · Find
We start from the tickets your team closed in the last 90 days, straight from Jira.
02 · Pick
A ticket is scorable when a fix was merged and tests touch the changed code. Here we follow three.
03 · Seal
Each ticket gets its own sandbox: your repo, no network beyond the model API, and no view of the real fix.
04 · Baseline
Before anyone fixes anything, the affected tests run on the pre-fix checkout. One fails: the bug is real.
05 · Solve
Same checkout, same sandbox. The agent reads the ticket, finds the race and writes a patch.
06 · Retest
The suite runs again, unchanged. All six pass and the patch applies cleanly, so it is marked clean.
07 · Compare
Blue is sudo's patch. Amber is the one your engineer merged in PR #1931.
08 · Judge
Both pass the tests, so the tests cannot separate them. A judge reads the ticket and the two patches as A and B, and is never told who wrote which.
09 · Report
Wins, ties and losses side by side, failures first.
Projects / Acme API / Issues
project = ACME AND status = Done AND resolved >= -90d ORDER BY resolved DESCORDER BY scorable DESC$ git checkout 9c41e0a # parent of the fix commit
$ cat TICKET.md
ACME-482 · Race in session refresh on token expiry
$ rg -n "def refresh" src/
src/auth/session.py:46
$ pytest tests/auth -k concurrent
1 failed · two tokens issued for one session
editing src/auth/session.py
$ pytest tests/auth
39 passed
patch ready · 2 files · +22 −6
38+from .locks import KeyedLocks39from .tokens import issue_token4041class SessionStore:42 def __init__(self, backend):43 self._backend = backend+ self._locks = KeyedLocks()4445 def refresh(self, session_id):46 session = self._backend.get(session_id)47 if not session.expired():48 return session.token+ with self._locks.hold(session_id):+ # re-read under the lock: another+ # request may have refreshed already+ session = self._backend.get(session_id)+ if not session.expired():+ return session.token49 token = issue_token(session.user_id)50 self._backend.put(session_id, token)+ token = issue_token(session.user_id)+ self._backend.put(session_id, token)51 return token
6 of 1,204 tests import the changed code. The same six run before the fix and on the patch.
before the fix 1 failed · 5 passed · the bug is realwith the patch 6 passed · the bug is fixed
src/auth/session.py@@ -38,6 +38,8 @@+from .locks import KeyedLocks from .tokens import issue_token+ self._locks = KeyedLocks()@@ -46,6 +48,10 @@ def refresh- session = self._backend.get(session_id)- if not session.expired():+ with self._locks.hold(session_id):+ session = self._backend.get(session_id)+ if not session.expired():+ return session.tokentests/auth/test_refresh.py@@ -88,0 +88,14 @@+def test_concurrent_refresh_issues_one_token(store):+ tokens = run_parallel(store.refresh, "s1", n=8)+ assert len(set(tokens)) == 1
6 of 6 affected tests pass · marked clean
src/auth/session.py@@ -1,4 +1,7 @@+import threading from .tokens import issue_token++_refresh_lock = threading.Lock()@@ -46,6 +49,10 @@ def refresh- session = self._backend.get(session_id)- if not session.expired():+ with _refresh_lock:+ session = self._backend.get(session_id)+ if not session.expired():+ return session.token token = issue_token(session.user_id) self._backend.put(session_id, token)
6 of 6 affected tests pass · merged in PR #1931
Same ticket, same starting commit, same six tests, and both pass, so the tests alone cannot separate them. Ours was written without access to the engineer's.
Judge · blind
Reads the ticket and patches A and B in random order. It is never told who wrote which.
Prefers A
A per-session lock instead of a process-wide one, and a regression test for the race.
betterA was sudo · B was your engineer
Report headline
Next
Get started