AI pull requests, proven on your own ticket history before you pay.

sudo ships merged PRs from your ticket queue and charges per PR that survives. Before that, we replay your last 90 days of closed tickets against your repo as it was, solve each one blind, and hand you an honest comparison against what your engineers actually shipped.

Pay per merged PR. The backtest comes first.

backtest · acme/api · 90 days · 87 ticketsillustrative

Report headline

X betterY parZ worseW no scorable attempt
Page one · failures first
#503Flaky CSV import on files with a UTF-8 BOMOur patch fails 2 of 14 tests · engineer patch passesworse
#511Webhook retries duplicate deliveries on 5xxTimed out at 40 min · no patch producedno scorable attempt
#482Race in session refresh on token expiryBoth pass · judge prefers ours: narrower lock, adds regression testbetter
#497Pagination off by one on CSV exportsBoth pass · judge calls it equivalentpar
Illustrative mock. Titles and counts are placeholders. A sample report will be linked here.

Walkthrough

One ticket, start to finish

Scroll to follow three tickets out of Jira, into sealed containers, and through to a verdict. The tickets, code and results below are illustrative.

01 · Find

Read 90 days of closed tickets

We start from the tickets your team closed in the last 90 days, straight from Jira.

02 · Pick

Rank the ones we can score

A ticket is scorable when a fix was merged and tests touch the changed code. Here we follow three.

03 · Seal

One sealed container per ticket

Each ticket gets its own sandbox: your repo, no network beyond the model API, and no view of the real fix.

04 · Baseline

Prove the bug on the repo as it was

Before anyone fixes anything, the affected tests run on the pre-fix checkout. One fails: the bug is real.

05 · Solve

Solve it from the ticket alone

Same checkout, same sandbox. The agent reads the ticket, finds the race and writes a patch.

06 · Retest

The same six tests, now on the patch

The suite runs again, unchanged. All six pass and the patch applies cleanly, so it is marked clean.

07 · Compare

Our patch next to your engineer's

Blue is sudo's patch. Amber is the one your engineer merged in PR #1931.

08 · Judge

A blind judge weighs both

Both pass the tests, so the tests cannot separate them. A judge reads the ticket and the two patches as A and B, and is never told who wrote which.

09 · Report

Every ticket, scored honestly

Wins, ties and losses side by side, failures first.

JiraYour workProjectsFiltersDashboardsSearchAT

Projects / Acme API / Issues

Issues

41 scorable1–12 of 87
JQLproject = ACME AND status = Done AND resolved >= -90d ORDER BY resolved DESCORDER BY scorable DESC
TKeySummaryStatusResolvedScoring
ACME-511Webhook retries duplicate deliveries on 5xxDoneOct 14scorable
ACME-509Add CSV export to the billing pageDoneOct 11no tests touched
ACME-503Flaky CSV import on files with a UTF-8 BOMDoneOct 7scorable
ACME-501Bump Postgres driver to 3.2DoneOct 2dependency bump
ACME-497Pagination off by one on CSV exportsDoneSep 26scorable
ACME-494Team admins can revoke API keysDoneSep 19no linked PR
ACME-490Rotate staging TLS certificatesDoneSep 12no code change
ACME-488Timezone drift in weekly digest emailsDoneSep 3scorable
ACME-482Race in session refresh on token expiryDoneAug 25scorable
ACME-479Onboarding checklist v2DoneAug 14design only
ACME-476Search ignores the archived projects filterDoneAug 2scorable
ACME-471Update incident runbook linksDoneJul 22docs only
  • ACME-482sandbox-1 · net: model API only
  • ACME-497sandbox-2 · net: model API only
  • ACME-503sandbox-3 · net: model API only

$ git checkout 9c41e0a # parent of the fix commit

$ cat TICKET.md

ACME-482 · Race in session refresh on token expiry

$ rg -n "def refresh" src/

src/auth/session.py:46

$ pytest tests/auth -k concurrent

1 failed · two tokens issued for one session

editing src/auth/session.py

$ pytest tests/auth

39 passed

patch ready · 2 files · +22 −6

session.pysrc/auth · @ 9c41e0a, before the fix
38+from .locks import KeyedLocks39from .tokens import issue_token4041class SessionStore:42    def __init__(self, backend):43        self._backend = backend+        self._locks = KeyedLocks()4445    def refresh(self, session_id):46        session = self._backend.get(session_id)47        if not session.expired():48            return session.token+        with self._locks.hold(session_id):+            # re-read under the lock: another+            # request may have refreshed already+            session = self._backend.get(session_id)+            if not session.expired():+                return session.token49        token = issue_token(session.user_id)50        self._backend.put(session_id, token)+            token = issue_token(session.user_id)+            self._backend.put(session_id, token)51        return token
Harness · pytest · 6 affected tests@ 9c41e0a · before the fix@ 9c41e0a + sudo's patch

6 of 1,204 tests import the changed code. The same six run before the fix and on the patch.

Testbefore fixwith patch
test_refresh.py::test_refresh_expired···passpass
test_refresh.py::test_refresh_valid···passpass
test_session.py::test_put_get···passpass
test_session.py::test_concurrent_refresh···failpass
test_login.py::test_login_rotates_token···passpass
test_logout.py::test_logout_revokes···passpass

before the fix 1 failed · 5 passed · the bug is realwith the patch 6 passed · the bug is fixed

sudopatch from sandbox-1+22 −6
src/auth/session.py+8 −6
tests/auth/test_refresh.py+14
  • Applies cleanly to 9c41e0a
  • 6 of 6 affected tests pass
clean
sudosandbox-1clean+22 −6
src/auth/session.py@@ -38,6 +38,8 @@+from .locks import KeyedLocks from .tokens import issue_token+        self._locks = KeyedLocks()@@ -46,6 +48,10 @@ def refresh-        session = self._backend.get(session_id)-        if not session.expired():+        with self._locks.hold(session_id):+            session = self._backend.get(session_id)+            if not session.expired():+                return session.tokentests/auth/test_refresh.py@@ -88,0 +88,14 @@+def test_concurrent_refresh_issues_one_token(store):+    tokens = run_parallel(store.refresh, "s1", n=8)+    assert len(set(tokens)) == 1

6 of 6 affected tests pass · marked clean

engineermerged in PR #1931+11 −6
src/auth/session.py@@ -1,4 +1,7 @@+import threading from .tokens import issue_token++_refresh_lock = threading.Lock()@@ -46,6 +49,10 @@ def refresh-        session = self._backend.get(session_id)-        if not session.expired():+        with _refresh_lock:+            session = self._backend.get(session_id)+            if not session.expired():+                return session.token             token = issue_token(session.user_id)             self._backend.put(session_id, token)

6 of 6 affected tests pass · merged in PR #1931

Same ticket, same starting commit, same six tests, and both pass, so the tests alone cannot separate them. Ours was written without access to the engineer's.

Judge · blind

Reads the ticket and patches A and B in random order. It is never told who wrote which.

Fixes the raceAtieB
Lock scopeAtieB
Regression testAtieB
ReadabilityAtieB

Prefers A

A per-session lock instead of a process-wide one, and a regression test for the race.

betterA was sudo · B was your engineer

backtest · acme/api · 90 days · 3 of 87 tickets shownillustrative

Report headline

1 better1 par1 worse
Page one · failures first
#503Flaky CSV import on files with a UTF-8 BOMOur patch fails 2 of 14 tests · engineer patch passesworse
#497Pagination off by one on CSV exportsBoth pass · judge calls it equivalentpar
#482Race in session refresh on token expiryBoth pass · judge prefers ours: narrower lock, adds regression testbetter
Illustrative. A real report covers every scorable ticket in the window.

Next

What comes after the backtest

  • A classifier built from your results.We train a classifier on the backtest verdicts to decide which of your tickets the software factory is equipped to take, and which stay with your team.
  • A three-month pilot on live tickets.The factory picks up live tickets from your queue as they arrive. You are charged per ticket successfully merged, and nothing for the rest.
  • An agent-friendly repo.After the backtest we make your repo agentic friendly: reproducible environments, clear test entry points, and the context agents need to work in your codebase without hand-holding.

Get started

Run a backtest on your repo

No spam, no drip campaign. One human reply.