Backtestslitellmrun-20261018-182052#44921
[Bug]: HiddenLayer JWT refresh blocks the event loop inside the async guardrail request
The merged fix was judged better than our patch.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The production change is byte-for-byte equivalent in both patches: both import asyncio and replace the synchronous `self.refresh_jwt_func()` call with `await asyncio.to_thread(self.refresh_jwt_func)` in both HiddenlayerGuardrail and HiddenlayerGuardrailV2. That is the correct root-cause fix (the refresh uses the blocking `requests` client, so moving it to a worker thread unblocks the loop), and the independent tests confirm both patches turn all six base failures into passes. The decision therefore rests on tests and craft. Patch A adds a single timing-based test: a loopback server sleeps 0.5s on the token endpoint and the test asserts the heartbeat gap stays under 0.25s. It covers both classes but only the 'request' input path, and timing thresholds are inherently load-sensitive. Patch B's main test is deterministic rather than timing-based: the mocked token endpoint blocks on a threading.Event that only an unrelated coroutine can set, so the test can only pass if the event loop actually runs during the refresh, and it would deadlock-then-timeout on base. It is parametrized over both classes and both request/response paths, verifies the Basic auth header, the expired-then-fresh bearer sequence, and the single refresh call. B also adds two edge-path tests the issue's retry logic implies: a refresh exception propagates (as the same object) without clobbering the token, and a second 401 after refresh propagates without another refresh. These are not tautological and pin behavior that asyncio.to_thread must preserve. Risks in B: it relies on `respx`, a `respx_mock` fixture, and an `httpx_transport` fixture whose presence I cannot verify from the diff, and the async tests lack an explicit asyncio marker; it also reorders imports (the 4 removed lines) which is minor churn. Risks in A: timing flakiness and narrower coverage. On balance B's tests more directly and robustly encode the issue's invariant and cover more of the affected code paths, so B is the better patch despite identical fixes.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
1h 2m
Wall-clock time in the container
Full test suite
No new failures
113 passed · 0 skipped · 113 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
98vs 142 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
The async HiddenLayer guardrail refreshes an expired access token synchronously after a 401. A deliberately slow local token endpoint stalls an independent event-loop heartbeat
Expected: asynchronous guardrail execution does not block unrelated requests during token refresh