Backtestslitellmrun-20261019-033420#44921
[Bug]: HiddenLayer JWT refresh blocks the event loop inside the async guardrail request
The judges split evenly between our patch and the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
The production change is byte-for-byte equivalent in both patches: each wraps `self.refresh_jwt_func()` in `await asyncio.to_thread(...)` for both HiddenlayerGuardrail and HiddenlayerGuardrailV2, which moves the blocking requests-based token refresh off the event loop and fixes the root cause. Independent tests confirm both pass identically. The difference lies in the tests. Patch A's primary test is deterministic: the token-refresh callback blocks on a threading.Event that only a peer coroutine can release, so the test can only pass if the event loop keeps running during the refresh. It also verifies the retry uses the fresh bearer token, and adds two edge tests (refresh exception propagates without clobbering the stored token; a second 401 after refresh propagates without another refresh), covering both v1/v2 and request/response paths. Patch B's test is a faithful port of the repro (real loopback server, 0.5s sleep, heartbeat gap < 0.25s), which is directly tied to the issue and self-contained, but it is timing-based and therefore inherently more prone to flakiness on loaded CI; its only other change is a docstring tweak. Patch A carries some risk in depending on `responses`/`respx` and an `httpx_transport` fixture that cannot be verified here, and its extra tests are beyond the strict ask, but they add no production risk. On balance A's deterministic concurrency test is the stronger regression guard for an event-loop-blocking bug, so A edges out B; the fix itself is a wash.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
20m 12s
Wall-clock time in the container
Full test suite
No new failures
113 passed · 0 skipped · 113 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
105vs 142 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
The async HiddenLayer guardrail refreshes an expired access token synchronously after a 401. A deliberately slow local token endpoint stalls an independent event-loop heartbeat
Expected: asynchronous guardrail execution does not block unrelated requests during token refresh