Backtestslitellmrun-20261018-182052#44921

Issue #44921 Merged fix PR #45437 Run 18 Oct 2026 · run-20261018-182052 BerriAI/litellm

[Bug]: HiddenLayer JWT refresh blocks the event loop inside the async guardrail request

Worse

The merged fix was judged better than our patch.

In this judgment, Patch A = our agent, Patch B = the merged fix.

The production change is byte-for-byte equivalent in both patches: both import asyncio and replace the synchronous `self.refresh_jwt_func()` call with `await asyncio.to_thread(self.refresh_jwt_func)` in both HiddenlayerGuardrail and HiddenlayerGuardrailV2. That is the correct root-cause fix (the refresh uses the blocking `requests` client, so moving it to a worker thread unblocks the loop), and the independent tests confirm both patches turn all six base failures into passes. The decision therefore rests on tests and craft. Patch A adds a single timing-based test: a loopback server sleeps 0.5s on the token endpoint and the test asserts the heartbeat gap stays under 0.25s. It covers both classes but only the 'request' input path, and timing thresholds are inherently load-sensitive. Patch B's main test is deterministic rather than timing-based: the mocked token endpoint blocks on a threading.Event that only an unrelated coroutine can set, so the test can only pass if the event loop actually runs during the refresh, and it would deadlock-then-timeout on base. It is parametrized over both classes and both request/response paths, verifies the Basic auth header, the expired-then-fresh bearer sequence, and the single refresh call. B also adds two edge-path tests the issue's retry logic implies: a refresh exception propagates (as the same object) without clobbering the token, and a second 401 after refresh propagates without another refresh. These are not tautological and pin behavior that asyncio.to_thread must preserve. Risks in B: it relies on `respx`, a `respx_mock` fixture, and an `httpx_transport` fixture whose presence I cannot verify from the diff, and the async tests lack an explicit asyncio marker; it also reorders imports (the 4 removed lines) which is minor churn. Risks in A: timing flakiness and narrower coverage. On balance B's tests more directly and robustly encode the issue's invariant and cover more of the affected code paths, so B is the better patch despite identical fixes.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

1h 2m

Wall-clock time in the container

Full test suite

No new failures

113 passed · 0 skipped · 113 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

98vs 142 merged

Changed lines, ours vs merged

Description

The async HiddenLayer guardrail refreshes an expired access token synchronously after a 401. A deliberately slow local token endpoint stalls an independent event-loop heartbeat

Expected: asynchronous guardrail execution does not block unrelated requests during token refresh

Verified against unmodified upstream main at 0 on 2026-10-08 UTC. Fresh controlled HTTP token refresh produces a 208 ms event-loop heartbeat gap

Config

The reproduction runs from an unmodified upstream checkout and starts a controlled loopback HTTP service. It does not require proxy YAML or provider credentials

LiteLLM Version

1.106.0, upstream main commit 0, verified 2026-10-08 UTC

Steps to Repro

Save this as repro.py in the upstream checkout

import asyncio
import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Service(BaseHTTPRequestHandler):
    token_requests = []
    evaluation_requests = []

    def log_message(self, *args):
        pass

    def do_POST(self):
        self.rfile.read(int(self.headers["Content-Length"]))
        status = 200
        if "token" in self.path:
            self.token_requests.append(self.path)
            time.sleep(0.2)
            payload = {"access_token": "local-token", "expires_in": 3600, "token_type": "Bearer"}
        elif self.path.endswith("interactions"):
            self.evaluation_requests.append(self.path)
            status = 401 if len(self.evaluation_requests) == 1 else 200
            payload = {"evaluation": {"action": "ALLOW"}, "analysis": []}
        else:
            payload = {"id": "chatcmpl-local", "object": "chat.completion", "created": int(time.time()),
                       "model": "validation-endpoint", "choices": [{"index": 0, "message": {"role": "assistant", "content": "hello"}, "finish_reason": "stop"}],
                       "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}}
        body = json.dumps(payload).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

server = ThreadingHTTPServer(("127.0.0.1", 0), Service)
threading.Thread(target=server.serve_forever, daemon=True).start()
base = f"http://127.0.0.1:{server.server_port}"

from functools import partial
from litellm.proxy.guardrails.guardrail_hooks.hiddenlayer.hiddenlayer import HiddenlayerGuardrail, _get_jwt

async def heartbeat(samples):
    while True:
        samples.append(time.perf_counter())
        await asyncio.sleep(0.01)

async def main():
    guardrail = HiddenlayerGuardrail(api_base=base, guardrail_name="local-check")
    guardrail.jwt_token = "expired-local-placeholder"
    guardrail.refresh_jwt_func = partial(_get_jwt, auth_url=base, api_id="local-client", api_key="local-placeholder")
    samples = []
    monitor = asyncio.create_task(heartbeat(samples))
    await asyncio.sleep(0.02)
    result = await guardrail.apply_guardrail(inputs={"texts": ["ordinary request"]}, request_data={}, input_type="request")
    await asyncio.sleep(0.02)
    monitor.cancel()
    await asyncio.gather(monitor, return_exceptions=True)
    print("guardrail result", result)
    print("evaluation requests", len(Service.evaluation_requests))
    print("refresh requests", len(Service.token_requests))
    print("largest heartbeat gap", round(max(b - a for a, b in zip(samples, samples[1:])), 3))

try:
    asyncio.run(main())
finally:
    server.shutdown()
    server.server_close()

Run it with the checkout on the import path

LITELLM_LOCAL_MODEL_COST_MAP=true PYTHONPATH=. python repro.py

Observed:

guardrail result {'texts': ['ordinary request']}
evaluation requests 2
refresh requests 1
largest heartbeat gap 0.208

Expected: the invariant described above holds without the observed failure

Which part of LiteLLM is this about?

Guardrails: moderation, PII masking, policies

How are you deploying?

pip / Python SDK

The two patches

Our patch+96 −2
litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.pyindex cfffff8eec..509fbde775 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import asyncio import os from collections.abc import Mapping, Sequence from typing import TYPE_CHECKING, Final, Literal, Protocol@@ -402,7 +403,8 @@ class HiddenlayerGuardrail(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                # The refresh uses a blocking HTTP client; run it off the event loop+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/detection/v1/interactions",@@ -609,7 +611,8 @@ class HiddenlayerGuardrailV2(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                # The refresh uses a blocking HTTP client; run it off the event loop+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/{path}",tests/unit/proxy/guardrails/guardrail_hooks/test_hiddenlayer.pyindex 954b9b9962..edeb86b75c 100644@@ -1153,3 +1153,94 @@ def test_get_jwt_gives_up_at_the_timeout_instead_of_blocking_the_event_loop(hang         guardrail.refresh_jwt_func()      assert [call.kwargs["timeout"] for call in get_jwt.call_args_list] == [2, 2]+++@pytest.fixture+def slow_token_hiddenlayer_server():+    """Loopback HiddenLayer stand-in: slow token endpoint, detection endpoints 401 once then allow."""+    import json+    from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++    token_requests: List[str] = []+    evaluation_requests: List[str] = []++    class Handler(BaseHTTPRequestHandler):+        def log_message(self, format, *args):+            pass++        def do_POST(self):+            self.rfile.read(int(self.headers.get("Content-Length", 0)))+            status = 200+            if "token" in self.path:+                token_requests.append(self.path)+                time.sleep(0.5)+                payload: dict = {"access_token": "fresh-token", "expires_in": 3600, "token_type": "Bearer"}+            else:+                evaluation_requests.append(self.headers.get("Authorization", ""))+                status = 401 if len(evaluation_requests) == 1 else 200+                payload = {+                    "evaluation": {"action": "ALLOW"},+                    "analysis": [],+                    "messages": [{"role": "user", "content": "ordinary request"}],+                }+            body = json.dumps(payload).encode()+            self.send_response(status)+            self.send_header("Content-Type", "application/json")+            self.send_header("Content-Length", str(len(body)))+            self.end_headers()+            self.wfile.write(body)++    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)+    thread = threading.Thread(target=server.serve_forever, daemon=True)+    thread.start()+    try:+        yield f"http://127.0.0.1:{server.server_port}", token_requests, evaluation_requests+    finally:+        server.shutdown()+        server.server_close()+        thread.join(timeout=5)+++@pytest.mark.asyncio+@pytest.mark.parametrize("guardrail_cls", [HiddenlayerGuardrail, HiddenlayerGuardrailV2])+async def test_jwt_refresh_after_401_does_not_block_the_event_loop(slow_token_hiddenlayer_server, guardrail_cls):+    """+    A 401 from HiddenLayer triggers a JWT refresh; a slow auth host must not stall+    unrelated coroutines on the same event loop while the token is fetched.+    """+    import asyncio+    from functools import partial++    base, token_requests, evaluation_requests = slow_token_hiddenlayer_server+    guardrail = guardrail_cls(api_base=base, guardrail_name="local-check")+    guardrail.jwt_token = "expired-token"+    guardrail.refresh_jwt_func = partial(_get_jwt, auth_url=base, api_id="id", api_key="secret")++    samples: List[float] = []++    async def heartbeat():+        while True:+            samples.append(time.perf_counter())+            await asyncio.sleep(0.01)++    monitor = asyncio.create_task(heartbeat())+    await asyncio.sleep(0.02)+    if guardrail_cls is HiddenlayerGuardrail:+        inputs = GenericGuardrailAPIInputs(texts=["ordinary request"])+    else:+        inputs = GenericGuardrailAPIInputs(+            texts=["ordinary request"],+            structured_messages=[{"role": "user", "content": "ordinary request"}],+            model="gpt-4o-mini",+        )+    result = await guardrail.apply_guardrail(inputs=inputs, request_data={}, input_type="request")+    await asyncio.sleep(0.02)+    monitor.cancel()+    await asyncio.gather(monitor, return_exceptions=True)++    assert result["texts"] == ["ordinary request"]+    assert len(token_requests) == 1+    assert evaluation_requests == ["Bearer expired-token", "Bearer fresh-token"]+    assert guardrail.jwt_token == "fresh-token"+    largest_gap = max(b - a for a, b in zip(samples, samples[1:]))+    assert largest_gap < 0.25, f"event loop stalled for {largest_gap:.3f}s during JWT refresh" 
Merged fix · PR #45437+136 −6
litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.pyindex cfffff8eec1..421cf2281db 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import asyncio import os from collections.abc import Mapping, Sequence from typing import TYPE_CHECKING, Final, Literal, Protocol@@ -402,7 +403,7 @@ class HiddenlayerGuardrail(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/detection/v1/interactions",@@ -609,7 +610,7 @@ class HiddenlayerGuardrailV2(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/{path}",tests/unit/proxy/guardrails/guardrail_hooks/test_hiddenlayer.pyindex 954b9b99622..a6e84a9db0a 100644@@ -1,15 +1,17 @@+import asyncio import os import threading import time import uuid-from typing import List, cast+from typing import Final, List, Literal, cast from unittest.mock import AsyncMock, MagicMock, patch  import pytest-from fastapi import HTTPException-from httpx import Request, Response import requests-+import responses+import respx+from fastapi import HTTPException+from httpx import HTTPStatusError, Request, Response  import litellm from litellm import ModelResponse@@ -1153,3 +1155,130 @@ def test_get_jwt_gives_up_at_the_timeout_instead_of_blocking_the_event_loop(hang         guardrail.refresh_jwt_func()      assert [call.kwargs["timeout"] for call in get_jwt.call_args_list] == [2, 2]+++@pytest.fixture(params=("v1", "v2"))+def hiddenlayer_jwt_guardrail(+    request: pytest.FixtureRequest, httpx_transport: None+) -> HiddenlayerGuardrail | HiddenlayerGuardrailV2:+    guardrail_cls: Final = HiddenlayerGuardrail if request.param == "v1" else HiddenlayerGuardrailV2+    with responses.RequestsMock() as auth:+        auth.post(+            "https://auth.example.test/oauth2/token?grant_type=client_credentials", json={"access_token": "expired"}+        )+        return guardrail_cls(+            guardrail_name="hiddenlayer",+            api_id="id",+            api_key="secret",+            api_base="https://api.hiddenlayer.ai",+            auth_url="https://auth.example.test",+            timeout=2,+        )+++@pytest.mark.parametrize("input_type", ("request", "response"))+async def test_jwt_refresh_allows_other_coroutines_to_run_before_it_finishes(+    hiddenlayer_jwt_guardrail: HiddenlayerGuardrail | HiddenlayerGuardrailV2,+    input_type: Literal["request", "response"],+    respx_mock: respx.MockRouter,+) -> None:+    guardrail: Final = hiddenlayer_jwt_guardrail+    loop: Final = asyncio.get_running_loop()+    refresh_started: Final = asyncio.Event()+    release_refresh: Final = threading.Event()+    successful_body: Final = (+        {"evaluation": {"action": "Allow"}}+        if isinstance(guardrail, HiddenlayerGuardrail)+        else {"messages": [{"role": "user", "content": "hi"}]}+        if input_type == "request"+        else {"choices": [{"message": {"role": "assistant", "content": "hi"}}]}+    )++    def refresh_token(request: requests.PreparedRequest) -> tuple[int, dict[str, str], str]:+        assert request.headers["Authorization"] == "Basic aWQ6c2VjcmV0"+        loop.call_soon_threadsafe(refresh_started.set)+        assert release_refresh.wait(timeout=5), "event loop could not run while JWT refresh was pending"+        return 200, {}, '{"access_token": "fresh"}'++    def evaluate(request: Request) -> Response:+        return Response(+            401 if request.headers["Authorization"] == "Bearer expired" else 200,+            json=successful_body,+        )++    async def other_request() -> None:+        await refresh_started.wait()+        release_refresh.set()++    route: Final = respx_mock.route(method="POST", host="api.hiddenlayer.ai").mock(side_effect=evaluate)+    peer: Final = asyncio.create_task(other_request())+    with responses.RequestsMock() as auth:+        auth.add_callback(+            responses.POST,+            "https://auth.example.test/oauth2/token?grant_type=client_credentials",+            callback=refresh_token,+            content_type="application/json",+        )+        try:+            result, _ = await asyncio.wait_for(+                asyncio.gather(+                    guardrail.apply_guardrail(inputs={"texts": ["hi"]}, request_data={}, input_type=input_type),+                    peer,+                ),+                timeout=10,+            )+        finally:+            release_refresh.set()+            peer.cancel()+            await asyncio.gather(peer, return_exceptions=True)+        assert len(auth.calls) == 1++    assert result["texts"] == ["hi"]+    assert guardrail.jwt_token == "fresh"+    assert route.call_count == 2+    assert route.calls[0].request.headers["Authorization"] == "Bearer expired"+    assert route.calls[1].request.headers["Authorization"] == "Bearer fresh"+++@pytest.mark.parametrize("input_type", ("request", "response"))+async def test_jwt_refresh_exception_propagates_without_replacing_the_token(+    hiddenlayer_jwt_guardrail: HiddenlayerGuardrail | HiddenlayerGuardrailV2,+    input_type: Literal["request", "response"],+    respx_mock: respx.MockRouter,+) -> None:+    guardrail: Final = hiddenlayer_jwt_guardrail+    refresh_error: Final = requests.exceptions.Timeout("token endpoint timed out")+    route: Final = respx_mock.route(method="POST", host="api.hiddenlayer.ai").respond(401)+    with responses.RequestsMock() as auth:+        auth.post("https://auth.example.test/oauth2/token?grant_type=client_credentials", body=refresh_error)+        with pytest.raises(requests.exceptions.Timeout, match="token endpoint timed out") as caught:+            await guardrail.apply_guardrail(inputs={"texts": ["hi"]}, request_data={}, input_type=input_type)++    assert caught.value is refresh_error+    assert guardrail.jwt_token == "expired"+    assert route.call_count == 1+    assert route.calls[0].request.headers["Authorization"] == "Bearer expired"+++@pytest.mark.parametrize("input_type", ("request", "response"))+async def test_second_detection_401_propagates_without_another_jwt_refresh(+    hiddenlayer_jwt_guardrail: HiddenlayerGuardrail | HiddenlayerGuardrailV2,+    input_type: Literal["request", "response"],+    respx_mock: respx.MockRouter,+) -> None:+    guardrail: Final = hiddenlayer_jwt_guardrail+    route: Final = respx_mock.route(method="POST", host="api.hiddenlayer.ai").respond(401)+    with responses.RequestsMock() as auth:+        auth.post(+            "https://auth.example.test/oauth2/token?grant_type=client_credentials", json={"access_token": "fresh"}+        )+        with pytest.raises(HTTPStatusError) as caught:+            await guardrail.apply_guardrail(inputs={"texts": ["hi"]}, request_data={}, input_type=input_type)+        assert len(auth.calls) == 1++    assert caught.value.response.status_code == 401+    assert caught.value.request.headers["Authorization"] == "Bearer fresh"+    assert guardrail.jwt_token == "fresh"+    assert route.call_count == 2+    assert route.calls[0].request.headers["Authorization"] == "Bearer expired"+    assert route.calls[1].request.headers["Authorization"] == "Bearer fresh"