Backtestslitellmrun-20261019-033420#44921

Issue #44921 Merged fix PR #45437 Run 19 Oct 2026 · run-20261019-033420 BerriAI/litellm

[Bug]: HiddenLayer JWT refresh blocks the event loop inside the async guardrail request

Tie

The judges split evenly between our patch and the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

The production change is byte-for-byte equivalent in both patches: each wraps `self.refresh_jwt_func()` in `await asyncio.to_thread(...)` for both HiddenlayerGuardrail and HiddenlayerGuardrailV2, which moves the blocking requests-based token refresh off the event loop and fixes the root cause. Independent tests confirm both pass identically. The difference lies in the tests. Patch A's primary test is deterministic: the token-refresh callback blocks on a threading.Event that only a peer coroutine can release, so the test can only pass if the event loop keeps running during the refresh. It also verifies the retry uses the fresh bearer token, and adds two edge tests (refresh exception propagates without clobbering the stored token; a second 401 after refresh propagates without another refresh), covering both v1/v2 and request/response paths. Patch B's test is a faithful port of the repro (real loopback server, 0.5s sleep, heartbeat gap < 0.25s), which is directly tied to the issue and self-contained, but it is timing-based and therefore inherently more prone to flakiness on loaded CI; its only other change is a docstring tweak. Patch A carries some risk in depending on `responses`/`respx` and an `httpx_transport` fixture that cannot be verified here, and its extra tests are beyond the strict ask, but they add no production risk. On balance A's deterministic concurrency test is the stronger regression guard for an event-loop-blocking bug, so A edges out B; the fix itself is a wash.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

20m 12s

Wall-clock time in the container

Full test suite

No new failures

113 passed · 0 skipped · 113 total

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

105vs 142 merged

Changed lines, ours vs merged

Description

The async HiddenLayer guardrail refreshes an expired access token synchronously after a 401. A deliberately slow local token endpoint stalls an independent event-loop heartbeat

Expected: asynchronous guardrail execution does not block unrelated requests during token refresh

Verified against unmodified upstream main at 0 on 2026-10-08 UTC. Fresh controlled HTTP token refresh produces a 208 ms event-loop heartbeat gap

Config

The reproduction runs from an unmodified upstream checkout and starts a controlled loopback HTTP service. It does not require proxy YAML or provider credentials

LiteLLM Version

1.106.0, upstream main commit 0, verified 2026-10-08 UTC

Steps to Repro

Save this as repro.py in the upstream checkout

import asyncio
import json
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

class Service(BaseHTTPRequestHandler):
    token_requests = []
    evaluation_requests = []

    def log_message(self, *args):
        pass

    def do_POST(self):
        self.rfile.read(int(self.headers["Content-Length"]))
        status = 200
        if "token" in self.path:
            self.token_requests.append(self.path)
            time.sleep(0.2)
            payload = {"access_token": "local-token", "expires_in": 3600, "token_type": "Bearer"}
        elif self.path.endswith("interactions"):
            self.evaluation_requests.append(self.path)
            status = 401 if len(self.evaluation_requests) == 1 else 200
            payload = {"evaluation": {"action": "ALLOW"}, "analysis": []}
        else:
            payload = {"id": "chatcmpl-local", "object": "chat.completion", "created": int(time.time()),
                       "model": "validation-endpoint", "choices": [{"index": 0, "message": {"role": "assistant", "content": "hello"}, "finish_reason": "stop"}],
                       "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}}
        body = json.dumps(payload).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

server = ThreadingHTTPServer(("127.0.0.1", 0), Service)
threading.Thread(target=server.serve_forever, daemon=True).start()
base = f"http://127.0.0.1:{server.server_port}"

from functools import partial
from litellm.proxy.guardrails.guardrail_hooks.hiddenlayer.hiddenlayer import HiddenlayerGuardrail, _get_jwt

async def heartbeat(samples):
    while True:
        samples.append(time.perf_counter())
        await asyncio.sleep(0.01)

async def main():
    guardrail = HiddenlayerGuardrail(api_base=base, guardrail_name="local-check")
    guardrail.jwt_token = "expired-local-placeholder"
    guardrail.refresh_jwt_func = partial(_get_jwt, auth_url=base, api_id="local-client", api_key="local-placeholder")
    samples = []
    monitor = asyncio.create_task(heartbeat(samples))
    await asyncio.sleep(0.02)
    result = await guardrail.apply_guardrail(inputs={"texts": ["ordinary request"]}, request_data={}, input_type="request")
    await asyncio.sleep(0.02)
    monitor.cancel()
    await asyncio.gather(monitor, return_exceptions=True)
    print("guardrail result", result)
    print("evaluation requests", len(Service.evaluation_requests))
    print("refresh requests", len(Service.token_requests))
    print("largest heartbeat gap", round(max(b - a for a, b in zip(samples, samples[1:])), 3))

try:
    asyncio.run(main())
finally:
    server.shutdown()
    server.server_close()

Run it with the checkout on the import path

LITELLM_LOCAL_MODEL_COST_MAP=true PYTHONPATH=. python repro.py

Observed:

guardrail result {'texts': ['ordinary request']}
evaluation requests 2
refresh requests 1
largest heartbeat gap 0.208

Expected: the invariant described above holds without the observed failure

Which part of LiteLLM is this about?

Guardrails: moderation, PII masking, policies

How are you deploying?

pip / Python SDK

The two patches

Our patch+101 −4
litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.pyindex cfffff8eec..3b67d9e88f 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import asyncio import os from collections.abc import Mapping, Sequence from typing import TYPE_CHECKING, Final, Literal, Protocol@@ -402,7 +403,8 @@ class HiddenlayerGuardrail(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                # The refresh is a blocking HTTP call; run it off the event loop.+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/detection/v1/interactions",@@ -609,7 +611,8 @@ class HiddenlayerGuardrailV2(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                # The refresh is a blocking HTTP call; run it off the event loop.+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/{path}",tests/unit/proxy/guardrails/guardrail_hooks/test_hiddenlayer.pyindex 954b9b9962..33aa4fdc19 100644@@ -1130,8 +1130,8 @@ def hanging_auth_server():  def test_get_jwt_gives_up_at_the_timeout_instead_of_blocking_the_event_loop(hanging_auth_server):     """-    `_get_jwt` runs synchronously inside `_call_hiddenlayer`, so an auth host that-    accepts and never answers used to park the whole worker's event loop.+    `_get_jwt` is a blocking call, so an auth host that accepts and never answers+    used to park the whole worker's event loop (and still ties up a refresh thread).     """     started = time.monotonic()     with pytest.raises(requests.exceptions.Timeout):@@ -1153,3 +1153,97 @@ def test_get_jwt_gives_up_at_the_timeout_instead_of_blocking_the_event_loop(hang         guardrail.refresh_jwt_func()      assert [call.kwargs["timeout"] for call in get_jwt.call_args_list] == [2, 2]+++@pytest.fixture+def slow_token_refresh_server():+    """Rejects the first evaluation with 401 and answers token requests slowly, so a refresh is forced."""+    import json+    from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++    requests_seen: dict = {"token": 0, "evaluation": 0, "authorization": []}++    class Handler(BaseHTTPRequestHandler):+        def log_message(self, format, *args):+            pass++        def do_POST(self):+            self.rfile.read(int(self.headers.get("Content-Length") or 0))+            status = 200+            if "oauth2/token" in self.path:+                requests_seen["token"] += 1+                time.sleep(0.5)+                payload: dict = {"access_token": "fresh-token", "expires_in": 3600, "token_type": "Bearer"}+            else:+                requests_seen["evaluation"] += 1+                requests_seen["authorization"].append(self.headers.get("Authorization"))+                status = 401 if requests_seen["evaluation"] == 1 else 200+                if self.path.endswith("interactions"):+                    payload = {"evaluation": {"action": "Allow"}, "analysis": []}+                else:+                    payload = {"messages": [{"role": "user", "content": "ordinary request"}]}+            body = json.dumps(payload).encode()+            self.send_response(status)+            self.send_header("Content-Type", "application/json")+            self.send_header("Content-Length", str(len(body)))+            self.end_headers()+            self.wfile.write(body)++    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)+    server.daemon_threads = True+    thread = threading.Thread(target=server.serve_forever, daemon=True)+    thread.start()+    try:+        yield f"http://127.0.0.1:{server.server_port}", requests_seen+    finally:+        server.shutdown()+        server.server_close()+        thread.join(timeout=5)+++@pytest.mark.asyncio+@pytest.mark.parametrize(+    "guardrail_cls, inputs, expected_texts",+    [+        (HiddenlayerGuardrail, {"texts": ["ordinary request"]}, ["ordinary request"]),+        (+            HiddenlayerGuardrailV2,+            {"structured_messages": [{"role": "user", "content": "ordinary request"}], "model": "gpt-4o"},+            ["ordinary request"],+        ),+    ],+)+async def test_jwt_refresh_after_401_does_not_block_the_event_loop(+    slow_token_refresh_server, guardrail_cls, inputs, expected_texts+):+    """A 401-triggered JWT refresh must not stall unrelated work on the event loop."""+    import asyncio+    from functools import partial++    base, requests_seen = slow_token_refresh_server+    guardrail = guardrail_cls(api_base=base, guardrail_name="hiddenlayer")+    guardrail.jwt_token = "expired-token"+    guardrail.refresh_jwt_func = partial(_get_jwt, auth_url=base, api_id="id", api_key="secret")++    samples: List[float] = []++    async def heartbeat():+        while True:+            samples.append(time.perf_counter())+            await asyncio.sleep(0.01)++    monitor = asyncio.create_task(heartbeat())+    await asyncio.sleep(0.02)+    try:+        result = await guardrail.apply_guardrail(inputs=inputs, request_data={}, input_type="request")+        await asyncio.sleep(0.02)+    finally:+        monitor.cancel()+        await asyncio.gather(monitor, return_exceptions=True)++    assert result["texts"] == expected_texts+    assert requests_seen["token"] == 1+    assert requests_seen["authorization"] == ["Bearer expired-token", "Bearer fresh-token"]+    assert guardrail.jwt_token == "fresh-token"+    largest_gap = max(b - a for a, b in zip(samples, samples[1:]))+    assert largest_gap < 0.25, f"event loop stalled for {largest_gap:.3f}s during the token refresh" 
Merged fix · PR #45437+136 −6
litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.pyindex cfffff8eec1..421cf2281db 100644@@ -1,5 +1,6 @@ from __future__ import annotations +import asyncio import os from collections.abc import Mapping, Sequence from typing import TYPE_CHECKING, Final, Literal, Protocol@@ -402,7 +403,7 @@ class HiddenlayerGuardrail(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/detection/v1/interactions",@@ -609,7 +610,7 @@ class HiddenlayerGuardrailV2(CustomGuardrail):                 verbose_proxy_logger.debug(                     "Unable to authenticate to Hiddenlayer, JWT token is invalid or expired, trying to refresh the token."                 )-                self.jwt_token = self.refresh_jwt_func()+                self.jwt_token = await asyncio.to_thread(self.refresh_jwt_func)                 headers["Authorization"] = f"Bearer {self.jwt_token}"                 response = await self._http_client.post(                     f"{self.api_base}/{path}",tests/unit/proxy/guardrails/guardrail_hooks/test_hiddenlayer.pyindex 954b9b99622..a6e84a9db0a 100644@@ -1,15 +1,17 @@+import asyncio import os import threading import time import uuid-from typing import List, cast+from typing import Final, List, Literal, cast from unittest.mock import AsyncMock, MagicMock, patch  import pytest-from fastapi import HTTPException-from httpx import Request, Response import requests-+import responses+import respx+from fastapi import HTTPException+from httpx import HTTPStatusError, Request, Response  import litellm from litellm import ModelResponse@@ -1153,3 +1155,130 @@ def test_get_jwt_gives_up_at_the_timeout_instead_of_blocking_the_event_loop(hang         guardrail.refresh_jwt_func()      assert [call.kwargs["timeout"] for call in get_jwt.call_args_list] == [2, 2]+++@pytest.fixture(params=("v1", "v2"))+def hiddenlayer_jwt_guardrail(+    request: pytest.FixtureRequest, httpx_transport: None+) -> HiddenlayerGuardrail | HiddenlayerGuardrailV2:+    guardrail_cls: Final = HiddenlayerGuardrail if request.param == "v1" else HiddenlayerGuardrailV2+    with responses.RequestsMock() as auth:+        auth.post(+            "https://auth.example.test/oauth2/token?grant_type=client_credentials", json={"access_token": "expired"}+        )+        return guardrail_cls(+            guardrail_name="hiddenlayer",+            api_id="id",+            api_key="secret",+            api_base="https://api.hiddenlayer.ai",+            auth_url="https://auth.example.test",+            timeout=2,+        )+++@pytest.mark.parametrize("input_type", ("request", "response"))+async def test_jwt_refresh_allows_other_coroutines_to_run_before_it_finishes(+    hiddenlayer_jwt_guardrail: HiddenlayerGuardrail | HiddenlayerGuardrailV2,+    input_type: Literal["request", "response"],+    respx_mock: respx.MockRouter,+) -> None:+    guardrail: Final = hiddenlayer_jwt_guardrail+    loop: Final = asyncio.get_running_loop()+    refresh_started: Final = asyncio.Event()+    release_refresh: Final = threading.Event()+    successful_body: Final = (+        {"evaluation": {"action": "Allow"}}+        if isinstance(guardrail, HiddenlayerGuardrail)+        else {"messages": [{"role": "user", "content": "hi"}]}+        if input_type == "request"+        else {"choices": [{"message": {"role": "assistant", "content": "hi"}}]}+    )++    def refresh_token(request: requests.PreparedRequest) -> tuple[int, dict[str, str], str]:+        assert request.headers["Authorization"] == "Basic aWQ6c2VjcmV0"+        loop.call_soon_threadsafe(refresh_started.set)+        assert release_refresh.wait(timeout=5), "event loop could not run while JWT refresh was pending"+        return 200, {}, '{"access_token": "fresh"}'++    def evaluate(request: Request) -> Response:+        return Response(+            401 if request.headers["Authorization"] == "Bearer expired" else 200,+            json=successful_body,+        )++    async def other_request() -> None:+        await refresh_started.wait()+        release_refresh.set()++    route: Final = respx_mock.route(method="POST", host="api.hiddenlayer.ai").mock(side_effect=evaluate)+    peer: Final = asyncio.create_task(other_request())+    with responses.RequestsMock() as auth:+        auth.add_callback(+            responses.POST,+            "https://auth.example.test/oauth2/token?grant_type=client_credentials",+            callback=refresh_token,+            content_type="application/json",+        )+        try:+            result, _ = await asyncio.wait_for(+                asyncio.gather(+                    guardrail.apply_guardrail(inputs={"texts": ["hi"]}, request_data={}, input_type=input_type),+                    peer,+                ),+                timeout=10,+            )+        finally:+            release_refresh.set()+            peer.cancel()+            await asyncio.gather(peer, return_exceptions=True)+        assert len(auth.calls) == 1++    assert result["texts"] == ["hi"]+    assert guardrail.jwt_token == "fresh"+    assert route.call_count == 2+    assert route.calls[0].request.headers["Authorization"] == "Bearer expired"+    assert route.calls[1].request.headers["Authorization"] == "Bearer fresh"+++@pytest.mark.parametrize("input_type", ("request", "response"))+async def test_jwt_refresh_exception_propagates_without_replacing_the_token(+    hiddenlayer_jwt_guardrail: HiddenlayerGuardrail | HiddenlayerGuardrailV2,+    input_type: Literal["request", "response"],+    respx_mock: respx.MockRouter,+) -> None:+    guardrail: Final = hiddenlayer_jwt_guardrail+    refresh_error: Final = requests.exceptions.Timeout("token endpoint timed out")+    route: Final = respx_mock.route(method="POST", host="api.hiddenlayer.ai").respond(401)+    with responses.RequestsMock() as auth:+        auth.post("https://auth.example.test/oauth2/token?grant_type=client_credentials", body=refresh_error)+        with pytest.raises(requests.exceptions.Timeout, match="token endpoint timed out") as caught:+            await guardrail.apply_guardrail(inputs={"texts": ["hi"]}, request_data={}, input_type=input_type)++    assert caught.value is refresh_error+    assert guardrail.jwt_token == "expired"+    assert route.call_count == 1+    assert route.calls[0].request.headers["Authorization"] == "Bearer expired"+++@pytest.mark.parametrize("input_type", ("request", "response"))+async def test_second_detection_401_propagates_without_another_jwt_refresh(+    hiddenlayer_jwt_guardrail: HiddenlayerGuardrail | HiddenlayerGuardrailV2,+    input_type: Literal["request", "response"],+    respx_mock: respx.MockRouter,+) -> None:+    guardrail: Final = hiddenlayer_jwt_guardrail+    route: Final = respx_mock.route(method="POST", host="api.hiddenlayer.ai").respond(401)+    with responses.RequestsMock() as auth:+        auth.post(+            "https://auth.example.test/oauth2/token?grant_type=client_credentials", json={"access_token": "fresh"}+        )+        with pytest.raises(HTTPStatusError) as caught:+            await guardrail.apply_guardrail(inputs={"texts": ["hi"]}, request_data={}, input_type=input_type)+        assert len(auth.calls) == 1++    assert caught.value.response.status_code == 401+    assert caught.value.request.headers["Authorization"] == "Bearer fresh"+    assert guardrail.jwt_token == "fresh"+    assert route.call_count == 2+    assert route.calls[0].request.headers["Authorization"] == "Bearer expired"+    assert route.calls[1].request.headers["Authorization"] == "Bearer fresh"