Backtestslitellmrun-20261019-033420#40846
[Bug]: Completed /v1/responses requests retain max_parallel_requests slots during deferred logging
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the root cause the same way: releasing the max_parallel_requests slot in async_post_call_success_hook and making the release idempotent so the later success/failure callback does not double-release. Both pass all five independent tests. The deciding differences are on the edges. Patch A guards the new post-call release with `not isinstance(response, AsyncIterator)`, so if a streaming wrapper ever reaches the success hook its slot stays held until the stream finishes; Patch B releases unconditionally, which would silently drop concurrency enforcement for any in-flight stream routed through that hook. A also resolves the stash via the request contextvar (`get_request_stash()`), which is the natural source inside a post-call hook, while B derives it from `_call_id_from_callback_kwargs(data)` on request data, adding a dependency on `litellm_call_id` being present in `data`. A's idempotence is achieved by detaching the acquisition before the await, a minimal change with no new state; B adds an `asyncio.Lock` field to the stash dataclass and refactors two additional pre-call release sites, more churn than the issue requires. On tests, A covers both aresponses and acompletion, asserts the late success event does not free the next request's slot, and adds an explicit in-flight-stream test; B's concurrent-redis-failure test is a good addition but B has no streaming coverage. Overall A is the tighter, safer fix with better-targeted tests.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
44m 48s
Wall-clock time in the container
Full test suite
No new failures
1,800 passed · 1 skipped · 1,801 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
223vs 199 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
A completed non-streaming /v1/responses request can continue occupying its max_parallel_requests slot until asynchronous success logging finishes. With a virtual key limited to one parallel request, the first request returns 200, but an identical request sent immediately afterward can incorrectly receive 429. The next request succeeds once success logging completes.
The slot should be released before the completed response is returned to the caller. Slow logging integrations should not extend the request's concurrency lifetime.
This is related to #27955, but differs from the persistent slot leaks reported there: in this case the slot is eventually released, just too late. A proposed fix is in #40843.
User Flow
Before a fix: a developer using the Responses API receives a false rate-limit response after the previous request has already completed.
- The proxy administrator creates a virtual key with
max_parallel_requests: 1. - The developer sends a non-streaming
POST http://localhost:4000/v1/responsesrequest. - The completed request returns
200. - The developer immediately sends the same request again and receives
429. - After success logging finishes, the developer retries and receives
200.
After a fix: the completed request releases its slot before returning, so the next request is admitted immediately.
- The proxy administrator creates a virtual key with
max_parallel_requests: 1. - The developer sends the same non-streaming
POST http://localhost:4000/v1/responsesrequest. - The completed request returns
200. - The developer immediately sends the same request again and also receives
200. - Later requests continue receiving
200after success logging finishes.
Proof the bug occurs
The issue was reproduced against a real OpenAI-compatible Responses deployment with a local LiteLLM proxy, a virtual key limited to one parallel request, and a custom success logger that takes three seconds to complete. Secrets and provider-specific endpoint values are represented by environment variables below.
slow_callback.py:
import asyncio
from litellm.integrations.custom_logger import CustomLogger
class SlowSuccessLogger(CustomLogger):
async def async_log_success_event(
self, kwargs, response_obj, start_time, end_time
) -> None:
await asyncio.sleep(3)
slow_success_logger = SlowSuccessLogger()
Relevant config.yaml:
model_list:
- model_name: responses-test
litellm_params:
model: openai/<model-name>
api_base: os.environ/OPENAI_API_BASE
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
callbacks: slow_callback.slow_success_logger
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
Version tested before the fix:
litellm_internal_staging at f90b5cad8a973eb8ddd94b3c8381c4ed83bba9f2
The same non-streaming request was sent three times: once initially, once immediately after the first response, and once after the three-second logging delay.
first=200
immediate_next=429
after_logging=200
With the proposed fix from #40843 applied:
first=200
immediate_next=200
after_logging=200
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
litellm_internal_staging at f90b5cad8a973eb8ddd94b3c8381c4ed83bba9f2
Twitter / LinkedIn details
_No response_
The two patches
litellm/proxy/hooks/parallel_request_limiter_v3.pyindex f72720b472..59ad8b7701 100644@@ -9,7 +9,7 @@ import binascii import logging import os import uuid-from collections.abc import AsyncGenerator, Awaitable, Callable, Mapping, Sequence, Set+from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable, Mapping, Sequence, Set from contextlib import asynccontextmanager from contextvars import ContextVar from dataclasses import dataclass, field@@ -4524,13 +4524,8 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): verbose_proxy_logger.debug("INSIDE parallel request limiter ASYNC SUCCESS LOGGING") stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs))- acquisition: Final = stash.parallel_slot if stash is not None else None- if stash is not None and acquisition is not None:- await self._release_parallel_request_slots(- acquisition=acquisition,- parent_otel_span=litellm_parent_otel_span,- )- stash.parallel_slot = None+ if stash is not None:+ await self._release_stashed_parallel_slot(stash=stash, parent_otel_span=litellm_parent_otel_span) pipeline_operations: Final = self._build_success_event_pipeline_operations( kwargs=kwargs,@@ -4650,13 +4645,8 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): pipeline_operations: Final[list[RedisPipelineIncrementOperation]] = [] stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs))- acquisition: Final = stash.parallel_slot if stash is not None else None- if stash is not None and acquisition is not None:- await self._release_parallel_request_slots(- acquisition=acquisition,- parent_otel_span=litellm_parent_otel_span,- )- stash.parallel_slot = None+ if stash is not None:+ await self._release_stashed_parallel_slot(stash=stash, parent_otel_span=litellm_parent_otel_span) # Skip the reservation refund if async_post_call_failure_hook # already released it (proxy-level rejection that also bubbles up@@ -4765,21 +4755,54 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): change mid-request) decides whether there is anything to release. """ stash: Final = get_request_stash()- if stash is None or stash.parallel_slot is None:+ if stash is None: return + await self._release_stashed_parallel_slot(stash=stash, parent_otel_span=None)++ async def _release_stashed_parallel_slot(+ self,+ stash: RequestRateLimiterStash,+ parent_otel_span: Span | None,+ ) -> None:+ """+ Release the request's max_parallel_requests slot at most once. The+ acquisition is detached from the stash before the release is awaited,+ so a sibling release path (post-call hook vs. the logging worker's+ success callback) that runs while this one is suspended sees no slot+ and cannot double-decrement the in-memory fallback.+ """+ acquisition: Final = stash.parallel_slot+ if acquisition is None:+ return+ stash.parallel_slot = None await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,- parent_otel_span=None,+ acquisition=acquisition,+ parent_otel_span=parent_otel_span, )- stash.parallel_slot = None async def async_post_call_success_hook(self, data: dict, user_api_key_dict: UserAPIKeyAuth, response): """- Post-call hook to update rate limit headers in the response.+ Post-call hook to update rate limit headers in the response, and to+ release the max_parallel_requests slot of a completed non-streaming+ response before it is returned to the caller. Success logging runs+ asynchronously after the response is sent, so leaving the release to+ ``async_log_success_event`` would keep the slot occupied for as long+ as the slowest logging integration takes. Streams are still in flight+ here; their slot stays held until the stream's success/failure event+ or the disconnect cleanup releases it. """+ stash: Final = get_request_stash()+ if stash is not None and not isinstance(response, AsyncIterator):+ try:+ await self._release_stashed_parallel_slot(+ stash=stash,+ parent_otel_span=user_api_key_dict.parent_otel_span,+ )+ except Exception as e: # noqa: BLE001 # slot release must never fail a completed response+ verbose_proxy_logger.exception("Error releasing parallel request slot in post-call hook: %s", e)+ try:- stash: Final = get_request_stash() litellm_proxy_rate_limit_response: Final = stash.rate_limit_response if stash is not None else None if litellm_proxy_rate_limit_response is not None and response_has_hidden_params(response):@@ -4848,12 +4871,10 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): stash: Final = get_request_stash() if stash is None: return- if stash.parallel_slot is not None:- await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,- parent_otel_span=user_api_key_dict.parent_otel_span,- )- stash.parallel_slot = None+ await self._release_stashed_parallel_slot(+ stash=stash,+ parent_otel_span=user_api_key_dict.parent_otel_span,+ ) if stash.batch_enqueued_reservation is not None: await self.batch_enqueued_token_store.refund(tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.pyindex 8d7ab89f35..fc56907c32 100644@@ -4254,6 +4254,154 @@ async def test_success_event_releases_parallel_slot_v3(monkeypatch): ) +def _completed_responses_api_response():+ from litellm.types.llms.openai import ResponsesAPIResponse++ return ResponsesAPIResponse(+ id="resp_1",+ created_at=int(time.time()),+ model="gpt-4o",+ object="response",+ output=[],+ status="completed",+ parallel_tool_calls=False,+ tool_choice="auto",+ tools=[],+ )+++@pytest.mark.parametrize(+ "call_type, make_response",+ [+ ("aresponses", _completed_responses_api_response),+ (+ "acompletion",+ lambda: ModelResponse(+ usage=Usage(prompt_tokens=5, completion_tokens=5, total_tokens=10)+ ),+ ),+ ],+)+@pytest.mark.asyncio+async def test_post_call_success_hook_releases_parallel_slot_before_deferred_logging_v3(+ monkeypatch, call_type, make_response+):+ """+ A completed non-streaming request must free its max_parallel_requests+ slot before the response is returned. Success logging runs later on the+ logging worker, so with max_parallel_requests=1 a request sent right+ after the first one returned was rejected with 429 until a slow logging+ integration finished. The late success event must not then free the+ slot the next request holds.+ """+ _api_key = hash_token("sk-12345")+ local_cache = DualCache()+ handler = _PROXY_MaxParallelRequestsHandler(+ internal_usage_cache=InternalUsageCache(local_cache)+ )+ monkeypatch.setattr(handler, "get_rate_limit_type", lambda: "total")+ user_api_key_dict = UserAPIKeyAuth(api_key=_api_key, max_parallel_requests=1)+ counter_key = f"{{api_key:{_api_key}}}:max_parallel_requests"++ async def in_flight() -> int:+ return handler._gauge_in_flight_from_cache_value(+ await local_cache.async_get_cache(key=counter_key)+ )++ first_response_returned = asyncio.Event()+ run_deferred_logging = asyncio.Event()++ async def first_request() -> None:+ data: Dict[str, Any] = {"model": "gpt-4o"}+ await handler.async_pre_call_hook(+ user_api_key_dict=user_api_key_dict,+ cache=local_cache,+ data=data,+ call_type=call_type,+ )+ response = make_response()+ await handler.async_post_call_success_hook(+ data=data, user_api_key_dict=user_api_key_dict, response=response+ )+ first_response_returned.set()+ # deferred success logging of the first request, run late+ await run_deferred_logging.wait()+ await handler.async_log_success_event(+ kwargs={+ "standard_logging_object": {+ "metadata": {"user_api_key_hash": _api_key}+ },+ },+ response_obj=response,+ start_time=datetime.now(),+ end_time=datetime.now(),+ )++ first_task = asyncio.create_task(first_request())+ await first_response_returned.wait()+ assert await in_flight() == 0++ async def second_request() -> None:+ await handler.async_pre_call_hook(+ user_api_key_dict=user_api_key_dict,+ cache=local_cache,+ data={"model": "gpt-4o"},+ call_type=call_type,+ )++ # admitted immediately instead of a 429+ await asyncio.create_task(second_request())+ assert await in_flight() == 1++ run_deferred_logging.set()+ await first_task+ assert await in_flight() == 1+++@pytest.mark.asyncio+async def test_post_call_success_hook_keeps_slot_for_in_flight_stream_v3():+ """+ A streaming response is still running when post-call hooks see it; its+ slot stays held until the stream's own success/failure event releases it.+ """+ _api_key = hash_token("sk-12345")+ local_cache = DualCache()+ handler = _PROXY_MaxParallelRequestsHandler(+ internal_usage_cache=InternalUsageCache(local_cache)+ )+ user_api_key_dict = UserAPIKeyAuth(api_key=_api_key, max_parallel_requests=1)+ counter_key = f"{{api_key:{_api_key}}}:max_parallel_requests"++ async def stream():+ yield "chunk"++ data: Dict[str, Any] = {"model": "gpt-4o"}+ await handler.async_pre_call_hook(+ user_api_key_dict=user_api_key_dict,+ cache=local_cache,+ data=data,+ call_type="aresponses",+ )+ response = stream()+ await handler.async_post_call_success_hook(+ data=data, user_api_key_dict=user_api_key_dict, response=response+ )+ await response.aclose()++ assert get_request_stash().parallel_slot is not None+ assert handler._gauge_in_flight_from_cache_value(+ await local_cache.async_get_cache(key=counter_key)+ ) == 1+ with pytest.raises(HTTPException) as exc_info:+ await handler.async_pre_call_hook(+ user_api_key_dict=user_api_key_dict,+ cache=local_cache,+ data={"model": "gpt-4o"},+ call_type="aresponses",+ )+ assert exc_info.value.status_code == 429++ @pytest.mark.asyncio async def test_read_only_gauge_check_counts_without_acquiring_v3(): """
litellm/proxy/hooks/parallel_request_limiter_v3.pyindex f72720b4726..89826694bb6 100644@@ -531,6 +531,7 @@ class RequestRateLimiterStash: owner_litellm_call_id: str | None = None rate_limit_response: RateLimitResponse | None = None parallel_slot: ParallelSlotAcquisition | None = None+ parallel_slot_release_lock: asyncio.Lock = field(default_factory=asyncio.Lock, repr=False, compare=False) reserved_tokens: int = 0 reserved_model: str | None = None reserved_scopes: frozenset[tuple[str, str]] = field(default_factory=frozenset)@@ -1620,6 +1621,20 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): statuses.append(self._gauge_status(gauge, in_flight + 1, "OK")) return RateLimitResponse(overall_code="OK", statuses=statuses) + async def _release_stashed_parallel_slot(+ self,+ stash: RequestRateLimiterStash | None,+ parent_otel_span: Span | None,+ ) -> None:+ if stash is None:+ return+ async with stash.parallel_slot_release_lock:+ acquisition: Final = stash.parallel_slot+ if acquisition is None:+ return+ await self._release_parallel_request_slots(acquisition, parent_otel_span)+ stash.parallel_slot = None # rebind-ok: marks this request's slot as released+ async def _release_parallel_request_slots( self, acquisition: ParallelSlotAcquisition,@@ -3379,13 +3394,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): parent_otel_span=user_api_key_dict.parent_otel_span, ) stash.reservation_released = True- acquisition: Final = stash.parallel_slot- if acquisition is not None:- await self._release_parallel_request_slots(- acquisition=acquisition,- parent_otel_span=user_api_key_dict.parent_otel_span,- )- stash.parallel_slot = None+ await self._release_stashed_parallel_slot(stash, user_api_key_dict.parent_otel_span) self._handle_rate_limit_error( response=io_response, descriptors=descriptors,@@ -3700,13 +3709,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): ) if tpm_response["overall_code"] == "OVER_LIMIT":- acquisition: Final = stash.parallel_slot- if acquisition is not None:- await self._release_parallel_request_slots(- acquisition=acquisition,- parent_otel_span=user_api_key_dict.parent_otel_span,- )- stash.parallel_slot = None+ await self._release_stashed_parallel_slot(stash, user_api_key_dict.parent_otel_span) self._handle_rate_limit_error( response=tpm_response, descriptors=descriptors,@@ -4524,13 +4527,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): verbose_proxy_logger.debug("INSIDE parallel request limiter ASYNC SUCCESS LOGGING") stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs))- acquisition: Final = stash.parallel_slot if stash is not None else None- if stash is not None and acquisition is not None:- await self._release_parallel_request_slots(- acquisition=acquisition,- parent_otel_span=litellm_parent_otel_span,- )- stash.parallel_slot = None+ await self._release_stashed_parallel_slot(stash, litellm_parent_otel_span) pipeline_operations: Final = self._build_success_event_pipeline_operations( kwargs=kwargs,@@ -4650,13 +4647,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): pipeline_operations: Final[list[RedisPipelineIncrementOperation]] = [] stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(kwargs))- acquisition: Final = stash.parallel_slot if stash is not None else None- if stash is not None and acquisition is not None:- await self._release_parallel_request_slots(- acquisition=acquisition,- parent_otel_span=litellm_parent_otel_span,- )- stash.parallel_slot = None+ await self._release_stashed_parallel_slot(stash, litellm_parent_otel_span) # Skip the reservation refund if async_post_call_failure_hook # already released it (proxy-level rejection that also bubbles up@@ -4764,23 +4755,23 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): object's current max_parallel_requests configuration, which can change mid-request) decides whether there is anything to release. """- stash: Final = get_request_stash()- if stash is None or stash.parallel_slot is None:- return-- await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,- parent_otel_span=None,- )- stash.parallel_slot = None+ await self._release_stashed_parallel_slot(get_request_stash(), None) async def async_post_call_success_hook(self, data: dict, user_api_key_dict: UserAPIKeyAuth, response): """- Post-call hook to update rate limit headers in the response.+ Release completed-request slots and update rate limit headers in the response. """ try:- stash: Final = get_request_stash()- litellm_proxy_rate_limit_response: Final = stash.rate_limit_response if stash is not None else None+ slot_stash: Final = get_request_stash_for_call(_call_id_from_callback_kwargs(data))+ await self._release_stashed_parallel_slot(slot_stash, user_api_key_dict.parent_otel_span)+ except Exception as e:+ verbose_proxy_logger.exception("Error releasing parallel request slot in post-call hook: %s", e)++ try:+ header_stash: Final = get_request_stash()+ litellm_proxy_rate_limit_response: Final = (+ header_stash.rate_limit_response if header_stash is not None else None+ ) if litellm_proxy_rate_limit_response is not None and response_has_hidden_params(response): additional_headers: Final = ensure_response_additional_headers(response)@@ -4848,12 +4839,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger): stash: Final = get_request_stash() if stash is None: return- if stash.parallel_slot is not None:- await self._release_parallel_request_slots(- acquisition=stash.parallel_slot,- parent_otel_span=user_api_key_dict.parent_otel_span,- )- stash.parallel_slot = None+ await self._release_stashed_parallel_slot(stash, user_api_key_dict.parent_otel_span) if stash.batch_enqueued_reservation is not None: await self.batch_enqueued_token_store.refund(tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.pyindex 8d7ab89f354..85023b94207 100644@@ -7,6 +7,7 @@ import logging import os import sys import time+from collections.abc import Sequence from contextlib import contextmanager from datetime import datetime, timedelta from typing import Any, Dict, List, Optional@@ -32,6 +33,7 @@ from litellm.proxy.hooks.parallel_request_limiter_v3 import ( ) from litellm.proxy.utils import InternalUsageCache, ProxyLogging, hash_token from litellm.types.caching import RedisPipelineIncrementOperation+from litellm.types.llms.openai import ResponsesAPIResponse from litellm.types.utils import ( EmbeddingResponse, ModelResponse,@@ -4054,6 +4056,125 @@ async def _seed_max_parallel_requests_slots( ) +@pytest.mark.asyncio+async def test_completed_responses_post_call_releases_parallel_slot() -> None:+ api_key = hash_token("sk-responses-post-call")+ local_cache = DualCache()+ handler = _PROXY_MaxParallelRequestsHandler(+ internal_usage_cache=InternalUsageCache(local_cache)+ )+ user_api_key_dict = UserAPIKeyAuth(api_key=api_key, max_parallel_requests=1)+ data = {+ "model": "gpt-4o-mini",+ "input": "hello",+ "litellm_call_id": "responses-owner",+ }+ parallel_key = f"{{api_key:{api_key}}}:max_parallel_requests"++ await handler.async_pre_call_hook(+ user_api_key_dict=user_api_key_dict,+ cache=local_cache,+ data=data,+ call_type="aresponses",+ )+ assert handler._gauge_in_flight_from_cache_value(+ await local_cache.async_get_cache(key=parallel_key)+ ) == 1++ await handler.async_post_call_success_hook(+ data=data,+ user_api_key_dict=user_api_key_dict,+ response=ResponsesAPIResponse(+ id="resp_parallel_slot",+ created_at=0,+ model="gpt-4o-mini",+ object="response",+ output=[],+ status="completed",+ ),+ )+ assert handler._gauge_in_flight_from_cache_value(+ await local_cache.async_get_cache(key=parallel_key)+ ) == 0++ await handler.async_log_success_event(+ kwargs={"litellm_call_id": data["litellm_call_id"]},+ response_obj=None,+ start_time=None,+ end_time=None,+ )+ assert handler._gauge_in_flight_from_cache_value(+ await local_cache.async_get_cache(key=parallel_key)+ ) == 0+++@pytest.mark.asyncio+async def test_concurrent_success_callbacks_release_parallel_slot_once_when_redis_fails() -> None:+ from unittest.mock import AsyncMock++ api_key = hash_token("sk-concurrent-release")+ local_cache = DualCache()+ handler = _PROXY_MaxParallelRequestsHandler(+ internal_usage_cache=InternalUsageCache(local_cache)+ )+ user_api_key_dict = UserAPIKeyAuth(api_key=api_key, max_parallel_requests=2)+ call_id = "concurrent-release-owner"+ parallel_key = f"{{api_key:{api_key}}}:max_parallel_requests"+ release_started = asyncio.Event()+ allow_redis_failure = asyncio.Event()++ async def failing_release(+ keys: Sequence[str], args: Sequence[object]+ ) -> list[int]:+ release_started.set()+ await allow_redis_failure.wait()+ raise ConnectionError("redis unavailable")++ release_script = AsyncMock(side_effect=failing_release)+ handler.parallel_release_script = release_script+ await local_cache.async_set_cache(key=parallel_key, value=2, local_only=True)+ stash = get_or_create_request_stash()+ stash.owner_litellm_call_id = call_id+ stash.parallel_slot = ParallelSlotAcquisition(+ slot_id="slot-concurrent-release",+ counter_keys=[parallel_key],+ )+ data = {"litellm_call_id": call_id}++ post_call_task = asyncio.create_task(+ handler.async_post_call_success_hook(+ data=data,+ user_api_key_dict=user_api_key_dict,+ response=ResponsesAPIResponse(+ id="resp_concurrent_release",+ created_at=0,+ model="gpt-4o-mini",+ object="response",+ output=[],+ status="completed",+ ),+ )+ )+ await asyncio.wait_for(release_started.wait(), timeout=5)+ logging_task = asyncio.create_task(+ handler.async_log_success_event(+ kwargs=data,+ response_obj=None,+ start_time=None,+ end_time=None,+ )+ )+ allow_redis_failure.set()+ await asyncio.wait_for(+ asyncio.gather(post_call_task, logging_task),+ timeout=5,+ )++ assert release_script.await_count == 1+ assert await local_cache.async_get_cache(key=parallel_key) == 1+ assert stash.parallel_slot is None++ async def _build_seeded_limiter(): """Build a v3 limiter whose api-key slot registry already holds the pre-call slot.""" api_key = hash_token("sk-disconnect")