Backtestslitellmrun-20261111-220245#49103
[Bug]: Router retry reuses success logging state and drops later attempt cost
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
The issue names the root cause explicitly: a shared Logging object carries has_logged_async_success/has_logged_sync_success across router retries, so the later successful attempt's success log is skipped. Patch A only handles the proxy's deferred ("parked") success result: in carry_failed_attempt_usage it moves a parked deferred_success_result into failed_attempts_usage so the serving attempt bills it. That fixes the proxy reproduction (both proxy tests pass) but leaves the non-deferred case untouched: when the first attempt's success log has already been emitted or enqueued (has_logged_async_success set), the retry's success log is still suppressed. The independent test that exercises exactly that quoted root cause (Router.acompletion with a shared logging object, first attempt already logged) still fails on A with recorded_costs=[0.147], identical to base. Patch B addresses both paths: it detects a superseded attempt that already logged or enqueued its success (async enqueued, sync scheduled, or has_logged_* set), clears the one-success-per-request markers and stale cost/usage state so the retry logs and prices its own response, and uses pending counters so the in-flight first-attempt log's completion marker does not re-suppress the retry; it separately handles the parked deferred result by carrying it as a served round. It also marks sync success logs as scheduled in utils.client so the sync executor path is covered. All three independent tests pass on B. B's tests are more substantive too: proxy-level chat, messages-refusal and retry scenarios plus router-level shared-logging tests, including a sync in-flight log race, versus A's single unit test that only drives the parked path through manual calls. B is larger and more intricate, which carries some risk, but the added state is targeted and documented, whereas A leaves the issue's stated root cause unfixed. B wins on correctness and tests.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
1h 4m
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
508vs 84 merged
Changed lines, ours vs merged
The issue
View on GitHubWhat happened?
When a retryable error is raised after a non-streaming provider response has already completed success processing, the router retries with the same Logging object. The first attempt has already set has_logged_async_success/has_logged_sync_success, so success logging for the later successful attempt is skipped.
This causes the final response usage and reported cost to describe different attempts. In the reproduction, the response body contains the final attempt's cache-read usage, while both the accounting callback and x-litellm-response-cost retain only the discarded first attempt's cache-write cost.
For example, a provider can return a billable response marked as a content-policy refusal. LiteLLM maps that provider refusal to ContentPolicyViolationError and retries. If the retry succeeds, its cost is then skipped because the first attempt already marked success logging as complete.
Reproduction
The test below uses only stock LiteLLM APIs and a local fake Bedrock endpoint. The first response is fully priced, then the test raises ContentPolicyViolationError; the configured router retry succeeds with different usage.
Save as test_retry_cost_accounting_upstream_repro.py, then run:
pip install 'litellm[proxy]==1.104.5' pytest pytest-asyncio
pytest -q test_retry_cost_accounting_upstream_repro.py
<details>
<summary>Self-contained failing test</summary>
"""Portable upstream reproduction for post-cost router retry accounting.
This file intentionally uses only public LiteLLM package APIs and a local fake
Bedrock endpoint. It is expected to fail on unpatched LiteLLM v1.104.5.
"""
from __future__ import annotations
import asyncio
import copy
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from datetime import datetime
from typing import Any
import litellm
import pytest
from aiohttp import web
from fastapi.testclient import TestClient
from litellm import Router
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy import proxy_server
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
MODEL_GROUP = "retry-cost-repro"
BEDROCK_MODEL = "bedrock/global.anthropic.retry-cost-repro"
CACHE_TOKENS = 100
FRESH_TOKENS = 2
FIRST_OUTPUT_TOKENS = 10
FINAL_OUTPUT_TOKENS = 12
INPUT_RATE = 0.001
CACHE_READ_RATE = 0.0001
CACHE_WRITE_RATE = 0.00125
OUTPUT_RATE = 0.002
@pytest.fixture(autouse=True)
def _register_model() -> None:
pricing = {
"litellm_provider": "bedrock",
"mode": "chat",
"input_cost_per_token": INPUT_RATE,
"cache_read_input_token_cost": CACHE_READ_RATE,
"cache_creation_input_token_cost": CACHE_WRITE_RATE,
"output_cost_per_token": OUTPUT_RATE,
}
litellm.register_model({BEDROCK_MODEL: pricing})
def _provider_response(attempt: int) -> dict[str, Any]:
first_attempt = attempt == 1
return {
"id": f"msg_attempt_{attempt}",
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": f"attempt {attempt}"}],
"model": BEDROCK_MODEL,
"stop_reason": "max_tokens",
"stop_sequence": None,
"usage": {
"input_tokens": FRESH_TOKENS,
"output_tokens": FIRST_OUTPUT_TOKENS if first_attempt else FINAL_OUTPUT_TOKENS,
"cache_creation_input_tokens": CACHE_TOKENS if first_attempt else 0,
"cache_read_input_tokens": 0 if first_attempt else CACHE_TOKENS,
},
}
class _Upstream:
def __init__(self) -> None:
self.hits = 0
async def handle(self, request: web.Request) -> web.Response:
await request.read()
self.hits += 1
return web.json_response(_provider_response(self.hits))
class _AccountingCapture(CustomLogger):
def __init__(self) -> None:
self.records: list[dict[str, Any]] = []
self.first_record = asyncio.Event()
async def async_log_success_event(
self,
kwargs: dict[str, Any],
response_obj: object,
start_time: datetime,
end_time: datetime,
) -> None:
self.records.append(
{
"response_cost": kwargs.get("response_cost"),
"response": copy.deepcopy(response_obj),
}
)
self.first_record.set()
@asynccontextmanager
async def _router(upstream: _Upstream) -> AsyncIterator[Router]:
app = web.Application()
app.router.add_post("/{path:.*}", upstream.handle)
runner = web.AppRunner(app)
await runner.setup()
site = web.TCPSite(runner, "127.0.0.1", 0)
await site.start()
port = site._server.sockets[0].getsockname()[1]
try:
yield Router(
model_list=[
{
"model_name": MODEL_GROUP,
"litellm_params": {
"model": BEDROCK_MODEL,
"aws_bedrock_runtime_endpoint": f"http://127.0.0.1:{port}",
"aws_access_key_id": "test-access-key",
"aws_secret_access_key": "test-secret-key",
"aws_region_name": "us-east-1",
"max_retries": 0,
},
"model_info": {
"input_cost_per_token": INPUT_RATE,
"cache_read_input_token_cost": CACHE_READ_RATE,
"cache_creation_input_token_cost": CACHE_WRITE_RATE,
"output_cost_per_token": OUTPUT_RATE,
},
}
],
num_retries=1,
retry_policy={"ContentPolicyViolationErrorRetries": 1},
)
finally:
await runner.cleanup()
@pytest.mark.anyio
async def test_post_cost_retry_accounts_for_both_priced_attempts(
monkeypatch: pytest.MonkeyPatch,
) -> None:
upstream = _Upstream()
capture = _AccountingCapture()
monkeypatch.setenv(
"LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY",
"true",
)
monkeypatch.setattr(litellm, "callbacks", [])
monkeypatch.setattr(litellm, "success_callback", [])
monkeypatch.setattr(litellm, "_async_success_callback", [capture])
async with _router(upstream) as router:
original_make_call = router.make_call
attempt = 0
async def reject_first_costed_response(*args: Any, **kwargs: Any) -> Any:
nonlocal attempt
attempt += 1
response = await original_make_call(*args, **kwargs)
if attempt == 1:
await asyncio.wait_for(capture.first_record.wait(), timeout=2)
raise litellm.ContentPolicyViolationError(
message="simulated safeguard refusal after response processing",
model=BEDROCK_MODEL,
llm_provider="bedrock",
)
return response
monkeypatch.setattr(router, "make_call", reject_first_costed_response)
with TestClient(proxy_server.app) as client:
monkeypatch.setattr(proxy_server, "llm_router", router)
monkeypatch.setattr(proxy_server, "llm_model_list", router.model_list)
monkeypatch.setattr(proxy_server, "general_settings", {})
monkeypatch.setattr(proxy_server, "prisma_client", None)
proxy_server.app.dependency_overrides[user_api_key_auth] = lambda: UserAPIKeyAuth(
api_key="hashed-key",
user_email="user@example.com",
user_role="internal_user",
)
try:
http_response = await asyncio.to_thread(
client.post,
"/v1/messages",
headers={"user-agent": "upstream-retry-cost-repro"},
json={
"model": MODEL_GROUP,
"messages": [{"role": "user", "content": "cached prefix"}],
"max_tokens": FINAL_OUTPUT_TOKENS,
},
)
finally:
proxy_server.app.dependency_overrides.pop(user_api_key_auth, None)
await asyncio.sleep(0.2)
assert http_response.status_code == 200, http_response.text
assert upstream.hits == 2
assert http_response.headers["x-litellm-attempted-retries"] == "1"
assert http_response.json()["usage"]["cache_read_input_tokens"] == CACHE_TOKENS
first_cost = (
FRESH_TOKENS * INPUT_RATE
+ CACHE_TOKENS * CACHE_WRITE_RATE
+ FIRST_OUTPUT_TOKENS * OUTPUT_RATE
)
final_cost = (
FRESH_TOKENS * INPUT_RATE
+ CACHE_TOKENS * CACHE_READ_RATE
+ FINAL_OUTPUT_TOKENS * OUTPUT_RATE
)
recorded_costs = [record["response_cost"] for record in capture.records]
header_cost = float(http_response.headers["x-litellm-response-cost"])
assert recorded_costs == pytest.approx(
[first_cost, final_cost]
), f"recorded_costs={recorded_costs}, header_cost={header_cost}"
assert header_cost == pytest.approx(first_cost + final_cost)
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
</details>
Observed on litellm[proxy]==1.104.5:
AssertionError: recorded_costs=[0.147], header_cost=0.147
Expected records: [0.147, 0.036]
Expected measurable request cost: 0.183
The request itself succeeds with HTTP 200, reports one attempted retry, and its response body contains the final attempt's cache-read usage.
Root cause
Router.async_function_with_retries carries litellm_logging_obj into later attempts. In litellm/litellm_core_utils/litellm_logging.py, Logging.should_run_logging() returns false once the shared object contains has_logged_<success event> = True. As a result, the later attempt does not run _process_hidden_params_and_response_cost and does not emit its accounting event.
The proxy subsequently reads stale first-attempt state when constructing x-litellm-response-cost.
Expected behavior
Every provider response with billable usage should be accounted exactly once, including a later successful attempt after a post-response retry. The request-level cost must not silently omit later attempts. Please confirm whether x-litellm-response-cost is intended to aggregate all billable attempts; the missing final-attempt accounting record is a defect independently of that public header contract.
Affected area
- Router retries after post-response errors
- Non-streaming success logging/accounting
- Reproduced with Anthropic Messages semantics through Bedrock
Related but distinct from #26475, #37105, #40736, #19216, and #42663.
The two patches
litellm/litellm_core_utils/litellm_logging.pyindex fcc521cf6c..b8b436f4ac 100644@@ -928,6 +928,10 @@ class Logging(LiteLLMLoggingBaseClass): self._async_success_scheduled: bool = False self._async_success_enqueued: bool = False self._sync_success_scheduled: bool = False+ # Success logs of attempts a retry replaced that were scheduled but had not logged yet: each one's+ # completion marker is theirs, not the retry's, so it must not drop the retry's success log.+ self._superseded_async_success_pending: int = 0+ self._superseded_sync_success_pending: int = 0 self.on_detached_stream_failure: Callable[[Exception], Awaitable[None]] | None = None self._stream_failure_claimed: bool = False self._stash_owed_on_success: bool = False@@ -2761,6 +2765,10 @@ class Logging(LiteLLMLoggingBaseClass): def carry_failed_attempt_usage(self) -> None: """Move what a failed earlier attempt stashed off the retry sharing this logging object, so the retry logs its own usage while the request's failure log still bills every attempt."""+ if self._superseded_attempt_logged_success():+ self._forget_superseded_success()+ return+ self._stash_parked_success_for_retry() usage: Final = self._model_call_details_view.get("combined_usage_object") response_cost: Final = self._model_call_details_view.get("response_cost") billed_by_continuation: Final = self._stash_billed_by_continuation@@ -2783,6 +2791,47 @@ class Logging(LiteLLMLoggingBaseClass): return self._failed_attempts_owed_on_success = (*self._failed_attempts_owed_on_success, attempt) + def _superseded_attempt_logged_success(self) -> bool:+ """The attempt a retry replaces already emitted (or enqueued) its own non-streaming success log, a provider+ response an error raised after it (a blocked response, a refusal) turned into a retry."""+ return (+ self._async_success_enqueued+ or self._sync_success_scheduled+ or self.model_call_details.get("has_logged_async_success") is True+ or self.model_call_details.get("has_logged_sync_success") is True+ )++ def _forget_superseded_success(self) -> None:+ """The replaced attempt's success log bills it, so none of its usage is carried; clearing the one-success-a-+ request markers lets the retry that replaces it log, and price, its own response."""+ self.model_call_details.pop("_accounting_pricing_snapshot", None)+ self.model_call_details.pop("_accounting_usage_observed", None)+ self.model_call_details.pop("combined_usage_object", None)+ self.model_call_details.pop("response_cost", None)+ self._stash_billed_by_continuation = False+ self._stash_owed_on_success = False+ if self._async_success_enqueued and self.model_call_details.get("has_logged_async_success") is not True:+ self._superseded_async_success_pending += 1+ if self._sync_success_scheduled and self.model_call_details.get("has_logged_sync_success") is not True:+ self._superseded_sync_success_pending += 1+ self._async_success_scheduled = False+ self._async_success_enqueued = False+ self._reset_success_emission_dedupe()++ def _stash_parked_success_for_retry(self) -> None:+ """The replaced attempt's success log is parked until the proxy's post-call hooks run, which would log it in+ place of the retry's response. Its response was served and billed by the provider, so it is carried like a+ failed attempt the request's success (or failure) log bills, and the retry claims the success log."""+ parked: Final = self.deferred_success_result+ if parked is None or not self._async_success_scheduled:+ return+ self.deferred_success_result = None+ self.enqueue_deferred_logging = None+ self._async_success_scheduled = False+ served: Final = self.served_round(parked[0])+ if served is not None:+ self.record_served_rounds_for_failure((served,))+ def mark_stash_billed_by_continuation(self) -> None: self._stash_billed_by_continuation = True @@ -2915,6 +2964,11 @@ class Logging(LiteLLMLoggingBaseClass): def mark_async_success_log_enqueued(self) -> None: self._async_success_enqueued = True + def mark_sync_success_log_scheduled(self) -> None:+ """The sync success log was handed to a background thread, so a router retry sharing this logging object+ treats it as this attempt's log even before it marks itself logged."""+ self._sync_success_scheduled = True+ def claim_async_success_log(self) -> bool: """One async success log per request. The innermost @client wrapper always exits first, since the outer one is awaiting it, so it claims the log here and the outer wrapper gets@@ -2933,6 +2987,12 @@ class Logging(LiteLLMLoggingBaseClass): Ignore check on stream, as there can be multiple chunks """ return+ if event_type == "async_success" and self._superseded_async_success_pending > 0:+ self._superseded_async_success_pending -= 1+ return+ if event_type == "sync_success" and self._superseded_sync_success_pending > 0:+ self._superseded_sync_success_pending -= 1+ return self.model_call_details[f"has_logged_{event_type}"] = True if event_type == "async_failure": self.model_call_details["async_failure_logged_attempt"] = self.model_call_details.get(litellm/utils.pyindex 66b5b274f9..b0280c886d 100644@@ -2284,6 +2284,8 @@ def client(original_function): # This is essential for OpenTelemetry span context propagation ctx: Final = contextvars.copy_context() executor: Final[BoundedLoggingThreadPoolExecutor] = litellm_utils.executor+ if not logging_obj.stream:+ logging_obj.mark_sync_success_log_scheduled() executor.submit( ctx.run, logging_obj.success_handler,tests/unit/proxy/test_superseded_attempt_accounting.pynew file mode 100644index 0000000000..53597eda0e@@ -0,0 +1,236 @@+"""+A provider response the router discards for a retry or a fallback (a content-filtered chat choice, a+safeguard refusal, an error raised after the response was served) was billed by the provider. The+proxy parks the request's success log until its post-call hooks run, so the discarded attempt must+not take the success log from the response the client receives, nor drop out of its cost.+"""++from __future__ import annotations++import asyncio+from collections.abc import Callable, Iterator, Mapping+from typing import Final++import httpx+import pytest+import respx+from fastapi.testclient import TestClient++import litellm+from litellm import Router+from litellm.integrations.custom_logger import CustomLogger+from litellm.proxy import proxy_server+from litellm.proxy._types import UserAPIKeyAuth+from litellm.proxy.auth.user_api_key_auth import user_api_key_auth++INPUT_RATE: Final = 0.001+OUTPUT_RATE: Final = 0.002+PROMPT_TOKENS: Final = 10+FIRST_COMPLETION_TOKENS: Final = 10+FINAL_COMPLETION_TOKENS: Final = 20+FIRST_COST: Final = PROMPT_TOKENS * INPUT_RATE + FIRST_COMPLETION_TOKENS * OUTPUT_RATE+FINAL_COST: Final = PROMPT_TOKENS * INPUT_RATE + FINAL_COMPLETION_TOKENS * OUTPUT_RATE+PRICING: Final = {"input_cost_per_token": INPUT_RATE, "output_cost_per_token": OUTPUT_RATE}+++class _SuccessCosts(CustomLogger):+ def __init__(self) -> None:+ self.costs: tuple[object, ...] = ()++ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time) -> None: # noqa: ANN001 # CustomLogger's untyped hook signature+ self.costs = (*self.costs, kwargs.get("response_cost"))+++def _chat_body(attempt: int, finish_reason: str) -> dict[str, object]: # mutable-ok: a JSON body+ completion_tokens: Final = FIRST_COMPLETION_TOKENS if attempt == 1 else FINAL_COMPLETION_TOKENS+ return {+ "id": f"chatcmpl-attempt-{attempt}",+ "object": "chat.completion",+ "created": 1,+ "model": "gpt-4o",+ "choices": [+ {+ "index": 0,+ "message": {"role": "assistant", "content": f"attempt {attempt}"},+ "finish_reason": finish_reason,+ }+ ],+ "usage": {+ "prompt_tokens": PROMPT_TOKENS,+ "completion_tokens": completion_tokens,+ "total_tokens": PROMPT_TOKENS + completion_tokens,+ },+ }+++def _messages_body(attempt: int) -> dict[str, object]: # mutable-ok: a JSON body+ refused: Final = attempt == 1+ return {+ "id": f"msg_attempt_{attempt}",+ "type": "message",+ "role": "assistant",+ "model": "claude-test",+ "content": [{"type": "text", "text": f"attempt {attempt}"}],+ "stop_reason": "refusal" if refused else "end_turn",+ **({"stop_details": {"type": "refusal", "category": None, "explanation": "declined"}} if refused else {}),+ "stop_sequence": None,+ "usage": {+ "input_tokens": PROMPT_TOKENS,+ "output_tokens": FIRST_COMPLETION_TOKENS if refused else FINAL_COMPLETION_TOKENS,+ },+ }+++def _deployment(name: str, model: str) -> dict[str, object]: # mutable-ok: a model_list entry+ return {"model_name": name, "litellm_params": {"model": model, "api_key": "test-key"}, "model_info": PRICING}+++@pytest.fixture+def success_costs(monkeypatch: pytest.MonkeyPatch) -> _SuccessCosts:+ capture: Final = _SuccessCosts()+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)+ monkeypatch.setenv("LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY", "true")+ monkeypatch.setattr(litellm, "callbacks", [])+ monkeypatch.setattr(litellm, "success_callback", [])+ monkeypatch.setattr(litellm, "_async_success_callback", [capture])+ return capture+++@pytest.fixture+def proxy_post(+ monkeypatch: pytest.MonkeyPatch,+) -> Iterator[Callable[[Router, str, Mapping[str, object]], httpx.Response]]:+ def post(router: Router, route: str, body: Mapping[str, object]) -> httpx.Response:+ with TestClient(proxy_server.app) as client:+ monkeypatch.setattr(proxy_server, "llm_router", router)+ monkeypatch.setattr(proxy_server, "general_settings", {})+ monkeypatch.setattr(proxy_server, "prisma_client", None)+ proxy_server.app.dependency_overrides[user_api_key_auth] = lambda: UserAPIKeyAuth(api_key="hashed-key")+ return client.post(route, json=body)++ yield post+ proxy_server.app.dependency_overrides.pop(user_api_key_auth, None)+++async def _settled(capture: _SuccessCosts) -> None:+ for _ in range(200):+ if capture.costs:+ return+ await asyncio.sleep(0.01)+++@pytest.mark.asyncio+async def test_a_content_policy_fallback_bills_the_filtered_response_with_the_served_one(+ respx_mock: respx.MockRouter,+ success_costs: _SuccessCosts,+ proxy_post: Callable[[Router, str, Mapping[str, object]], httpx.Response],+) -> None:+ hits: Final = [0]++ def upstream(_: httpx.Request) -> httpx.Response:+ hits[0] += 1+ return httpx.Response(200, json=_chat_body(hits[0], "content_filter" if hits[0] == 1 else "stop"))++ respx_mock.post("https://api.openai.com/v1/chat/completions").mock(side_effect=upstream)+ router: Final = Router(+ model_list=[_deployment("primary", "openai/gpt-4o"), _deployment("backup", "openai/gpt-4o")],+ content_policy_fallbacks=[{"primary": ["backup"]}],+ )++ response: Final = await asyncio.to_thread(+ proxy_post,+ router,+ "/v1/chat/completions",+ {"model": "primary", "messages": [{"role": "user", "content": "hi"}]},+ )+ await _settled(success_costs)++ assert response.status_code == 200, response.text+ assert hits[0] == 2+ assert response.json()["choices"][0]["message"]["content"] == "attempt 2"+ assert success_costs.costs == (pytest.approx(FIRST_COST + FINAL_COST),)+ assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(FIRST_COST + FINAL_COST)+++@pytest.mark.asyncio+async def test_a_retry_after_a_served_response_bills_both_attempts(+ respx_mock: respx.MockRouter,+ success_costs: _SuccessCosts,+ proxy_post: Callable[[Router, str, Mapping[str, object]], httpx.Response],+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ hits: Final = [0]++ def upstream(_: httpx.Request) -> httpx.Response:+ hits[0] += 1+ return httpx.Response(200, json=_chat_body(hits[0], "stop"))++ respx_mock.post("https://api.openai.com/v1/chat/completions").mock(side_effect=upstream)+ real_acompletion: Final = litellm.acompletion++ async def refused_after_the_first_response(*args: object, **kwargs: object) -> object: # kwargs-ok: the attempt+ # The error a provider-side safeguard raises once the first response was served and priced.+ response: Final = await real_acompletion(*args, **kwargs)+ if hits[0] == 1:+ raise litellm.ContentPolicyViolationError(+ message="refused after the response was served", model="gpt-4o", llm_provider="openai"+ )+ return response++ monkeypatch.setattr(litellm, "acompletion", refused_after_the_first_response)+ router: Final = Router(+ model_list=[_deployment("primary", "openai/gpt-4o")],+ num_retries=1,+ retry_policy={"ContentPolicyViolationErrorRetries": 1},+ )++ response: Final = await asyncio.to_thread(+ proxy_post,+ router,+ "/v1/chat/completions",+ {"model": "primary", "messages": [{"role": "user", "content": "hi"}]},+ )+ await _settled(success_costs)++ assert response.status_code == 200, response.text+ assert hits[0] == 2+ assert response.headers["x-litellm-attempted-retries"] == "1"+ assert response.json()["usage"]["completion_tokens"] == FINAL_COMPLETION_TOKENS+ assert success_costs.costs == (pytest.approx(FIRST_COST + FINAL_COST),)+ assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(FIRST_COST + FINAL_COST)+++@pytest.mark.asyncio+async def test_a_messages_refusal_fallback_bills_the_refused_response_with_the_served_one(+ respx_mock: respx.MockRouter,+ success_costs: _SuccessCosts,+ proxy_post: Callable[[Router, str, Mapping[str, object]], httpx.Response],+) -> None:+ hits: Final = [0]++ def upstream(_: httpx.Request) -> httpx.Response:+ hits[0] += 1+ return httpx.Response(200, json=_messages_body(hits[0]))++ respx_mock.post("https://api.anthropic.com/v1/messages").mock(side_effect=upstream)+ router: Final = Router(+ model_list=[+ _deployment("primary", "anthropic/claude-test"),+ _deployment("backup", "anthropic/claude-test"),+ ],+ content_policy_fallbacks=[{"primary": ["backup"]}],+ )++ response: Final = await asyncio.to_thread(+ proxy_post,+ router,+ "/v1/messages",+ {"model": "primary", "max_tokens": 32, "messages": [{"role": "user", "content": "hi"}]},+ )+ await _settled(success_costs)++ assert response.status_code == 200, response.text+ assert hits[0] == 2+ assert response.json()["usage"]["output_tokens"] == FINAL_COMPLETION_TOKENS+ assert success_costs.costs == (pytest.approx(FIRST_COST + FINAL_COST),)+ assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(FIRST_COST + FINAL_COST)tests/unit/test_router/test_router_superseded_attempt_logging.pynew file mode 100644index 0000000000..9cc4a7ff15@@ -0,0 +1,210 @@+"""+A router call that shares one logging object across its attempts (``litellm_logging_obj``) and does not+defer its success log: an attempt the router discards after its success log was scheduled is billed by+that log, and the attempt that replaces it still logs and prices its own response.+"""++from __future__ import annotations++import asyncio+import datetime+import time+from typing import Final++import httpx+import pytest+import respx++import litellm+from litellm import Router+from litellm.integrations.custom_logger import CustomLogger+from litellm.litellm_core_utils.litellm_logging import Logging++INPUT_RATE: Final = 0.001+OUTPUT_RATE: Final = 0.002+PROMPT_TOKENS: Final = 10+MESSAGES: Final = [{"role": "user", "content": "hi"}]+PRICING: Final = {"input_cost_per_token": INPUT_RATE, "output_cost_per_token": OUTPUT_RATE}+++def _cost(attempt: int) -> float:+ return PROMPT_TOKENS * INPUT_RATE + 10 * attempt * OUTPUT_RATE+++class _SuccessCosts(CustomLogger):+ def __init__(self) -> None:+ self.costs: tuple[object, ...] = ()++ async def async_log_success_event(self, kwargs, response_obj, start_time, end_time) -> None: # noqa: ANN001 # CustomLogger's untyped hook signature+ self.costs = (*self.costs, kwargs.get("response_cost"))+++def _chat_body(attempt: int, finish_reason: str) -> dict[str, object]: # mutable-ok: a JSON body+ return {+ "id": f"chatcmpl-attempt-{attempt}",+ "object": "chat.completion",+ "created": 1,+ "model": "gpt-4o",+ "choices": [+ {+ "index": 0,+ "message": {"role": "assistant", "content": f"attempt {attempt}"},+ "finish_reason": finish_reason,+ }+ ],+ "usage": {+ "prompt_tokens": PROMPT_TOKENS,+ "completion_tokens": 10 * attempt,+ "total_tokens": PROMPT_TOKENS + 10 * attempt,+ },+ }+++def _deployment(name: str) -> dict[str, object]: # mutable-ok: a model_list entry+ return {+ "model_name": name,+ "litellm_params": {"model": "openai/gpt-4o", "api_key": "test-key"},+ "model_info": PRICING,+ }+++def _shared_logging(call_id: str) -> Logging:+ logging_obj, _ = litellm.utils.function_setup(+ original_function="acompletion",+ rules_obj=litellm.utils.Rules(),+ start_time=datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc),+ model="primary",+ messages=MESSAGES,+ litellm_call_id=call_id,+ )+ assert isinstance(logging_obj, Logging)+ return logging_obj+++@pytest.fixture+def success_costs(monkeypatch: pytest.MonkeyPatch) -> _SuccessCosts:+ capture: Final = _SuccessCosts()+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)+ monkeypatch.setattr(litellm, "callbacks", [capture])+ return capture+++async def _logged(capture: _SuccessCosts, count: int) -> None:+ for _ in range(200):+ if len(capture.costs) >= count:+ return+ await asyncio.sleep(0.01)+++@pytest.mark.asyncio+async def test_a_content_policy_fallback_logs_the_served_response_after_the_filtered_one(+ respx_mock: respx.MockRouter, success_costs: _SuccessCosts+) -> None:+ hits: Final = [0]++ def upstream(_: httpx.Request) -> httpx.Response:+ hits[0] += 1+ return httpx.Response(200, json=_chat_body(hits[0], "content_filter" if hits[0] == 1 else "stop"))++ respx_mock.post("https://api.openai.com/v1/chat/completions").mock(side_effect=upstream)+ router: Final = Router(+ model_list=[_deployment("primary"), _deployment("backup")], content_policy_fallbacks=[{"primary": ["backup"]}]+ )++ response: Final = await router.acompletion(+ model="primary", messages=MESSAGES, litellm_logging_obj=_shared_logging("superseded-fallback")+ )+ await _logged(success_costs, 2)++ assert hits[0] == 2+ assert response.choices[0].message.content == "attempt 2"+ assert success_costs.costs == (pytest.approx(_cost(1)), pytest.approx(_cost(2)))+ assert response._hidden_params["response_cost"] == pytest.approx(_cost(2))+++@pytest.mark.asyncio+async def test_a_retry_after_a_logged_response_logs_the_retry_too(+ respx_mock: respx.MockRouter, success_costs: _SuccessCosts, monkeypatch: pytest.MonkeyPatch+) -> None:+ hits: Final = [0]++ def upstream(_: httpx.Request) -> httpx.Response:+ hits[0] += 1+ return httpx.Response(200, json=_chat_body(hits[0], "stop"))++ respx_mock.post("https://api.openai.com/v1/chat/completions").mock(side_effect=upstream)+ real_acompletion: Final = litellm.acompletion++ async def refused_after_the_first_response(*args: object, **kwargs: object) -> object: # kwargs-ok: the attempt+ # The error a provider-side safeguard raises once the first response was served and logged.+ response: Final = await real_acompletion(*args, **kwargs)+ if hits[0] == 1:+ await _logged(success_costs, 1)+ raise litellm.ContentPolicyViolationError(+ message="refused after the response was served", model="gpt-4o", llm_provider="openai"+ )+ return response++ monkeypatch.setattr(litellm, "acompletion", refused_after_the_first_response)+ router: Final = Router(+ model_list=[_deployment("primary")], num_retries=1, retry_policy={"ContentPolicyViolationErrorRetries": 1}+ )++ response: Final = await router.acompletion(+ model="primary", messages=MESSAGES, litellm_logging_obj=_shared_logging("superseded-retry")+ )+ await _logged(success_costs, 2)++ assert hits[0] == 2+ assert response.usage.completion_tokens == 20+ assert success_costs.costs == (pytest.approx(_cost(1)), pytest.approx(_cost(2)))+++class _SyncSuccessLogs(CustomLogger):+ def __init__(self) -> None:+ self.hooked = 0+ self.logged: tuple[str, ...] = ()++ def logging_hook(self, kwargs, result, call_type): # noqa: ANN001, ANN201 # CustomLogger's untyped hook signature+ self.hooked += 1+ if self.hooked == 1:+ # The discarded attempt's background log has not marked itself logged when the router falls back,+ # and does before the fallback's response arrives.+ time.sleep(0.1)+ return kwargs, result++ def log_success_event(self, kwargs, response_obj, start_time, end_time) -> None: # noqa: ANN001 # CustomLogger's untyped hook signature+ self.logged = (*self.logged, response_obj.choices[0].message.content)+++def test_a_sync_content_policy_fallback_logs_both_served_responses_while_the_first_log_is_in_flight(+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch+) -> None:+ capture: Final = _SyncSuccessLogs()+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)+ monkeypatch.setattr(litellm, "callbacks", [capture])+ hits: Final = [0]++ def upstream(_: httpx.Request) -> httpx.Response:+ hits[0] += 1+ if hits[0] == 2:+ time.sleep(0.4)+ return httpx.Response(200, json=_chat_body(hits[0], "content_filter" if hits[0] == 1 else "stop"))++ respx_mock.post("https://api.openai.com/v1/chat/completions").mock(side_effect=upstream)+ router: Final = Router(+ model_list=[_deployment("primary"), _deployment("backup")], content_policy_fallbacks=[{"primary": ["backup"]}]+ )++ response: Final = router.completion(+ model="primary", messages=MESSAGES, litellm_logging_obj=_shared_logging("superseded-sync-fallback")+ )+ for _ in range(200):+ if len(capture.logged) >= 2:+ break+ time.sleep(0.01)+ time.sleep(0.1)++ assert hits[0] == 2+ assert response.choices[0].message.content == "attempt 2"+ assert sorted(capture.logged) == ["attempt 1", "attempt 2"]
litellm/litellm_core_utils/litellm_logging.pyindex fcc521cf6cf..e9342a765bc 100644@@ -2761,6 +2761,8 @@ class Logging(LiteLLMLoggingBaseClass): def carry_failed_attempt_usage(self) -> None: """Move what a failed earlier attempt stashed off the retry sharing this logging object, so the retry logs its own usage while the request's failure log still bills every attempt."""+ if self._carry_parked_success():+ return usage: Final = self._model_call_details_view.get("combined_usage_object") response_cost: Final = self._model_call_details_view.get("response_cost") billed_by_continuation: Final = self._stash_billed_by_continuation@@ -2783,6 +2785,25 @@ class Logging(LiteLLMLoggingBaseClass): return self._failed_attempts_owed_on_success = (*self._failed_attempts_owed_on_success, attempt) + def _carry_parked_success(self) -> bool:+ """A billed response the router discards after its success log was parked (a safeguard refusal it falls+ back from) is owed by whichever attempt ends the request, instead of being logged in its place."""+ parked: Final = self.deferred_success_result+ if parked is None or self.enqueue_deferred_logging is None:+ return False+ self.deferred_success_result = None+ self.enqueue_deferred_logging = None+ self._async_success_scheduled = False+ served: Final = self.served_round(parked[0])+ if served is None:+ return True+ self.model_call_details["failed_attempts_usage"] = (+ *self.model_call_details.get("failed_attempts_usage", ()),+ served,+ )+ self._failed_attempts_owed_on_success = (*self._failed_attempts_owed_on_success, served)+ return True+ def mark_stash_billed_by_continuation(self) -> None: self._stash_billed_by_continuation = True tests/unit/litellm_core_utils/test_litellm_logging.pyindex 58fd03e0cbc..2f5edbd6350 100644@@ -14690,3 +14690,66 @@ def test_standard_logging_payload_reports_the_per_call_prices_the_request_was_bi model_map_value: Final = payload["model_map_information"]["model_map_value"] assert model_map_value is not None assert (model_map_value["input_cost_per_token"], model_map_value["output_cost_per_token"]) == (3e-06, 7e-06)+++@pytest.mark.asyncio+async def test_a_response_discarded_after_its_success_log_was_parked_is_billed_once_on_the_serving_attempt(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ from litellm.litellm_core_utils.logging_worker import GLOBAL_LOGGING_WORKER+ from litellm.responses.streaming_iterator import drain_logging_tasks+ from litellm.utils import _schedule_async_success_logging++ class Recorder(CustomLogger):+ def __init__(self) -> None:+ super().__init__()+ self.events: tuple[tuple[dict[str, object], object], ...] = ()++ async def async_log_success_event(+ self, kwargs: dict[str, object], response_obj: object, start_time: object, end_time: object+ ) -> None:+ self.events = (*self.events, (dict(kwargs), response_obj))++ recorder: Final = Recorder()+ monkeypatch.setattr(litellm, "_async_success_callback", [])+ monkeypatch.setattr(litellm, "success_callback", [])+ logging_obj: Final = LitellmLogging(+ model="gpt-4o-mini",+ messages=[{"role": "user", "content": "hi"}],+ stream=False,+ call_type="acompletion",+ start_time=time.time(),+ litellm_call_id="refused-then-served",+ function_id="refused-then-served",+ dynamic_async_success_callbacks=[recorder],+ )+ logging_obj.update_environment_variables(+ model="gpt-4o-mini", user="", optional_params={}, litellm_params={"metadata": {}}, custom_llm_provider="openai"+ )+ logging_obj.defer_async_logging = True+ refused: Final = ModelResponse(+ id="refused", model="gpt-4o-mini", usage=Usage(prompt_tokens=1000, completion_tokens=500, total_tokens=1500)+ )+ served: Final = ModelResponse(+ id="served", model="gpt-4o-mini", usage=Usage(prompt_tokens=10, completion_tokens=5, total_tokens=15)+ )+ now: Final = datetime.datetime.now()++ _schedule_async_success_logging(logging_obj, refused, now, now, is_completion_with_fallbacks=False)+ logging_obj.carry_failed_attempt_usage()+ _schedule_async_success_logging(logging_obj, served, now, now, is_completion_with_fallbacks=False)+ prepared: Final = logging_obj.prepare_response_cost(served, now)+ enqueue: Final = logging_obj.enqueue_deferred_logging+ assert enqueue is not None+ enqueue()+ await drain_logging_tasks()+ await GLOBAL_LOGGING_WORKER.flush()++ both_attempts: Final = litellm.completion_cost(refused, model="gpt-4o-mini") + litellm.completion_cost(+ served, model="gpt-4o-mini"+ )+ ((kwargs, response_obj),) = recorder.events+ assert getattr(response_obj, "id", None) == "served"+ assert kwargs["response_cost"] == pytest.approx(both_attempts)+ assert prepared is not None+ assert prepared.amount == pytest.approx(both_attempts)