Backtestslitellmrun-20261019-033420#42078
[Feature]: Expose OpenTelemetry trace IDs on LiteLLM Proxy responses
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
The issue asks for a specific header, `x-litellm-trace-id`, carrying the bare 32-character lowercase W3C trace ID. Patch A adds exactly that: a `TraceIdHeaderMiddleware` nested inside the OTel server-span middleware that anchors the recording server span on `request.state`, writes the header on `http.response.start` (so JSON and streaming initial headers both get it), stamps `litellm.call_id` onto the root span from the response's `x-litellm-call-id`, and handles the two out-of-middleware paths (the `Exception` handler via `with_trace_id_header`, and the legacy-logger span that `_close_dangling_otel_server_span` ends early) plus the OpenTelemetry logger's own early-end paths. It covers both the V2 FastAPI-instrumented span and the legacy `user_api_key_auth`-created span, omits the header when OTel is absent, unsampled, or the route is excluded, and preserves incoming `traceparent`. Its unit tests exercise success, streaming, 400/401/403/429/502 ProxyExceptions, 422 validation, unhandled 500, traceparent propagation, omission cases, legacy mode, and middleware ordering. Patch B instead emits a `traceresponse` header in `00-<trace>-<span>-<flags>` format and never adds `x-litellm-trace-id` at all, so it does not deliver the requested contract; the independent tests confirm every trace-id-header test fails on B while passing on A (only the `litellm.call_id` span-attribute tests pass on B). B is also V2-only (the issue's repro uses V2, but the feature statement is unconditional on a recording server span), and it adds substantial unrelated scope: upstream `traceresponse` relaying and span linking for passthroughs, CORS expose-header changes, a propagator override, and a ~700-line integration test rig. That extra work does not offset the missing deliverable and adds risk. A resolves the issue; B does not.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
44m 25s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
429vs 1,192 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate
The Feature
Add x-litellm-trace-id to LiteLLM Proxy HTTP responses whenever the request has a recording OpenTelemetry server span. Its value should be the root trace ID as a 32-character lowercase W3C trace ID
The response already includes x-litellm-call-id, but a gateway user who receives that ID cannot reliably open the exact trace that handled the request. The proposed response header would make a client response, the request trace, and the existing call ID directly navigable across tracing and logging backends
For any response that has both values, the root server span should also carry the same litellm.call_id as x-litellm-call-id. This establishes an explicit relationship between the response, root span, and downstream LLM spans
Expected behavior:
- Return the header on successful JSON responses and the initial headers of streaming responses
- Return the header on validation, authentication, authorization, rate-limit, provider, and unhandled-error responses
- Preserve incoming
traceparentpropagation and return the resulting request trace ID - Omit the header when OpenTelemetry is disabled or no recording request span exists
- Keep
x-litellm-call-idbehavior unchanged
This is a LiteLLM Proxy / AI Gateway feature only. It does not change the Python SDK or WebSocket behavior
Related: #31864 covers request identity in JSON logs. This request covers the gateway HTTP response and its OpenTelemetry root span
User Flow
Before this feature (today): a gateway developer investigating a failed model request reaches the response's call ID but cannot navigate directly to the trace that handled it
- They send
POST https://gateway.example.com/v1/chat/completionswith their model and messages - The gateway returns
400 Bad Requestor a provider error withx-litellm-call-id: 7ac48d62-40a1-44e6-8b50-19da641835a5 - They search the observability backend for that call ID and manually inspect matching LLM spans, logs, or request time ranges to identify the parent request trace
- Their workaround is to correlate timestamps and child-span fields, which is slow and ambiguous when several retries or requests are in flight
After this feature (ideal user flow): the same developer can open the exact request trace from the gateway response
- They send the same
POST https://gateway.example.com/v1/chat/completionswith the same model and messages - The gateway returns the same status and
x-litellm-call-id, plusx-litellm-trace-id: af353cab815b907430a117c9b97b4d0e - They paste
af353cab815b907430a117c9b97b4d0einto their observability backend and open the one server trace for that gateway request - They use the call ID visible on that trace to correlate request logs and downstream model spans without matching timestamps by hand
How far you got
Config / setup the proxy ran with:
model_list: []
general_settings:
master_key: sk-local-trace-repro
litellm_settings:
telemetry: false
Version or commit: 5417abd58672 on the current main branch
Environment variables: LITELLM_OTEL_V2=true, OTEL_EXPORTER=console, and OTEL_SERVICE_NAME=litellm-trace-header-repro
Commands and their full output, up to the step that dead-ends:
$ LITELLM_OTEL_V2=true OTEL_EXPORTER=console OTEL_SERVICE_NAME=litellm-trace-header-repro uv run litellm --config /private/tmp/litellm-trace-header-repro.yaml --host 127.0.0.1 --port 4000
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:4000 (Press CTRL+C to quit)
$ curl --max-time 15 -i -X POST http://127.0.0.1:4000/v1/chat/completions -H 'Authorization: Bearer sk-local-trace-repro' -H 'Content-Type: application/json' --data '{"model":"missing-model","messages":[{"role":"user","content":"hello"}]}'
HTTP/1.1 400 Bad Request
date: Sun, 20 Sep 2026 02:32:23 GMT
x-litellm-call-id: 7ac48d62-40a1-44e6-8b50-19da641835a5
x-litellm-response-cost: 0
x-litellm-key-spend: 0.0
content-length: 371
content-type: application/json
x-frame-options: DENY
content-security-policy: frame-ancestors 'none'
x-content-type-options: nosniff
{"error":{"message":"/chat/completions: Invalid model name passed in model=missing-model. Call `/v1/models` to view available models for your key.","type":"invalid_request_error","param":null,"code":"400","provider_specific_fields":{"error":"/chat/completions: Invalid model name passed in model=missing-model. Call `/v1/models` to view available models for your key."}}}
The console exporter emitted this server span for the same request:
{
"name": "POST /v1/chat/completions",
"context": {
"trace_id": "0xaf353cab815b907430a117c9b97b4d0e",
"span_id": "0x20cb45e2212a998e"
},
"kind": "SpanKind.SERVER",
"status": {
"status_code": "ERROR"
}
}
What stopped me there: the live proxy created a server trace with ID af353cab815b907430a117c9b97b4d0e and returned a different call ID, but the HTTP response did not expose the trace ID. A client receiving the error therefore cannot navigate straight to the trace
Which part of LiteLLM is this about?
Logging: callbacks, Langfuse, Datadog, OTel, Prometheus, alerting
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
No
The two patches
litellm/integrations/opentelemetry.pyindex 90383ddb60..ff9b6c97de 100644@@ -999,6 +999,16 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger): parent_span=parent_otel_span, ) + # Tie the root span to the response's x-litellm-call-id before it is+ # ended here, ahead of the proxy's response being sent.+ litellm_call_id: Final = request_data.get("litellm_call_id")+ if litellm_call_id:+ self.safe_set_attribute(+ span=parent_otel_span,+ key="litellm.call_id",+ value=litellm_call_id,+ )+ # End Parent OTEL Sspan parent_otel_span.end(end_time=self._to_ns(datetime.now())) @@ -1350,6 +1360,14 @@ class OpenTelemetry(OTELGenAISemconvMixin, CustomLogger): and hasattr(proxy_span, "is_recording") and proxy_span.is_recording() ):+ # May end before the response's x-litellm-call-id is sent, so stamp it now.+ litellm_call_id: Final = kwargs.get("litellm_call_id")+ if litellm_call_id:+ self.safe_set_attribute(+ span=proxy_span,+ key="litellm.call_id",+ value=litellm_call_id,+ ) self._close_proxy_span_ok(proxy_span, end_time) def _close_proxy_span_ok(self, span: Span, end_time) -> None:litellm/proxy/anthropic_endpoints/endpoints.pyindex d4727cc8d8..d6f4941025 100644@@ -72,7 +72,7 @@ def _anthropic_error_json_response(exc: ProxyException, request: Request) -> JSO ) status_code: Final = int(exc.code) if exc.code is not None and exc.code.isdigit() else 500- _close_dangling_otel_server_span(request, status_code, exc=exc)+ _close_dangling_otel_server_span(request, status_code, exc=exc, call_id=exc.headers.get(LITELLM_CALL_ID_HEADER)) envelope: Final = AnthropicExceptionMapping.transform_to_anthropic_error( status_code=status_code, raw_message=exc.message,litellm/proxy/auth/user_api_key_auth.pyindex 46eb366228..684fe920b1 100644@@ -164,6 +164,7 @@ from litellm.proxy.db.db_lookup_gate import bounded_db_lookup from litellm.proxy.db.db_span import db_span from litellm.proxy.db.exception_handler import PrismaDBExceptionHandler from litellm.proxy.litellm_pre_call_utils import LiteLLMProxyRequestSetup+from litellm.proxy.middleware.trace_id_header_middleware import anchor_request_root_span from litellm.proxy.spend_tracking.carried_budget_state import carry_team_and_user_budget_state from litellm.proxy.spend_tracking.spend_counter_batch import ( bind_admission_counter_keys,@@ -1950,6 +1951,7 @@ def _ensure_parent_otel_span_on_request_state(request: Request) -> None: http_route=get_request_route_template(request), ) request.state.parent_otel_span = parent_otel_span+ anchor_request_root_span(request.state, parent_otel_span) async def _read_request_body_deferring_parse_failure(litellm/proxy/middleware/trace_id_header_middleware.pynew file mode 100644index 0000000000..4c5eab53d9@@ -0,0 +1,114 @@+"""+Exposes the request's OpenTelemetry trace ID as ``x-litellm-trace-id`` on every+HTTP response, and stamps the response's ``x-litellm-call-id`` onto the request's+root server span as ``litellm.call_id``.++The root span is anchored on ``request.state`` the first time the proxy sees it:+under OTel V2 that is the FastAPI instrumentation's SERVER span, active when this+middleware runs because it is registered inside ``OpenTelemetryMiddleware``; under+the legacy logger it is the span ``user_api_key_auth`` creates. Anchoring on+``request.state`` rather than reading the active span at response time lets the+unhandled-exception handler, which runs in ``ServerErrorMiddleware`` after the+server span has ended, still answer with the trace ID.++The header is omitted when OpenTelemetry is not installed, disabled, or the+request has no recording span (excluded route, sampled out).+"""++from importlib.util import find_spec+from typing import Final++from starlette.datastructures import MutableHeaders, State+from starlette.requests import Request+from starlette.responses import Response+from starlette.types import ASGIApp, Message, Receive, Scope, Send++from litellm.proxy.common_utils.openai_error_payload import LITELLM_CALL_ID_HEADER++LITELLM_TRACE_ID_HEADER: Final = "x-litellm-trace-id"+CALL_ID_SPAN_ATTRIBUTE: Final = "litellm.call_id"+_TRACE_ID_STATE_KEY: Final = "litellm_trace_id"+_ROOT_SPAN_STATE_KEY: Final = "litellm_trace_root_span"+# opentelemetry is an optional proxy dependency; resolve it once instead of+# paying a failed import on every request when it is absent.+_OTEL_AVAILABLE: Final = find_spec("opentelemetry") is not None+++def _recording_trace_id(span: object) -> str | None:+ """The 32-char lowercase W3C trace ID of a recording span with a valid context."""+ if not _OTEL_AVAILABLE:+ return None+ from opentelemetry.trace import Span, format_trace_id++ if not isinstance(span, Span) or not span.is_recording():+ return None+ ctx: Final = span.get_span_context()+ return format_trace_id(ctx.trace_id) if ctx.is_valid else None+++def anchor_request_root_span(state: State, span: object) -> None:+ """Remember ``span`` as the request's root span if it records and none is anchored yet."""+ if getattr(state, _TRACE_ID_STATE_KEY, None) is not None:+ return+ trace_id: Final = _recording_trace_id(span)+ if trace_id is None:+ return+ setattr(state, _TRACE_ID_STATE_KEY, trace_id)+ setattr(state, _ROOT_SPAN_STATE_KEY, span)+++def request_trace_id(state: State) -> str | None:+ trace_id: Final = getattr(state, _TRACE_ID_STATE_KEY, None)+ return trace_id if isinstance(trace_id, str) else None+++def set_call_id_on_root_span(span: object, call_id: str | None) -> None:+ """Stamp ``litellm.call_id`` onto a still-recording root span."""+ if not call_id or not _OTEL_AVAILABLE:+ return+ from opentelemetry.trace import Span++ if not isinstance(span, Span) or not span.is_recording():+ return+ span.set_attribute(CALL_ID_SPAN_ATTRIBUTE, call_id)+++def response_trace_id(state: State, call_id: str | None) -> str | None:+ """The ``x-litellm-trace-id`` value for a response, tying its call ID to the root span."""+ trace_id: Final = request_trace_id(state)+ if trace_id is not None:+ set_call_id_on_root_span(getattr(state, _ROOT_SPAN_STATE_KEY, None), call_id)+ return trace_id+++def with_trace_id_header(state: State, response: Response) -> Response:+ """``response`` carrying ``x-litellm-trace-id``, for responses built outside this middleware."""+ headers: Final = response.headers+ trace_id: Final = response_trace_id(state, headers.get(LITELLM_CALL_ID_HEADER))+ if trace_id is not None:+ headers[LITELLM_TRACE_ID_HEADER] = trace_id+ return response+++class TraceIdHeaderMiddleware:+ def __init__(self, app: ASGIApp) -> None:+ self.app = app++ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:+ if scope["type"] != "http" or not _OTEL_AVAILABLE:+ await self.app(scope, receive, send)+ return+ from opentelemetry.trace import get_current_span++ state: Final = Request(scope).state+ anchor_request_root_span(state, get_current_span())++ async def send_with_trace_id(message: Message) -> None:+ if message["type"] == "http.response.start":+ headers: Final = MutableHeaders(scope=message)+ trace_id: Final = response_trace_id(state, headers.get(LITELLM_CALL_ID_HEADER))+ if trace_id is not None:+ headers[LITELLM_TRACE_ID_HEADER] = trace_id+ await send(message)++ await self.app(scope, receive, send_with_trace_id)litellm/proxy/proxy_server.pyindex b51df22d07..b13bdb4f52 100644@@ -801,6 +801,11 @@ from litellm.proxy.middleware.request_size_limit_middleware import ( from litellm.proxy.middleware.security_headers_middleware import ( SecurityHeadersMiddleware, )+from litellm.proxy.middleware.trace_id_header_middleware import (+ TraceIdHeaderMiddleware,+ set_call_id_on_root_span,+ with_trace_id_header,+) from litellm.proxy.moyai_endpoints import router as moyai_router from litellm.proxy.ocr_endpoints.endpoints import router as ocr_router from litellm.proxy.openai_files_endpoints.files_endpoints import (@@ -1908,6 +1913,10 @@ app = FastAPI( ## no-op when the gate is off or the instrumentation package is unavailable. from litellm.integrations.otel.mount import instrument_fastapi_app +## Added before ``instrument_fastapi_app`` so it nests *inside* the OTel server-span+## middleware (Starlette makes the last-added middleware outermost) and sees that+## span as current when it anchors the request's trace ID for ``x-litellm-trace-id``.+app.add_middleware(TraceIdHeaderMiddleware) instrument_fastapi_app(app) vertex_live_passthrough_vertex_base: Final = VertexBase()@@ -2058,7 +2067,7 @@ async def openai_exception_handler(request: Request, exc: ProxyException): error_body_call_id(general_settings_view(), headers.get(LITELLM_CALL_ID_HEADER)), ) status_code: Final = int(exc.code) if exc.code else status.HTTP_500_INTERNAL_SERVER_ERROR- _close_dangling_otel_server_span(request, status_code, exc=exc)+ _close_dangling_otel_server_span(request, status_code, exc=exc, call_id=headers.get(LITELLM_CALL_ID_HEADER)) otlp_response: Final = tracing_endpoints.otlp_error_response(request, status_code, headers) if otlp_response is not None: return otlp_response@@ -2098,12 +2107,17 @@ def _websocket_close_code(status_code: int) -> int: return status.WS_1011_INTERNAL_ERROR if status_code >= 500 else status.WS_1008_POLICY_VIOLATION -def _close_dangling_otel_server_span(request: Request, status_code: int, exc: Exception | None = None) -> None:+def _close_dangling_otel_server_span(+ request: Request, status_code: int, exc: Exception | None = None, call_id: str | None = None+) -> None: parent_otel_span: Final[_Span | None] = getattr(request.state, "parent_otel_span", None) if parent_otel_span is None: return if open_telemetry_logger is None: return+ # The legacy span is ended below, before the response's x-litellm-call-id is+ # sent, so tie the call ID to it now rather than at response start.+ set_call_id_on_root_span(parent_otel_span, call_id) # Under OTel V2 the FastAPI instrumentor owns the server span (parent_otel_span # is that same span) and ends it itself with the http.* attributes stamped on # completion. The instrumentor only records an error when the exception reaches@@ -2221,6 +2235,12 @@ async def otel_request_validation_exception_handler(request: Request, exc: Reque @app.exception_handler(Exception) async def otel_unhandled_exception_handler(request: Request, exc: Exception):+ # Runs in ServerErrorMiddleware, outside TraceIdHeaderMiddleware, so the trace+ # ID it anchored on request.state is stamped here instead.+ return with_trace_id_header(request.state, await _unhandled_exception_response(request, exc))+++async def _unhandled_exception_response(request: Request, exc: Exception) -> Response: if isinstance(exc, (ProxyException, HTTPException, RequestValidationError)): raise exc if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(exc):tests/unit/integrations/open_telemetry/test_otel_unified_endpoints.pyindex 0279c22154..6285a24f38 100644@@ -177,6 +177,47 @@ def test_async_post_call_failure_hook_defaults_to_500( ) +def _finished_span_named(exporter, name):+ return next(s for s in exporter.get_finished_spans() if s.name == name)+++# The logger ends the legacy root span itself, before the proxy sends the+# response's x-litellm-call-id, so the call ID must be stamped at that close.+def test_end_proxy_span_from_kwargs_stamps_call_id_on_root_span(+ otel_with_exporter, server_span_factory+):+ from datetime import datetime++ otel, exporter = otel_with_exporter+ server_span = server_span_factory(CHAT_PATH)+ kwargs = {+ "litellm_call_id": "call-success-1",+ "litellm_params": {"metadata": {"litellm_parent_otel_span": server_span}},+ }+ otel._end_proxy_span_from_kwargs(kwargs, datetime.now())++ root = _finished_span_named(exporter, server_span.name)+ assert root.attributes.get("litellm.call_id") == "call-success-1", dict(root.attributes)+++def test_async_post_call_failure_hook_stamps_call_id_on_root_span(+ otel_with_exporter, server_span_factory+):+ otel, exporter = otel_with_exporter+ server_span = server_span_factory(CHAT_PATH)++ asyncio.run(+ otel.async_post_call_failure_hook(+ request_data={"litellm_call_id": "call-failure-1"},+ original_exception=make_fastapi_http_exception(429, "rate limit"),+ user_api_key_dict=_real_user_api_key_dict(server_span),+ )+ )++ root = _finished_span_named(exporter, server_span.name)+ assert root.attributes.get("litellm.call_id") == "call-failure-1", dict(root.attributes)++ SMOKE_ENDPOINTS = [ "/v1/embeddings", "/v1/completions",tests/unit/proxy/middleware/test_trace_id_header_middleware.pynew file mode 100644index 0000000000..2dcb49b640@@ -0,0 +1,228 @@+"""``x-litellm-trace-id`` on proxy responses and ``litellm.call_id`` on the root server span.++The app here mirrors the proxy's wiring: ``TraceIdHeaderMiddleware`` is added before+the FastAPI OTel instrumentation (so it nests inside the server span) and the+proxy's real exception handlers are registered.+"""++import pytest++pytest.importorskip("opentelemetry")+pytest.importorskip("opentelemetry.instrumentation.fastapi")++from fastapi import FastAPI, Request # noqa: E402+from fastapi.exceptions import RequestValidationError # noqa: E402+from fastapi.responses import JSONResponse, StreamingResponse # noqa: E402+from fastapi.testclient import TestClient # noqa: E402+from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor, OpenTelemetryMiddleware # noqa: E402+from opentelemetry.sdk.trace import TracerProvider # noqa: E402+from opentelemetry.sdk.trace.export import SimpleSpanProcessor # noqa: E402+from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter # noqa: E402+from opentelemetry.trace import SpanKind # noqa: E402+from pydantic import BaseModel # noqa: E402++import litellm.proxy.proxy_server as proxy_server_module # noqa: E402+from litellm.integrations.opentelemetry import OpenTelemetry # noqa: E402+from litellm.integrations.otel.model.config import is_otel_v2_enabled # noqa: E402+from litellm.proxy._types import ProxyException # noqa: E402+from litellm.proxy.auth.user_api_key_auth import _ensure_parent_otel_span_on_request_state # noqa: E402+from litellm.proxy.middleware.trace_id_header_middleware import TraceIdHeaderMiddleware # noqa: E402+from litellm.proxy.proxy_server import ( # noqa: E402+ openai_exception_handler,+ otel_request_validation_exception_handler,+ otel_unhandled_exception_handler,+)++CALL_ID = "7ac48d62-40a1-44e6-8b50-19da641835a5"+INCOMING_TRACE_ID = "af353cab815b907430a117c9b97b4d0e"+++class _Body(BaseModel):+ model: str+++def _app(instrument: bool = True) -> tuple[FastAPI, InMemorySpanExporter]:+ app = FastAPI()++ @app.post("/v1/chat/completions")+ def chat(body: _Body):+ return JSONResponse({"model": body.model}, headers={"x-litellm-call-id": CALL_ID})++ @app.post("/stream")+ def stream():+ return StreamingResponse(iter([b"data: 1\n\n"]), headers={"x-litellm-call-id": CALL_ID})++ @app.post("/proxy-error/{code}")+ def proxy_error(code: int):+ raise ProxyException(+ message="denied", type="auth_error", param=None, code=code, headers={"x-litellm-call-id": CALL_ID}+ )++ @app.post("/boom")+ def boom():+ raise RuntimeError("boom")++ @app.get("/health")+ def health():+ return {"ok": True}++ app.add_exception_handler(ProxyException, openai_exception_handler)+ app.add_exception_handler(RequestValidationError, otel_request_validation_exception_handler)+ app.add_exception_handler(Exception, otel_unhandled_exception_handler)++ exporter = InMemorySpanExporter()+ provider = TracerProvider()+ provider.add_span_processor(SimpleSpanProcessor(exporter))+ app.add_middleware(TraceIdHeaderMiddleware)+ if instrument:+ FastAPIInstrumentor.instrument_app(app, tracer_provider=provider, excluded_urls="/health")+ return app, exporter+++def _server_span(exporter: InMemorySpanExporter):+ spans = [s for s in exporter.get_finished_spans() if s.kind is SpanKind.SERVER]+ assert len(spans) == 1+ return spans[0]+++def _assert_trace_header(response, exporter: InMemorySpanExporter) -> str:+ trace_id = response.headers["x-litellm-trace-id"]+ assert len(trace_id) == 32 and trace_id == trace_id.lower()+ assert int(trace_id, 16) == _server_span(exporter).context.trace_id+ return trace_id+++@pytest.fixture(autouse=True)+def _no_otel_logger(monkeypatch):+ monkeypatch.setattr(proxy_server_module, "open_telemetry_logger", None)+++def test_success_json_response_carries_trace_id_and_call_id_on_root_span():+ app, exporter = _app()+ response = TestClient(app).post("/v1/chat/completions", json={"model": "gpt"})+ assert response.status_code == 200+ _assert_trace_header(response, exporter)+ assert response.headers["x-litellm-call-id"] == CALL_ID+ assert (_server_span(exporter).attributes or {})["litellm.call_id"] == CALL_ID+++def test_streaming_response_initial_headers_carry_trace_id():+ app, exporter = _app()+ response = TestClient(app).post("/stream")+ assert response.status_code == 200+ _assert_trace_header(response, exporter)+ assert (_server_span(exporter).attributes or {})["litellm.call_id"] == CALL_ID+++@pytest.mark.parametrize("code", [400, 401, 403, 429, 502])+def test_proxy_error_responses_carry_trace_id(code: int):+ app, exporter = _app()+ response = TestClient(app).post(f"/proxy-error/{code}")+ assert response.status_code == code+ _assert_trace_header(response, exporter)+ assert response.headers["x-litellm-call-id"] == CALL_ID+ assert (_server_span(exporter).attributes or {})["litellm.call_id"] == CALL_ID+++def test_validation_error_response_carries_trace_id():+ app, exporter = _app()+ response = TestClient(app).post("/v1/chat/completions", json={})+ assert response.status_code == 422+ _assert_trace_header(response, exporter)+++def test_unhandled_error_response_carries_trace_id():+ app, exporter = _app()+ response = TestClient(app, raise_server_exceptions=False).post("/boom")+ assert response.status_code == 500+ assert response.json()["error"]["type"] == "internal_server_error"+ _assert_trace_header(response, exporter)+++def test_incoming_traceparent_is_propagated_to_the_header():+ app, exporter = _app()+ response = TestClient(app).post(+ "/v1/chat/completions",+ json={"model": "gpt"},+ headers={"traceparent": f"00-{INCOMING_TRACE_ID}-00f067aa0ba902b7-01"},+ )+ assert _assert_trace_header(response, exporter) == INCOMING_TRACE_ID+++def test_header_omitted_without_otel_instrumentation():+ app, _ = _app(instrument=False)+ response = TestClient(app).post("/v1/chat/completions", json={"model": "gpt"})+ assert response.status_code == 200+ assert "x-litellm-trace-id" not in response.headers+ assert response.headers["x-litellm-call-id"] == CALL_ID+++def test_header_omitted_when_route_has_no_server_span():+ app, exporter = _app()+ response = TestClient(app).get("/health")+ assert response.status_code == 200+ assert "x-litellm-trace-id" not in response.headers+ assert exporter.get_finished_spans() == ()+++def test_header_omitted_when_server_span_is_sampled_out():+ from opentelemetry.sdk.trace.sampling import ALWAYS_OFF++ app = FastAPI()++ @app.get("/ping")+ def ping():+ return {"ok": True}++ app.add_middleware(TraceIdHeaderMiddleware)+ FastAPIInstrumentor.instrument_app(app, tracer_provider=TracerProvider(sampler=ALWAYS_OFF))+ response = TestClient(app).get("/ping")+ assert response.status_code == 200+ assert "x-litellm-trace-id" not in response.headers+++def test_legacy_otel_server_span_trace_id_and_call_id(monkeypatch):+ """Legacy (non-V2) OTel: the SERVER span is created at auth and ended by the+ error handler before the response is sent; both values still line up."""+ monkeypatch.delenv("LITELLM_OTEL_V2", raising=False)+ is_otel_v2_enabled.cache_clear()+ exporter = InMemorySpanExporter()+ provider = TracerProvider()+ provider.add_span_processor(SimpleSpanProcessor(exporter))+ otel = OpenTelemetry()+ otel.tracer = provider.get_tracer("trace-id-header-tests")+ monkeypatch.setattr(proxy_server_module, "open_telemetry_logger", otel)++ app = FastAPI()++ @app.post("/v1/chat/completions")+ def chat(request: Request):+ _ensure_parent_otel_span_on_request_state(request)+ raise ProxyException(+ message="bad model",+ type="invalid_request_error",+ param=None,+ code=400,+ headers={"x-litellm-call-id": CALL_ID},+ )++ app.add_exception_handler(ProxyException, openai_exception_handler)+ app.add_middleware(TraceIdHeaderMiddleware)++ response = TestClient(app).post(+ "/v1/chat/completions", headers={"traceparent": f"00-{INCOMING_TRACE_ID}-00f067aa0ba902b7-01"}+ )+ is_otel_v2_enabled.cache_clear()++ assert response.status_code == 400+ assert response.headers["x-litellm-trace-id"] == INCOMING_TRACE_ID+ assert response.headers["x-litellm-call-id"] == CALL_ID+ assert (_server_span(exporter).attributes or {})["litellm.call_id"] == CALL_ID+++def test_proxy_app_nests_trace_middleware_inside_otel_server_span():+ middleware = [m.cls for m in proxy_server_module.app.user_middleware]+ assert TraceIdHeaderMiddleware in middleware+ if OpenTelemetryMiddleware in middleware:+ # user_middleware is listed outermost first.+ assert middleware.index(TraceIdHeaderMiddleware) > middleware.index(OpenTelemetryMiddleware)
litellm/integrations/otel/README.mdindex 340bd71c983..7414a451038 100644@@ -194,7 +194,10 @@ matching the FastAPI server span); only a genuine error sets `ERROR`. - **Server spans** (one per HTTP route) are created by the `opentelemetry-instrumentation-fastapi` package. It stamps `http.*` attributes and extracts inbound `traceparent` headers. This package does **not** create- or modify server spans — request routes never touch spans.+ server spans. It renames passthrough server spans, sets `litellm.call_id` on+ them, and returns each sampled server span to the client in a `traceresponse`+ response header (`trace_response.py`). A relayed upstream `traceresponse` is+ kept, and the server span links to it when that upstream restarted the trace. - **Gen-AI spans** (LLM calls, guardrails, internal service calls) are created by this package from LiteLLM's logging callbacks. Request-level spans parent to the server span via the captured anchor; DB/service spans parent to the activelitellm/integrations/otel/mount.pyindex 9340f6e9e15..cb52ad24c8e 100644@@ -12,6 +12,8 @@ when the feature gate is off. """ import os+from collections.abc import Mapping+from types import MappingProxyType from typing import TYPE_CHECKING, Final, Protocol from litellm._logging import verbose_logger@@ -19,6 +21,7 @@ from litellm.integrations.otel.model.config import is_otel_v2_enabled if TYPE_CHECKING: from fastapi import FastAPI+ from opentelemetry.trace import Span, TracerProvider # Routes excluded from server-span tracing by default: high-frequency pollers and # static UI/docs assets, none of which are LLM traffic. Entries are substring-matched@@ -76,7 +79,7 @@ class _RenameableSpan(Protocol): def set_attribute(self, key: str, value: str) -> None: ... -def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: dict) -> None:+def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: Mapping[str, object]) -> None: """FastAPI ``server_request_hook``: give passthrough server spans a useful name. The instrumentation matches the route at span creation, so both the span name@@ -89,8 +92,8 @@ def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: dict) -> try: if span is None or not span.is_recording(): return- path: Final = scope.get("path") or ""- method: Final = scope.get("method") or ""+ path: Final = str(scope.get("path") or "")+ method: Final = str(scope.get("method") or "") first_segment: Final = path.lstrip("/").split("/", 1)[0] if first_segment in PASSTHROUGH_PREFIXES: span.update_name(f"{method} {path}".strip())@@ -99,7 +102,7 @@ def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: dict) -> pass -def instrument_fastapi_app(app: "FastAPI") -> None:+def instrument_fastapi_app(app: "FastAPI", tracer_provider: "TracerProvider | None" = None) -> None: """Attach OTel server-span instrumentation to the proxy FastAPI app. Safe no-op when the V2 gate is off or ``opentelemetry-instrumentation-fastapi``@@ -111,6 +114,14 @@ def instrument_fastapi_app(app: "FastAPI") -> None: ``ProxyTracerProvider``; the proxy publishes the real provider as the global after config load (see ``proxy_startup_event``), and the proxy delegates to it. That way server spans and gen-ai spans share one provider and the same trace.++ Once instrumented, every sampled HTTP response this app sends carries a+ ``traceresponse`` header naming its server span, unless the response already+ relays an upstream's ``traceresponse`` or another response propagator is already+ registered. Other OTel-instrumented apps in the same process get no header. A relayed+ ``traceresponse`` from an upstream that restarted the trace becomes a link on the+ server span. The server span also gets ``litellm.call_id`` from the response's+ ``x-litellm-call-id`` header. ``tracer_provider`` defaults to the global provider. """ try: if not is_otel_v2_enabled():@@ -122,6 +133,20 @@ def instrument_fastapi_app(app: "FastAPI") -> None: # ``proxy_server``'s unconditional ``import`` of this module crash when the # package is absent, even with the gate off. from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor+ from opentelemetry.instrumentation.propagators import (+ get_global_response_propagator,+ set_global_response_propagator, # pyright: ignore[reportUnknownVariableType] # upstream leaves its parameter unannotated+ )++ from litellm.integrations.otel.trace_response import (+ UpstreamFirstTraceResponsePropagator,+ claim_server_span,+ )++ def server_request_hook(span: "Span | None", scope: Mapping[str, object]) -> None:+ if span is not None:+ claim_server_span(span)+ _passthrough_span_name_hook(span, scope) excluded_urls: Final = ( os.environ.get("OTEL_PYTHON_FASTAPI_EXCLUDED_URLS")@@ -130,12 +155,47 @@ def instrument_fastapi_app(app: "FastAPI") -> None: ) FastAPIInstrumentor.instrument_app( app,+ tracer_provider=tracer_provider, excluded_urls=excluded_urls,- server_request_hook=_passthrough_span_name_hook,+ server_request_hook=server_request_hook, # Drop the ASGI "http receive"/"http send" lifecycle sub-spans: they # are low-value noise and (for passthrough) carry the catch-all route # template in their name, which can't be rewritten from a hook. exclude_spans=["receive", "send"], )+ if get_global_response_propagator() is None:+ set_global_response_propagator(UpstreamFirstTraceResponsePropagator()) except Exception as e: verbose_logger.debug("Skipping OTel V2 FastAPI instrumentation: %s", e)+++def trace_response_expose_headers() -> tuple[str, ...]:+ return ("traceresponse",) if is_otel_v2_enabled() else ()+++def trace_response_headers(span: "Span | None") -> Mapping[str, str]:+ """Response headers naming ``span`` for a response built outside the FastAPI instrumentation.++ Starlette runs the ``Exception`` handler outside every middleware, so its 500 never reaches the response+ propagator and the handler has to add these itself.+ """+ if span is None or not is_otel_v2_enabled():+ return MappingProxyType({})+ try:+ from opentelemetry import trace+ from opentelemetry.instrumentation.asgi import ( # pyright: ignore[reportMissingTypeStubs] # the ASGI instrumentation ships no py.typed marker+ asgi_getter,+ asgi_setter,+ )+ from opentelemetry.instrumentation.propagators import get_global_response_propagator++ from litellm.integrations.otel.trace_response import claim_server_span+ except ImportError:+ return MappingProxyType({})+ claim_server_span(span)+ propagator: Final = get_global_response_propagator() # pyright: ignore[reportUnknownVariableType] # upstream leaves inject unannotated+ if propagator is None:+ return MappingProxyType({})+ start: Final[dict[str, object]] = {"type": "http.response.start"} # mutable-ok: the propagator writes into it+ propagator.inject(start, context=trace.set_span_in_context(span), setter=asgi_setter) # pyright: ignore[reportUnknownMemberType] # upstream leaves inject unannotated+ return MappingProxyType({key: ", ".join(asgi_getter.get(start, key) or ()) for key in asgi_getter.keys(start)}) # pyright: ignore[reportUnknownMemberType] # the ASGI instrumentation ships no py.typed markerlitellm/integrations/otel/trace_response.pynew file mode 100644index 00000000000..d602e372788@@ -0,0 +1,74 @@+from contextvars import ContextVar+from typing import Final++from opentelemetry import trace+from opentelemetry.context import Context+from opentelemetry.instrumentation.asgi import ( # pyright: ignore[reportMissingTypeStubs] # the ASGI instrumentation ships no py.typed marker+ asgi_getter,+)+from opentelemetry.instrumentation.propagators import ResponsePropagator+from opentelemetry.propagators.textmap import CarrierT, Setter, default_setter+from opentelemetry.trace import Span, format_span_id, format_trace_id+from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator++from litellm.integrations.otel.model.semconv import LiteLLM++_LITELLM_SERVER_SPAN: Final[ContextVar[Span | None]] = ContextVar(+ "litellm_otel_traceresponse_server_span", default=None+)+++def claim_server_span(span: Span) -> None:+ _LITELLM_SERVER_SPAN.set(span)+++def _response_header(message: dict[str, object], name: str) -> str | None: # mutable-ok: asgi_getter takes a dict+ values: Final = asgi_getter.get(message, name) # pyright: ignore[reportUnknownMemberType] # the ASGI instrumentation ships no py.typed marker+ return values[0] if values else None+++def _http_response_start(message: object) -> dict[str, object] | None: # mutable-ok: the propagator writes into it+ if isinstance(message, dict) and message.get("type") == "http.response.start": # pyright: ignore[reportUnknownMemberType] # an ASGI message is an untyped dict+ return message # pyright: ignore[reportUnknownVariableType] # an ASGI message is an untyped dict+ return None+++def _link_a_restarted_upstream_trace(span: Span, upstream_traceresponse: str) -> None:+ upstream: Final = trace.get_current_span(+ TraceContextTextMapPropagator().extract({"traceparent": upstream_traceresponse})+ ).get_span_context()+ if upstream.trace_id != span.get_span_context().trace_id:+ span.add_link(upstream)+++class UpstreamFirstTraceResponsePropagator(ResponsePropagator):+ def inject(+ self,+ carrier: CarrierT,+ context: Context | None = None,+ setter: Setter[CarrierT] = default_setter,+ ) -> None:+ start: Final = _http_response_start(carrier)+ if start is None:+ return+ span: Final = trace.get_current_span(context)+ if span is not _LITELLM_SERVER_SPAN.get():+ return+ call_id: Final = _response_header(start, "x-litellm-call-id")+ if call_id:+ span.set_attribute(LiteLLM.CALL_ID, call_id)+ upstream_traceresponse: Final = _response_header(start, "traceresponse")+ if upstream_traceresponse is not None:+ _link_a_restarted_upstream_trace(span, upstream_traceresponse)+ return+ own: Final = span.get_span_context()+ if not own.trace_flags.sampled:+ return+ sent: Final = start.get("headers")+ if isinstance(sent, tuple):+ start["headers"] = [*sent]+ setter.set(+ carrier,+ "traceresponse",+ f"00-{format_trace_id(own.trace_id)}-{format_span_id(own.span_id)}-{own.trace_flags:02x}",+ )litellm/proxy/proxy_server.pyindex b51df22d07a..72a10dd1ec9 100644@@ -1906,7 +1906,11 @@ app = FastAPI( ## middleware after an application has started". See ## ``litellm.integrations.otel.mount`` for the full rationale; the call is a safe ## no-op when the gate is off or the instrumentation package is unavailable.-from litellm.integrations.otel.mount import instrument_fastapi_app+from litellm.integrations.otel.mount import (+ instrument_fastapi_app,+ trace_response_expose_headers,+ trace_response_headers,+) instrument_fastapi_app(app) @@ -2223,6 +2227,12 @@ async def otel_request_validation_exception_handler(request: Request, exc: Reque async def otel_unhandled_exception_handler(request: Request, exc: Exception): if isinstance(exc, (ProxyException, HTTPException, RequestValidationError)): raise exc+ response: Final = await _unhandled_exception_response(request, exc)+ response.headers.update(trace_response_headers(getattr(request.state, "parent_otel_span", None)))+ return response+++async def _unhandled_exception_response(request: Request, exc: Exception) -> Response: if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(exc): verbose_proxy_logger.warning("Database unavailable during request: %s", type(exc).__name__) return await openai_exception_handler(@@ -2606,7 +2616,7 @@ app.add_middleware( allow_credentials=allow_cors_credentials, allow_methods=["*"], allow_headers=["*"],- expose_headers=LITELLM_UI_ALLOW_HEADERS,+ expose_headers=[*LITELLM_UI_ALLOW_HEADERS, *trace_response_expose_headers()], ) app.add_middleware(PrometheusAuthMiddleware)tests/integration/observability/test_otel_trace_response_header.pynew file mode 100644index 00000000000..e84aeab48cf@@ -0,0 +1,735 @@+import itertools+import json+import re+import threading+import uuid+from collections import deque+from collections.abc import Callable, Iterator, Mapping, Sequence+from concurrent.futures import ThreadPoolExecutor+from dataclasses import dataclass+from pathlib import Path+from types import MappingProxyType+from typing import Final+from urllib.parse import parse_qs, urlsplit++import anthropic+import httpx+import openai+import pytest+import yaml+from integration._support.client import Gateway, eventually, gateway_from_environment, list_value, object_value+from integration._support.database import read_rows+from integration._support.process import OwnedProxy, graceful_stop_seconds, owned_proxy_process+from integration._support.wire import Reply, Request, Wire, wire_server+from pydantic import JsonValue, TypeAdapter++MARKER: Final = re.compile(rb"traceresp-[0-9a-f]{32}")+TRACERESPONSE: Final = re.compile(r"00-([0-9a-f]{32})-([0-9a-f]{16})-01")+SERVER: Final = 2+CALL_ID: Final = "litellm.call_id"+ANSWER: Final = "trace ok"+JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])+UPSTREAM_CALL_ID: Final = "upstream-chosen-call-id"+HOSTILE_UPSTREAM_VALUES: Final = MappingProxyType(+ {+ "empty": "",+ "numeric": "12345",+ "big": "00-" + "a" * 5000,+ "garbage": "not-a-traceparent",+ "joined": f"00-{'1' * 32}-{'2' * 16}-01, 00-{'3' * 32}-{'4' * 16}-01",+ }+)+RELAY_CASES: Final = ("none", "continued", "restarted", "callid", *HOSTILE_UPSTREAM_VALUES)+++def _marker() -> str:+ return "traceresp-" + uuid.uuid4().hex+++def _trace_id() -> str:+ return uuid.uuid4().hex+++def _span_id() -> str:+ return uuid.uuid4().hex[:16]+++def _chat_reply(identity: str, stream: bool) -> Reply:+ message: Final = {"role": "assistant", "content": ANSWER}+ usage: Final = {"prompt_tokens": 7, "completion_tokens": 2, "total_tokens": 9}+ if not stream:+ return Reply(+ body=json.dumps(+ {+ "id": identity,+ "object": "chat.completion",+ "created": 1,+ "model": "gpt-4o-mini",+ "choices": [{"index": 0, "message": message, "finish_reason": "stop"}],+ "usage": usage,+ }+ ).encode()+ )+ chunk: Final = {"id": identity, "object": "chat.completion.chunk", "created": 1, "model": "gpt-4o-mini"}+ frames: Final = (+ {**chunk, "choices": [{"index": 0, "delta": message}]},+ {**chunk, "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}], "usage": usage},+ )+ return Reply(+ content_type="text/event-stream",+ chunks=(*(b"data: " + json.dumps(frame).encode() + b"\n\n" for frame in frames), b"data: [DONE]\n\n"),+ )+++def _responses_reply(identity: str, stream: bool) -> Reply:+ response: Final = {+ "id": identity,+ "object": "response",+ "created_at": 1,+ "status": "completed",+ "model": "gpt-4o-mini",+ "output": [+ {+ "id": "msg_" + identity,+ "type": "message",+ "role": "assistant",+ "status": "completed",+ "content": [{"type": "output_text", "text": ANSWER, "annotations": []}],+ }+ ],+ "usage": {"input_tokens": 7, "output_tokens": 2, "total_tokens": 9},+ }+ if not stream:+ return Reply(body=json.dumps(response).encode())+ events: Final = (+ {"type": "response.created", "sequence_number": 0, "response": {**response, "status": "in_progress"}},+ {+ "type": "response.output_text.delta",+ "sequence_number": 1,+ "item_id": "msg_" + identity,+ "output_index": 0,+ "content_index": 0,+ "delta": ANSWER,+ },+ {"type": "response.completed", "sequence_number": 2, "response": response},+ )+ return Reply(+ content_type="text/event-stream",+ chunks=tuple(f"event: {event['type']}\ndata: {json.dumps(event)}\n\n".encode() for event in events),+ )+++def _provider(request: Request) -> Reply:+ found: Final = MARKER.search(request.body)+ if found is None:+ return Reply(status=404, body=b'{"error":"no marker"}')+ marker: Final = found.group(0).decode()+ stream: Final = JSON_OBJECT.validate_json(request.body).get("stream") is True+ if request.target.endswith("/responses"):+ return _responses_reply(f"resp_{marker}", stream)+ return _chat_reply(f"chatcmpl-{marker}", stream)+++def _query(request: Request) -> Mapping[str, str]:+ return MappingProxyType({name: values[0] for name, values in parse_qs(urlsplit(request.target).query).items()})+++def _continued_traceresponse(request: Request) -> Reply:+ incoming: Final = TRACERESPONSE.fullmatch(request.headers.get("traceparent", ""))+ if incoming is None:+ return Reply(status=400, body=b'{"error":"no sampled traceparent reached the upstream"}')+ return Reply(body=b'{"case":"continued"}', headers={"traceresponse": f"00-{incoming.group(1)}-{_span_id()}-01"})+++def _relay(request: Request) -> Reply:+ case: Final = urlsplit(request.target).path.rsplit("/", 1)[-1]+ query: Final = _query(request)+ body: Final = json.dumps({"case": case}).encode()+ if case == "continued":+ return _continued_traceresponse(request)+ if case == "restarted":+ return Reply(body=body, headers={"traceresponse": f"00-{query['trace']}-{query['span']}-01"})+ if case == "callid":+ return Reply(status=int(query.get("status", "200")), body=body, headers={"x-litellm-call-id": UPSTREAM_CALL_ID})+ if case in HOSTILE_UPSTREAM_VALUES:+ return Reply(body=body, headers={"traceresponse": HOSTILE_UPSTREAM_VALUES[case]})+ return Reply(body=body)+++@dataclass(frozen=True, slots=True)+class ExportedSpan:+ trace_id: str+ span_id: str+ parent_span_id: str+ kind: int+ attributes: Mapping[str, JsonValue]+ links: tuple[tuple[str, str], ...]+++def _attribute(value: JsonValue) -> JsonValue:+ typed: Final = object_value(value)+ return next((typed[key] for key in ("stringValue", "intValue", "boolValue", "doubleValue") if key in typed), None)+++def _link(value: JsonValue) -> tuple[str, str]:+ link: Final = object_value(value)+ return str(link["traceId"]), str(link["spanId"])+++def _exported(value: JsonValue) -> ExportedSpan:+ span: Final = object_value(value)+ attributes: Final = tuple(object_value(attribute) for attribute in list_value(span.get("attributes", [])))+ return ExportedSpan(+ trace_id=str(span["traceId"]),+ span_id=str(span["spanId"]),+ parent_span_id=str(span.get("parentSpanId", "")),+ kind=int(str(span.get("kind", 0))),+ attributes=MappingProxyType({str(item["key"]): _attribute(item["value"]) for item in attributes}),+ links=tuple(_link(link) for link in list_value(span.get("links", []))),+ )+++def _scope_spans(resource: JsonValue) -> Iterator[JsonValue]:+ return itertools.chain.from_iterable(+ list_value(object_value(scope).get("spans", [])) for scope in list_value(object_value(resource)["scopeSpans"])+ )+++def _batch_spans(body: bytes) -> tuple[ExportedSpan, ...]:+ resources: Final = list_value(JSON_OBJECT.validate_json(body).get("resourceSpans", []))+ return tuple(_exported(span) for span in itertools.chain.from_iterable(map(_scope_spans, resources)))+++@dataclass(frozen=True, slots=True)+class Collector:+ wire: Wire+ outage: threading.Event+ accepted: Sequence[Request]+ guard: threading.Lock++ def spans(self) -> tuple[ExportedSpan, ...]:+ with self.guard:+ batches: Final = tuple(self.accepted)+ return tuple(itertools.chain.from_iterable(_batch_spans(batch.body) for batch in batches))++ def with_span_id(self, span_id: str) -> tuple[ExportedSpan, ...]:+ return tuple(span for span in self.spans() if span.span_id == span_id)++ def span(self, span_id: str) -> ExportedSpan:+ eventually(lambda: self.with_span_id(span_id), lambda found: len(found) >= 1, seconds=30)+ found: Final = self.with_span_id(span_id)+ assert len(found) == 1, found+ return found[0]++ def server_spans_for_call(self, call_id: str) -> tuple[ExportedSpan, ...]:+ return tuple(span for span in self.spans() if span.kind == SERVER and span.attributes.get(CALL_ID) == call_id)++ def server_span_for_call(self, call_id: str) -> ExportedSpan:+ eventually(lambda: self.server_spans_for_call(call_id), lambda found: len(found) >= 1, seconds=30)+ found: Final = self.server_spans_for_call(call_id)+ assert len(found) == 1, found+ return found[0]+++@pytest.fixture(scope="module")+def collector() -> Iterator[Collector]:+ outage: Final = threading.Event()+ accepted: Final[deque[Request]] = deque() # mutable-ok: the sink thread appends each accepted export batch+ guard: Final = threading.Lock()++ def sink(request: Request) -> Reply:+ if outage.is_set():+ return Reply(status=503, body=b'{"error":"sink down"}')+ with guard:+ accepted.append(request)+ return Reply()++ with wire_server(sink) as wire:+ yield Collector(wire, outage, accepted, guard)+++@pytest.fixture(scope="module")+def provider() -> Iterator[Wire]:+ with wire_server(_provider) as wire:+ yield wire+++@pytest.fixture(scope="module")+def relay() -> Iterator[Wire]:+ with wire_server(_relay) as wire:+ yield wire+++@dataclass(frozen=True, slots=True)+class Answer:+ headers: httpx.Headers+ text: str+++@dataclass(frozen=True, slots=True)+class Rig:+ proxy: Gateway+ process: OwnedProxy+ model: str+ upstream: Wire+ sink: Collector++ def base_url(self) -> str:+ return str(self.proxy.client.base_url)++ def openai_client(self) -> openai.OpenAI:+ return openai.OpenAI(base_url=self.base_url() + "/v1", api_key=self.proxy.key, max_retries=0)++ def anthropic_client(self) -> anthropic.Anthropic:+ return anthropic.Anthropic(base_url=self.base_url(), api_key=self.proxy.key, max_retries=0)++ def post(self, path: str, body: Mapping[str, JsonValue], headers: Sequence[tuple[str, str]] = ()) -> httpx.Response:+ return self.proxy.client.post(+ path, json=body, headers=[("Authorization", f"Bearer {self.proxy.key}"), *headers]+ )++ def chat(self, marker: str, *, headers: Sequence[tuple[str, str]] = ()) -> httpx.Response:+ return self.post(+ "/v1/chat/completions",+ {"model": self.model, "messages": [{"role": "user", "content": marker}], "cache": {"no-cache": True}},+ headers,+ )++ def upstream_count(self, marker: str) -> int:+ return sum(1 for request in self.upstream.drain() if marker.encode() in request.body)+++def _config(provider: Wire, relay: Wire, sink: Collector, model: str) -> dict[str, JsonValue]:+ config: Final = object_value(yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text()))+ return {+ **config,+ "model_list": [+ *list_value(config.get("model_list") or []),+ {+ "model_name": model,+ "litellm_params": {+ "model": "openai/gpt-4o-mini",+ "api_key": "integration-provider-key",+ "api_base": provider.url + "/v1",+ },+ },+ ],+ "litellm_settings": {**object_value(config["litellm_settings"]), "callbacks": ["otel"]},+ "callback_settings": {"otel": {"exporter": "http/json", "endpoint": sink.wire.url}},+ "general_settings": {+ **object_value(config["general_settings"]),+ "pass_through_endpoints": [+ {"path": f"/relay/{case}", "target": f"{relay.url}/relay/{case}"} for case in RELAY_CASES+ ],+ },+ }+++@dataclass(frozen=True, slots=True)+class RigFactory:+ provider: Wire+ relay: Wire+ sink: Collector+ directory: Path+ otel_v2: bool++ def start(self) -> Iterator[Rig]:+ model: Final = "traceresp-model-" + uuid.uuid4().hex[:12]+ path: Final = self.directory / f"traceresponse-{uuid.uuid4().hex}.yaml"+ path.write_text(yaml.safe_dump(_config(self.provider, self.relay, self.sink, model)))+ gate: Final = {"LITELLM_OTEL_V2": "1"} if self.otel_v2 else {}+ with (+ gateway_from_environment() as gateway,+ owned_proxy_process(+ gateway,+ self.directory,+ {**gate, "OTEL_BSP_SCHEDULE_DELAY": "300"},+ config=path,+ remove_environment=() if self.otel_v2 else ("LITELLM_OTEL_V2",),+ workers=2,+ ) as owned,+ ):+ yield Rig(owned.gateway, owned, model, self.provider, self.sink)+++@pytest.fixture(scope="module")+def factory(+ provider: Wire, relay: Wire, collector: Collector, tmp_path_factory: pytest.TempPathFactory+) -> Callable[[bool], RigFactory]:+ return lambda otel_v2: RigFactory(provider, relay, collector, tmp_path_factory.mktemp("traceresponse"), otel_v2)+++@pytest.fixture(scope="module")+def rig(factory: Callable[[bool], RigFactory]) -> Iterator[Rig]:+ yield from factory(True).start()+++@pytest.fixture(scope="module")+def control(factory: Callable[[bool], RigFactory]) -> Iterator[Rig]:+ yield from factory(False).start()+++def _named(headers: httpx.Headers) -> tuple[str, str]:+ values: Final = headers.get_list("traceresponse")+ assert len(values) == 1, values+ match: Final = TRACERESPONSE.fullmatch(values[0])+ assert match is not None, values+ return match.group(1), match.group(2)+++def _assert_names_its_server_span(rig: Rig, headers: httpx.Headers) -> ExportedSpan:+ trace_id, span_id = _named(headers)+ span: Final = rig.sink.span(span_id)+ assert (span.kind, span.trace_id) == (SERVER, trace_id), span+ call_ids: Final = headers.get_list("x-litellm-call-id")+ assert len(call_ids) == 1 and span.attributes.get(CALL_ID) == call_ids[0], (call_ids, span)+ return span+++def _chat_sdk(rig: Rig, marker: str) -> Answer:+ completion: Final = rig.openai_client().chat.completions.with_raw_response.create(+ model=rig.model, messages=[{"role": "user", "content": marker}], extra_body={"cache": {"no-cache": True}}+ )+ return Answer(completion.headers, completion.parse().choices[0].message.content or "")+++def _chat_stream_sdk(rig: Rig, marker: str) -> Answer:+ with rig.openai_client().chat.completions.with_streaming_response.create(+ model=rig.model,+ messages=[{"role": "user", "content": marker}],+ stream=True,+ extra_body={"cache": {"no-cache": True}},+ ) as chunks:+ text: Final = "".join(chunk.choices[0].delta.content or "" for chunk in chunks.parse() if chunk.choices)+ return Answer(chunks.headers, text)+++def _messages_sdk(rig: Rig, marker: str) -> Answer:+ reply: Final = rig.anthropic_client().messages.with_raw_response.create(+ model=rig.model, max_tokens=16, messages=[{"role": "user", "content": marker}]+ )+ block: Final = reply.parse().content[0]+ assert isinstance(block, anthropic.types.TextBlock), block+ return Answer(reply.headers, block.text)+++def _messages_stream_sdk(rig: Rig, marker: str) -> Answer:+ with rig.anthropic_client().messages.with_streaming_response.create(+ model=rig.model, max_tokens=16, messages=[{"role": "user", "content": marker}], stream=True+ ) as streamed:+ text: Final = "".join(+ event.delta.text+ for event in streamed.parse()+ if event.type == "content_block_delta" and event.delta.type == "text_delta"+ )+ return Answer(streamed.headers, text)+++def _responses_sdk(rig: Rig, marker: str) -> Answer:+ created: Final = rig.openai_client().responses.with_raw_response.create(model=rig.model, input=marker)+ return Answer(created.headers, created.parse().output_text)+++def _responses_stream_sdk(rig: Rig, marker: str) -> Answer:+ with rig.openai_client().responses.with_streaming_response.create(+ model=rig.model, input=marker, stream=True+ ) as events:+ text: Final = "".join(event.delta for event in events.parse() if event.type == "response.output_text.delta")+ return Answer(events.headers, text)+++SDK_ROUTES: Final[Mapping[str, Callable[[Rig, str], Answer]]] = MappingProxyType(+ {+ "chat": _chat_sdk,+ "chat_stream": _chat_stream_sdk,+ "messages": _messages_sdk,+ "messages_stream": _messages_stream_sdk,+ "responses": _responses_sdk,+ "responses_stream": _responses_stream_sdk,+ }+)+++@pytest.mark.parametrize("route", tuple(SDK_ROUTES))+def test_sdk_response_names_the_server_span_that_carries_its_call_id(rig: Rig, route: str) -> None:+ marker: Final = _marker()+ answer: Final = SDK_ROUTES[route](rig, marker)+ assert answer.text == ANSWER, answer+ _assert_names_its_server_span(rig, answer.headers)+ assert rig.upstream_count(marker) == 1+++def _spend_rows(response_id: str) -> int:+ return len(read_rows('SELECT request_id FROM "LiteLLM_SpendLogs" WHERE request_id=%s', (response_id,)))+++@pytest.mark.parametrize("stream", (False, True), ids=("plain", "stream"))+def test_a_response_cache_hit_names_its_own_server_span(rig: Rig, stream: bool) -> None:+ marker: Final = _marker()+ body: Final[dict[str, JsonValue]] = {+ "model": rig.model,+ "messages": [{"role": "user", "content": marker}],+ "stream": stream,+ }+ first: Final = rig.post("/v1/chat/completions", body)+ assert first.status_code == 200, first.text+ first_span: Final = _assert_names_its_server_span(rig, first.headers)+ eventually(lambda: _spend_rows(f"chatcmpl-{marker}"), lambda rows: rows == 1, seconds=70)+ second: Final = rig.post("/v1/chat/completions", body)+ assert second.status_code == 200, second.text+ assert f"chatcmpl-{marker}" in second.text, second.text+ second_span: Final = _assert_names_its_server_span(rig, second.headers)+ assert second_span.span_id != first_span.span_id+ assert second.headers["x-litellm-call-id"] != first.headers["x-litellm-call-id"]+ assert rig.upstream_count(marker) == 1+++def test_a_sampled_caller_traceparent_is_continued_in_the_header(rig: Rig) -> None:+ trace_id: Final = _trace_id()+ parent: Final = _span_id()+ response: Final = rig.chat(_marker(), headers=(("traceparent", f"00-{trace_id}-{parent}-01"),))+ assert response.status_code == 200, response.text+ span: Final = _assert_names_its_server_span(rig, response.headers)+ assert (span.trace_id, span.parent_span_id) == (trace_id, parent), span+++def test_an_unsampled_caller_traceparent_gets_no_header(rig: Rig) -> None:+ response: Final = rig.chat(_marker(), headers=(("traceparent", f"00-{_trace_id()}-{_span_id()}-00"),))+ assert response.status_code == 200, response.text+ assert response.headers.get_list("traceresponse") == []+++@pytest.mark.parametrize(+ "traceparent",+ ("", "12345", "00-" + "f" * 5000, "not-a-traceparent", f"00-{'5' * 32}-{'6' * 16}-01, 00-{'7' * 32}-{'8' * 16}-01"),+ ids=("empty", "numeric", "five_kilobytes", "garbage", "list"),+)+def test_a_malformed_caller_traceparent_starts_a_fresh_trace_in_the_header(rig: Rig, traceparent: str) -> None:+ response: Final = rig.chat(_marker(), headers=(("traceparent", traceparent),))+ assert response.status_code == 200, response.text+ span: Final = _assert_names_its_server_span(rig, response.headers)+ assert span.parent_span_id == "", span+ assert span.trace_id not in traceparent, span+++def test_a_traceparent_sent_twice_is_continued_once(rig: Rig) -> None:+ trace_id: Final = _trace_id()+ parent: Final = _span_id()+ value: Final = f"00-{trace_id}-{parent}-01"+ response: Final = rig.chat(_marker(), headers=(("traceparent", value), ("traceparent", value)))+ assert response.status_code == 200, response.text+ span: Final = _assert_names_its_server_span(rig, response.headers)+ assert (span.trace_id, span.parent_span_id) == (trace_id, parent), span+++def _error_bad_key(rig: Rig) -> httpx.Response:+ return rig.proxy.client.post(+ "/v1/chat/completions",+ json={"model": rig.model, "messages": [{"role": "user", "content": _marker()}]},+ headers={"Authorization": "Bearer sk-not-a-real-key"},+ )+++def _error_unknown_model(rig: Rig) -> httpx.Response:+ return rig.post(+ "/v1/chat/completions",+ {"model": "no-such-model-" + uuid.uuid4().hex, "messages": [{"role": "user", "content": "x"}]},+ )+++def _error_invalid_body(rig: Rig) -> httpx.Response:+ return rig.post("/key/generate", {"models": "not-a-list"})+++def _error_unhandled(rig: Rig) -> httpx.Response:+ return rig.proxy.client.post(+ "/v1/chat/completions",+ content=b"[1]",+ headers={"Authorization": f"Bearer {rig.proxy.key}", "content-type": "application/json"},+ )+++ERRORS: Final[Mapping[str, tuple[int, Callable[[Rig], httpx.Response]]]] = MappingProxyType(+ {+ "bad_key": (401, _error_bad_key),+ "unknown_model": (400, _error_unknown_model),+ "invalid_body": (422, _error_invalid_body),+ "unhandled": (500, _error_unhandled),+ }+)+++@pytest.mark.parametrize("error", tuple(ERRORS))+def test_an_error_response_names_its_own_server_span(rig: Rig, error: str) -> None:+ status, send = ERRORS[error]+ response: Final = send(rig)+ assert response.status_code == status, response.text+ trace_id, span_id = _named(response.headers)+ span: Final = rig.sink.span(span_id)+ assert (span.kind, span.trace_id) == (SERVER, trace_id), span+++def test_an_excluded_health_route_gets_no_header(rig: Rig) -> None:+ response: Final = rig.proxy.client.get("/health/liveliness")+ assert response.status_code == 200, response.text+ assert response.headers.get_list("traceresponse") == []+++def test_a_pass_through_without_an_upstream_header_gets_the_gateway_header(rig: Rig) -> None:+ response: Final = rig.post("/relay/none", {"marker": _marker()})+ assert response.status_code == 200, response.text+ assert response.json() == {"case": "none"}+ span: Final = _assert_names_its_server_span(rig, response.headers)+ assert span.links == (), span+++def test_a_pass_through_upstream_header_in_the_same_trace_is_relayed_unchanged_without_a_link(rig: Rig) -> None:+ trace_id: Final = _trace_id()+ response: Final = rig.post(+ "/relay/continued", {"marker": _marker()}, (("traceparent", f"00-{trace_id}-{_span_id()}-01"),)+ )+ assert response.status_code == 200, response.text+ relayed: Final = response.headers.get_list("traceresponse")+ assert len(relayed) == 1 and relayed[0].startswith(f"00-{trace_id}-"), relayed+ span: Final = rig.sink.server_span_for_call(response.headers["x-litellm-call-id"])+ assert span.trace_id == trace_id, span+ assert span.span_id not in relayed[0], (span, relayed)+ assert span.links == (), span+++def test_a_pass_through_upstream_that_restarted_the_trace_is_relayed_and_linked(rig: Rig) -> None:+ upstream_trace: Final = _trace_id()+ upstream_span: Final = _span_id()+ response: Final = rig.proxy.client.post(+ "/relay/restarted",+ params={"trace": upstream_trace, "span": upstream_span},+ json={"marker": _marker()},+ headers={"Authorization": f"Bearer {rig.proxy.key}"},+ )+ assert response.status_code == 200, response.text+ assert response.headers.get_list("traceresponse") == [f"00-{upstream_trace}-{upstream_span}-01"]+ span: Final = rig.sink.server_span_for_call(response.headers["x-litellm-call-id"])+ assert span.trace_id != upstream_trace, span+ assert span.links == ((upstream_trace, upstream_span),), span+++@pytest.mark.parametrize("case", tuple(HOSTILE_UPSTREAM_VALUES))+def test_a_hostile_pass_through_upstream_header_is_relayed_once_unchanged_and_never_linked(rig: Rig, case: str) -> None:+ response: Final = rig.post(f"/relay/{case}", {"marker": _marker()})+ assert response.status_code == 200, response.text+ assert response.headers.get_list("traceresponse") == [HOSTILE_UPSTREAM_VALUES[case]]+ span: Final = rig.sink.server_span_for_call(response.headers["x-litellm-call-id"])+ assert span.links == (), span+++@pytest.mark.parametrize("status", (200, 404, 500))+def test_a_pass_through_upstream_call_id_never_replaces_the_gateway_call_id(rig: Rig, status: int) -> None:+ response: Final = rig.proxy.client.post(+ "/relay/callid",+ params={"status": str(status)},+ json={"marker": _marker()},+ headers={"Authorization": f"Bearer {rig.proxy.key}"},+ )+ assert response.status_code == status, response.text+ call_ids: Final = response.headers.get_list("x-litellm-call-id")+ assert len(call_ids) == 1 and call_ids[0] != UPSTREAM_CALL_ID, call_ids+ _assert_names_its_server_span(rig, response.headers)+ assert rig.sink.server_spans_for_call(UPSTREAM_CALL_ID) == ()+++def _exposed(response: httpx.Response) -> frozenset[str]:+ listed: Final = response.headers.get("access-control-expose-headers", "")+ return frozenset(name.strip().lower() for name in listed.split(",") if name.strip())+++def test_a_browser_origin_can_read_the_header(rig: Rig) -> None:+ response: Final = rig.proxy.request("GET", "/v1/models", headers={"Origin": "http://dashboard.example"})+ assert response.status_code == 200, response.text+ assert "traceresponse" in _exposed(response), response.headers+ _named(response.headers)+++def test_with_the_v2_gate_off_no_response_carries_or_exposes_the_header(control: Rig) -> None:+ marker: Final = _marker()+ chat: Final = control.chat(marker)+ assert chat.status_code == 200, chat.text+ assert chat.headers.get_list("traceresponse") == []+ assert control.upstream_count(marker) == 1+ models: Final = control.proxy.request("GET", "/v1/models", headers={"Origin": "http://dashboard.example"})+ assert models.status_code == 200, models.text+ assert _exposed(models), models.headers+ assert "traceresponse" not in _exposed(models), models.headers+ assert models.headers.get_list("traceresponse") == []+++def _raw_send(rig: Rig, index: int) -> httpx.Response:+ marker: Final = _marker()+ stream: Final = index % 2 == 0+ route: Final = index % 3+ if route == 0:+ return rig.post(+ "/v1/chat/completions",+ {"model": rig.model, "messages": [{"role": "user", "content": marker}], "stream": stream},+ )+ if route == 1:+ return rig.post("/v1/responses", {"model": rig.model, "input": marker, "stream": stream})+ return rig.post(+ "/v1/messages",+ {"model": rig.model, "max_tokens": 16, "messages": [{"role": "user", "content": marker}], "stream": stream},+ )+++def _burst(rig: Rig, count: int) -> tuple[httpx.Response, ...]:+ with ThreadPoolExecutor(max_workers=8) as pool:+ return tuple(pool.map(lambda index: _raw_send(rig, index), range(count)))+++def _assert_each_names_its_own_span(rig: Rig, responses: Sequence[httpx.Response]) -> None:+ assert [response.status_code for response in responses] == [200] * len(responses), [+ response.text for response in responses+ ]+ spans: Final = tuple(_assert_names_its_server_span(rig, response.headers) for response in responses)+ assert len({span.span_id for span in spans}) == len(responses), spans+ assert len({span.trace_id for span in spans}) == len(responses), spans+++def test_concurrent_mixed_requests_each_name_their_own_server_span(rig: Rig) -> None:+ _assert_each_names_its_own_span(rig, _burst(rig, 24))+++def test_a_sink_outage_keeps_every_header_and_the_spans_land_after_recovery(rig: Rig) -> None:+ rig.sink.outage.set()+ try:+ responses: Final = _burst(rig, 20)+ assert all(len(TRACERESPONSE.findall(response.headers.get("traceresponse", ""))) == 1 for response in responses)+ eventually(lambda: any(True for _ in rig.sink.wire.drain()), lambda seen: seen, seconds=30)+ finally:+ rig.sink.outage.clear()+ _assert_each_names_its_own_span(rig, responses)+++def _fresh_chat(rig: Rig) -> httpx.Response:+ with httpx.Client(base_url=rig.base_url(), timeout=30, trust_env=False) as client:+ return client.post(+ "/v1/chat/completions",+ json={+ "model": rig.model,+ "messages": [{"role": "user", "content": _marker()}],+ "cache": {"no-cache": True},+ },+ headers={"Authorization": f"Bearer {rig.proxy.key}"},+ )+++@pytest.mark.timeout(2 * graceful_stop_seconds() + 120)+def test_replacement_workers_still_name_their_server_spans(factory: Callable[[bool], RigFactory]) -> None:+ started: Final = factory(True).start()+ fresh: Final = next(started)+ try:+ originals: Final = eventually(fresh.process.worker_pids, lambda found: len(found) == 2, seconds=30)+ replacements: Final = frozenset(fresh.process.kill_worker(pid) for pid in sorted(originals))+ assert not replacements & originals, (originals, replacements)+ assert fresh.process.process.poll() is None, "Proxy root exited after its workers were replaced"+ _assert_each_names_its_own_span(fresh, tuple(_fresh_chat(fresh) for _ in range(8)))+ finally:+ started.close()tests/unit/integrations/otel/test_otel_v2_mount.pyindex e2007486a40..3a27c3ec4e4 100644@@ -5,6 +5,7 @@ surface and the server-span + shared-provider behavior it produces. """ +import contextvars from datetime import datetime, timezone import pytest@@ -14,15 +15,28 @@ pytest.importorskip("opentelemetry") pytest.importorskip("opentelemetry.instrumentation.fastapi") fastapi = pytest.importorskip("fastapi") +from fastapi.middleware.cors import CORSMiddleware # noqa: E402+from fastapi.responses import StreamingResponse # noqa: E402 from fastapi.testclient import TestClient # noqa: E402+from starlette.routing import Route # noqa: E402+import httpx # noqa: E402 from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor # noqa: E402+from opentelemetry.instrumentation.propagators import ( # noqa: E402+ ResponsePropagator,+ get_global_response_propagator,+ set_global_response_propagator,+)+from opentelemetry.sdk.trace import TracerProvider # noqa: E402 from opentelemetry.sdk.trace.export import SimpleSpanProcessor # noqa: E402 from opentelemetry.sdk.trace.export.in_memory_span_exporter import ( # noqa: E402 InMemorySpanExporter, ) from opentelemetry import trace # noqa: E402+from opentelemetry.sdk.trace.sampling import ALWAYS_OFF # noqa: E402 from opentelemetry.trace import SpanKind # noqa: E402 +from litellm.constants import LITELLM_UI_ALLOW_HEADERS # noqa: E402+ from litellm.integrations.otel.model.config import ( # noqa: E402 OpenTelemetryV2Config, is_otel_v2_enabled,@@ -32,6 +46,11 @@ from litellm.integrations.otel.mount import ( # noqa: E402 PASSTHROUGH_PREFIXES, _passthrough_span_name_hook, instrument_fastapi_app,+ trace_response_expose_headers,+ trace_response_headers,+)+from litellm.proxy.pass_through_endpoints.pass_through_endpoints import ( # noqa: E402+ HttpPassThroughEndpointHelpers, ) from litellm.integrations.otel.plumbing.context import ( # noqa: E402 request_root_http_route,@@ -52,6 +71,13 @@ def _reset_request_root_span(): _otel_context._mcp_message_transport_span.set(None) +@pytest.fixture(autouse=True)+def _reset_response_propagator():+ set_global_response_propagator(None)+ yield+ set_global_response_propagator(None)++ @pytest.fixture(autouse=True) def _clear_otel_v2_flag_cache(): is_otel_v2_enabled.cache_clear()@@ -227,6 +253,20 @@ def test_blank_route_on_the_server_span_is_omitted(): assert request_root_http_route() is None +def test_span_hook_does_not_break_requests_when_the_span_provider_fails():+ class _FailingSpan:+ def is_recording(self) -> bool:+ raise RuntimeError("span provider unavailable")++ def update_name(self, name: str) -> None:+ raise AssertionError(name)++ def set_attribute(self, key: str, value: str) -> None:+ raise AssertionError((key, value))++ assert _passthrough_span_name_hook(_FailingSpan(), {}) is None++ def test_known_passthrough_prefixes_present(): """Guard the prefix set against accidental edits.""" assert {"openai", "anthropic", "vertex_ai", "bedrock"} <= PASSTHROUGH_PREFIXES@@ -254,3 +294,255 @@ def test_instrument_fastapi_app_attaches_when_gate_on(monkeypatch): assert getattr(app, "_is_instrumented_by_opentelemetry", False) is True finally: FastAPIInstrumentor.uninstrument_app(app)+++_UPSTREAM_TRACERESPONSE = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"+++class _TupleHeadersApp:+ async def __call__(self, scope, receive, send):+ await send({"type": "http.response.start", "status": 200, "headers": ((b"x-litellm-call-id", b"call-tuple"),)})+ await send({"type": "http.response.body", "body": b"ok"})+++def _traced_app(monkeypatch, tracer_provider):+ from litellm.proxy.proxy_server import otel_unhandled_exception_handler++ monkeypatch.setenv("LITELLM_OTEL_V2", "1")+ is_otel_v2_enabled.cache_clear()+ app = fastapi.FastAPI()+ app.add_exception_handler(Exception, otel_unhandled_exception_handler)++ @app.get("/success")+ async def success():+ return fastapi.Response(headers={"x-litellm-call-id": "call-success"})++ @app.get("/failure")+ async def failure():+ raise fastapi.HTTPException(status_code=400, headers={"x-litellm-call-id": "call-failure"})++ async def chunks():+ yield b"data: first\n\n"+ yield b"data: [DONE]\n\n"++ @app.get("/stream")+ async def stream():+ return StreamingResponse(chunks(), headers={"x-litellm-call-id": "call-stream"})++ async def relay_upstream():+ upstream = httpx.Headers({"traceresponse": _UPSTREAM_TRACERESPONSE})+ return fastapi.Response(headers=HttpPassThroughEndpointHelpers.get_response_headers(upstream, "call-relayed"))++ app.add_api_route("/relayed", relay_upstream)+ app.add_api_route("/relayed/health", relay_upstream)++ @app.get("/relayed-untraced")+ async def relay_untraced_upstream():+ upstream = httpx.Headers({"access-control-expose-headers": "x-upstream-request-id"})+ return fastapi.Response(headers=HttpPassThroughEndpointHelpers.get_response_headers(upstream, "call-untraced"))++ app.router.routes.append(Route("/tuple-headers", _TupleHeadersApp()))++ @app.get("/crash")+ async def crash(request: fastapi.Request):+ request.state.parent_otel_span = trace.get_current_span()+ raise RuntimeError("unhandled")++ @app.websocket("/ws")+ async def websocket_route(websocket: fastapi.WebSocket):+ await websocket.accept()+ await websocket.send_text("ok")+ await websocket.close()++ instrument_fastapi_app(app, tracer_provider=tracer_provider)+ return app+++def _exporting_provider():+ logger = OpenTelemetryV2(config=OpenTelemetryV2Config(exporter="in_memory"))+ exporter = InMemorySpanExporter()+ logger._tracer_provider.add_span_processor(SimpleSpanProcessor(exporter))+ return logger._tracer_provider, exporter+++def _server_spans_by_route(exporter):+ return {(s.attributes or {})["http.route"]: s for s in exporter.get_finished_spans() if s.kind is SpanKind.SERVER}+++def test_every_traced_response_names_its_own_server_span(monkeypatch):+ provider, exporter = _exporting_provider()+ client = TestClient(_traced_app(monkeypatch, provider), raise_server_exceptions=False)++ responses = {path: client.get(path) for path in ("/success", "/failure", "/stream", "/tuple-headers", "/crash")}++ server_spans = _server_spans_by_route(exporter)+ assert responses["/failure"].status_code == 400+ assert responses["/tuple-headers"].status_code == 200+ assert responses["/crash"].status_code == 500+ assert responses["/stream"].text == "data: first\n\ndata: [DONE]\n\n"+ for path, response in responses.items():+ context = server_spans[path].context+ assert response.headers.get_list("traceresponse") == [f"00-{context.trace_id:032x}-{context.span_id:016x}-01"]+++@pytest.mark.parametrize("path", ["/relayed", "/relayed/health"])+def test_a_relayed_upstream_traceresponse_is_returned_once_and_unchanged(monkeypatch, path):+ provider, _ = _exporting_provider()++ response = TestClient(_traced_app(monkeypatch, provider)).get(path)++ assert response.headers.get_list("traceresponse") == [_UPSTREAM_TRACERESPONSE]+++@pytest.mark.parametrize(+ ("caller_headers", "linked"),+ [({}, True), ({"traceparent": f"00-{_UPSTREAM_TRACERESPONSE.split('-')[1]}-00f067aa0ba902b7-01"}, False)],+)+def test_a_relayed_traceresponse_from_a_restarted_trace_is_linked(monkeypatch, caller_headers, linked):+ provider, exporter = _exporting_provider()++ TestClient(_traced_app(monkeypatch, provider)).get("/relayed", headers=caller_headers)++ links = _server_spans_by_route(exporter)["/relayed"].links+ upstream_ids = [f"00-{link.context.trace_id:032x}-{link.context.span_id:016x}-01" for link in links]+ assert upstream_ids == ([_UPSTREAM_TRACERESPONSE] if linked else [])+++def test_traceresponse_leaves_access_control_expose_headers_to_cors(monkeypatch):+ provider, _ = _exporting_provider()+ client = TestClient(_traced_app(monkeypatch, provider), raise_server_exceptions=False)++ responses = [client.get(path) for path in ("/success", "/crash", "/relayed-untraced")]++ assert all(response.headers.get_list("traceresponse") for response in responses)+ exposed = [response.headers.get_list("access-control-expose-headers") for response in responses]+ assert exposed == [[], [], ["x-upstream-request-id"]]+++def test_server_span_carries_the_call_id_its_response_returns(monkeypatch):+ provider, exporter = _exporting_provider()+ client = TestClient(_traced_app(monkeypatch, provider))++ responses = {path: client.get(path) for path in ("/success", "/failure", "/stream", "/relayed", "/tuple-headers")}++ server_spans = _server_spans_by_route(exporter)+ for path, response in responses.items():+ assert (server_spans[path].attributes or {})["litellm.call_id"] == response.headers["x-litellm-call-id"]+++def test_traceresponse_keeps_the_caller_trace(monkeypatch):+ provider, _ = _exporting_provider()+ caller_trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"++ response = TestClient(_traced_app(monkeypatch, provider)).get(+ "/success", headers={"traceparent": f"00-{caller_trace_id}-00f067aa0ba902b7-01"}+ )++ assert response.headers["traceresponse"].split("-")[1] == caller_trace_id+++def test_unsampled_requests_get_no_traceresponse(monkeypatch):+ client = TestClient(_traced_app(monkeypatch, TracerProvider(sampler=ALWAYS_OFF)), raise_server_exceptions=False)++ responses = [client.get(path) for path in ("/success", "/crash")]++ assert [response.headers.get_list("traceresponse") for response in responses] == [[], []]+++def test_websocket_handshakes_carry_no_traceresponse(monkeypatch):+ provider, _ = _exporting_provider()++ with TestClient(_traced_app(monkeypatch, provider)).websocket_connect("/ws") as websocket:+ assert websocket.receive_text() == "ok"+ handshake_header_names = [name for name, _ in websocket.extra_headers or ()]++ assert b"traceresponse" not in handshake_header_names+++def test_gate_off_responses_carry_no_traceresponse(monkeypatch):+ monkeypatch.delenv("LITELLM_OTEL_V2", raising=False)+ app = fastapi.FastAPI()++ @app.get("/ping")+ def ping():+ return {"ok": True}++ instrument_fastapi_app(app)++ assert "traceresponse" not in TestClient(app).get("/ping").headers+++def test_failed_instrumentation_registers_no_response_propagator(monkeypatch):+ monkeypatch.setenv("LITELLM_OTEL_V2", "1")+ app = fastapi.FastAPI()++ with TestClient(app):+ instrument_fastapi_app(app)++ assert get_global_response_propagator() is None+++def test_an_operator_response_propagator_is_kept(monkeypatch):+ class _OperatorPropagator(ResponsePropagator):+ def inject(self, carrier, context=None, setter=None):+ setter.set(carrier, "x-operator-trace", "kept")++ set_global_response_propagator(_OperatorPropagator())+ provider, _ = _exporting_provider()++ response = TestClient(_traced_app(monkeypatch, provider)).get("/relayed")++ assert response.headers["x-operator-trace"] == "kept"+ assert response.headers["traceresponse"] == _UPSTREAM_TRACERESPONSE+++def test_browsers_can_read_traceresponse_through_cors(monkeypatch):+ provider, _ = _exporting_provider()+ app = _traced_app(monkeypatch, provider)+ app.add_middleware(+ CORSMiddleware, allow_origins=["*"], expose_headers=[*LITELLM_UI_ALLOW_HEADERS, *trace_response_expose_headers()]+ )++ response = TestClient(app).get("/success", headers={"origin": "https://ui.example.com"})++ exposed = {name.strip() for name in response.headers["access-control-expose-headers"].split(",")}+ assert "traceresponse" in exposed+ assert response.headers["traceresponse"]+++def test_cors_leaves_traceresponse_unexposed_with_the_gate_off(monkeypatch):+ monkeypatch.delenv("LITELLM_OTEL_V2", raising=False)+ is_otel_v2_enabled.cache_clear()++ assert "traceresponse" not in [*LITELLM_UI_ALLOW_HEADERS, *trace_response_expose_headers()]+++def test_other_instrumented_apps_in_the_process_get_no_traceresponse(monkeypatch):+ provider, _ = _exporting_provider()+ litellm_client = TestClient(_traced_app(monkeypatch, provider))+ operator_app = fastapi.FastAPI()++ @operator_app.get("/operator")+ async def operator_route():+ return {"ok": True}++ FastAPIInstrumentor.instrument_app(operator_app, tracer_provider=provider)+ try:+ operator_response = TestClient(operator_app).get("/operator")+ litellm_response = litellm_client.get("/success")+ finally:+ FastAPIInstrumentor.uninstrument_app(operator_app)++ assert "traceresponse" not in operator_response.headers+ assert litellm_response.headers["traceresponse"]+++def test_the_unhandled_error_helper_names_the_span_it_is_given_in_any_context(monkeypatch):+ provider, _ = _exporting_provider()+ _traced_app(monkeypatch, provider)+ span = provider.get_tracer(__name__).start_span("POST /v1/chat/completions", kind=SpanKind.SERVER)+ context = span.get_span_context()++ headers = contextvars.Context().run(trace_response_headers, span)++ assert headers["traceresponse"] == f"00-{context.trace_id:032x}-{context.span_id:016x}-01"tests/unit/proxy/test_proxy_server_endpoints_and_startup.pyindex b981e8282bf..09646baa07f 100644@@ -102,7 +102,7 @@ def test_cors_exposes_cache_key_header_to_browser_js(): from litellm.constants import LITELLM_UI_ALLOW_HEADERS cors_middleware = next(m for m in app.user_middleware if m.cls is CORSMiddleware)- assert cors_middleware.kwargs["expose_headers"] is LITELLM_UI_ALLOW_HEADERS+ assert set(LITELLM_UI_ALLOW_HEADERS) <= set(cors_middleware.kwargs["expose_headers"]) assert "x-litellm-cache-key" in cors_middleware.kwargs["expose_headers"]