Backtestslitellmrun-20261018-182052#42078

Issue #42078 Merged fix PR #42079 Run 18 Oct 2026 · run-20261018-182052 BerriAI/litellm

[Feature]: Expose OpenTelemetry trace IDs on LiteLLM Proxy responses

Better

Our patch was judged better than the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

The issue asks for a specific header, `x-litellm-trace-id`, carrying the root trace ID as a 32-character lowercase W3C trace ID. Patch B implements exactly that: an ASGI middleware mounted inside the FastAPI instrumentor adds the header on the first `http.response.start` (covering JSON, streaming initial headers, and all exception-handler responses), stamps `litellm.call_id` on the server span from the response's `x-litellm-call-id`, preserves inbound `traceparent` (the server span already continues it), omits the header when the span is not recording or the gate is off, and handles the Starlette `ServerErrorMiddleware` 500 path by stashing the id on `request.state` and copying it in the unhandled-exception handler. Its unit tests exercise every scenario the issue enumerates (200, stream, 400/401/403/429/502/500, 422, traceparent round-trip, call_id on span, excluded route, no-op provider, gate off) and use the proxy's real exception handlers. Patch A instead emits a `traceresponse` header in `00-<trace>-<span>-<flags>` format and never adds `x-litellm-trace-id` at all. However well-engineered its propagator, upstream-relay linking, and CORS exposure are, it does not deliver the header the issue requests, so a client following the issue's user flow still cannot find the trace ID under the documented name. The independent tests confirm this: all ten trace-id tests fail on Patch A exactly as on base and pass on Patch B. Patch A also widens scope considerably (upstream traceresponse relaying, span links, CORS changes, a large integration test suite), which adds risk without addressing the ask. Patch B is the correct, scoped fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

17m 17s

Wall-clock time in the container

Full test suite

Not run

The harness did not reach the test stage

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

380vs 1,192 merged

Changed lines, ours vs merged

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate

The Feature

Add x-litellm-trace-id to LiteLLM Proxy HTTP responses whenever the request has a recording OpenTelemetry server span. Its value should be the root trace ID as a 32-character lowercase W3C trace ID

The response already includes x-litellm-call-id, but a gateway user who receives that ID cannot reliably open the exact trace that handled the request. The proposed response header would make a client response, the request trace, and the existing call ID directly navigable across tracing and logging backends

For any response that has both values, the root server span should also carry the same litellm.call_id as x-litellm-call-id. This establishes an explicit relationship between the response, root span, and downstream LLM spans

Expected behavior:

  1. Return the header on successful JSON responses and the initial headers of streaming responses
  2. Return the header on validation, authentication, authorization, rate-limit, provider, and unhandled-error responses
  3. Preserve incoming traceparent propagation and return the resulting request trace ID
  4. Omit the header when OpenTelemetry is disabled or no recording request span exists
  5. Keep x-litellm-call-id behavior unchanged

This is a LiteLLM Proxy / AI Gateway feature only. It does not change the Python SDK or WebSocket behavior

Related: #31864 covers request identity in JSON logs. This request covers the gateway HTTP response and its OpenTelemetry root span

User Flow

Before this feature (today): a gateway developer investigating a failed model request reaches the response's call ID but cannot navigate directly to the trace that handled it

  1. They send POST https://gateway.example.com/v1/chat/completions with their model and messages
  2. The gateway returns 400 Bad Request or a provider error with x-litellm-call-id: 7ac48d62-40a1-44e6-8b50-19da641835a5
  3. They search the observability backend for that call ID and manually inspect matching LLM spans, logs, or request time ranges to identify the parent request trace
  4. Their workaround is to correlate timestamps and child-span fields, which is slow and ambiguous when several retries or requests are in flight

After this feature (ideal user flow): the same developer can open the exact request trace from the gateway response

  1. They send the same POST https://gateway.example.com/v1/chat/completions with the same model and messages
  2. The gateway returns the same status and x-litellm-call-id, plus x-litellm-trace-id: af353cab815b907430a117c9b97b4d0e
  3. They paste af353cab815b907430a117c9b97b4d0e into their observability backend and open the one server trace for that gateway request
  4. They use the call ID visible on that trace to correlate request logs and downstream model spans without matching timestamps by hand

How far you got

Config / setup the proxy ran with:

model_list: []
general_settings:
  master_key: sk-local-trace-repro
litellm_settings:
  telemetry: false

Version or commit: 5417abd58672 on the current main branch

Environment variables: LITELLM_OTEL_V2=true, OTEL_EXPORTER=console, and OTEL_SERVICE_NAME=litellm-trace-header-repro

Commands and their full output, up to the step that dead-ends:

$ LITELLM_OTEL_V2=true OTEL_EXPORTER=console OTEL_SERVICE_NAME=litellm-trace-header-repro uv run litellm --config /private/tmp/litellm-trace-header-repro.yaml --host 127.0.0.1 --port 4000
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:4000 (Press CTRL+C to quit)

$ curl --max-time 15 -i -X POST http://127.0.0.1:4000/v1/chat/completions -H 'Authorization: Bearer sk-local-trace-repro' -H 'Content-Type: application/json' --data '{"model":"missing-model","messages":[{"role":"user","content":"hello"}]}'
HTTP/1.1 400 Bad Request
date: Sun, 20 Sep 2026 02:32:23 GMT
x-litellm-call-id: 7ac48d62-40a1-44e6-8b50-19da641835a5
x-litellm-response-cost: 0
x-litellm-key-spend: 0.0
content-length: 371
content-type: application/json
x-frame-options: DENY
content-security-policy: frame-ancestors 'none'
x-content-type-options: nosniff

{"error":{"message":"/chat/completions: Invalid model name passed in model=missing-model. Call `/v1/models` to view available models for your key.","type":"invalid_request_error","param":null,"code":"400","provider_specific_fields":{"error":"/chat/completions: Invalid model name passed in model=missing-model. Call `/v1/models` to view available models for your key."}}}

The console exporter emitted this server span for the same request:

{
    "name": "POST /v1/chat/completions",
    "context": {
        "trace_id": "0xaf353cab815b907430a117c9b97b4d0e",
        "span_id": "0x20cb45e2212a998e"
    },
    "kind": "SpanKind.SERVER",
    "status": {
        "status_code": "ERROR"
    }
}

What stopped me there: the live proxy created a server trace with ID af353cab815b907430a117c9b97b4d0e and returned a different call ID, but the HTTP response did not expose the trace ID. A client receiving the error therefore cannot navigate straight to the trace

Which part of LiteLLM is this about?

Logging: callbacks, Langfuse, Datadog, OTel, Prometheus, alerting

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

The two patches

Our patch+379 −1
litellm/integrations/otel/README.mdindex 340bd71c98..88ba6d65f8 100644@@ -224,6 +224,12 @@ becomes the global, so server spans export to that backend too.    `/metrics` also drops the `/model/metrics` admin-analytics spans. Set    `OTEL_PYTHON_FASTAPI_EXCLUDED_URLS` to override the whole set (e.g. `""` to trace    everything, or your own comma-separated path list).+   Just inside the instrumentor it mounts `mount._TraceIdResponseHeaderMiddleware`,+   which adds `x-litellm-trace-id` (the server span's 32-char W3C trace id) to every+   HTTP response of a request with a recording server span. It also stamps the+   response's `x-litellm-call-id` on that span as `litellm.call_id`. The bare+   `Exception` handler's 500/503 is sent from outside the middleware, so it copies+   the header on itself via `mount.stamp_trace_id_header`. 2. **Startup** (`proxy_server.proxy_startup_event`): after the config (and    callbacks) is loaded, the already-registered preset `OpenTelemetryV2` logger    is reused — or a generic one reading `OTEL_*` envs is built when no preset islitellm/integrations/otel/mount.pyindex 9340f6e9e1..2a97d87b99 100644@@ -16,9 +16,22 @@ from typing import TYPE_CHECKING, Final, Protocol  from litellm._logging import verbose_logger from litellm.integrations.otel.model.config import is_otel_v2_enabled+from litellm.integrations.otel.model.semconv import LiteLLM  if TYPE_CHECKING:     from fastapi import FastAPI+    from starlette.requests import Request+    from starlette.responses import Response+    from starlette.types import ASGIApp, Message, Receive, Scope, Send++# Response header carrying the request's W3C trace id (32 lowercase hex chars), so+# a client can open the exact trace that served its request.+TRACE_ID_HEADER: Final = "x-litellm-trace-id"+_CALL_ID_HEADER: Final = "x-litellm-call-id"+# ``request.state`` key the trace id is stashed under, for the responses Starlette+# sends from ``ServerErrorMiddleware`` (the bare ``Exception`` handler), which sits+# outside every user middleware and so never passes through the send wrapper.+_TRACE_ID_STATE_KEY: Final = "litellm_trace_id"  # Routes excluded from server-span tracing by default: high-frequency pollers and # static UI/docs assets, none of which are LLM traffic. Entries are substring-matched@@ -99,6 +112,68 @@ def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: dict) ->         pass  +class _TraceIdResponseHeaderMiddleware:+    """Expose the request's trace id as ``x-litellm-trace-id`` on HTTP responses.++    Mounted directly inside the instrumentor's ``OpenTelemetryMiddleware`` so the+    current span on entry is the request's server span, already parented on any+    inbound ``traceparent``. The id is added to the first ``http.response.start``+    message, which covers JSON bodies, error responses raised through the exception+    middleware and the initial headers of streaming responses alike. The same+    message's ``x-litellm-call-id`` is stamped on the server span as+    ``litellm.call_id`` so the response, the root span and the LLM spans share it.+    Requests with no recording span (excluded routes, provider not yet published)+    get no header.+    """++    def __init__(self, app: "ASGIApp") -> None:+        self.app = app++    async def __call__(self, scope: "Scope", receive: "Receive", send: "Send") -> None:+        if scope["type"] != "http":+            await self.app(scope, receive, send)+            return++        from opentelemetry.trace import format_trace_id, get_current_span+        from starlette.datastructures import MutableHeaders+        from starlette.requests import Request++        server_span: Final = get_current_span()+        span_context: Final = server_span.get_span_context()+        if not server_span.is_recording() or not span_context.is_valid:+            await self.app(scope, receive, send)+            return++        trace_id: Final = format_trace_id(span_context.trace_id)+        setattr(Request(scope).state, _TRACE_ID_STATE_KEY, trace_id)++        async def send_with_trace_id(message: "Message") -> None:+            if message["type"] == "http.response.start":+                headers: Final = MutableHeaders(scope=message)+                # Assignment replaces any value already present (e.g. an upstream+                # proxy's, copied through by a pass-through route).+                headers[TRACE_ID_HEADER] = trace_id+                call_id: Final = headers.get(_CALL_ID_HEADER)+                if call_id and server_span.is_recording():+                    server_span.set_attribute(LiteLLM.CALL_ID, call_id)+            await send(message)++        await self.app(scope, receive, send_with_trace_id)+++def stamp_trace_id_header(request: "Request", response: "Response") -> "Response":+    """Add ``x-litellm-trace-id`` to a response sent outside the header middleware.++    Starlette sends the bare ``Exception`` handler's response from+    ``ServerErrorMiddleware``, outside every user middleware, so the handler copies+    the trace id the middleware recorded on ``request.state`` itself.+    """+    trace_id: Final[object] = getattr(request.state, _TRACE_ID_STATE_KEY, None)+    if isinstance(trace_id, str):+        response.headers[TRACE_ID_HEADER] = trace_id+    return response++ def instrument_fastapi_app(app: "FastAPI") -> None:     """Attach OTel server-span instrumentation to the proxy FastAPI app. @@ -123,6 +198,10 @@ def instrument_fastapi_app(app: "FastAPI") -> None:         # package is absent, even with the gate off.         from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor +        # Added before the instrumentor so it nests inside ``OpenTelemetryMiddleware``+        # (``add_middleware`` makes the latest addition outermost) and sees the+        # server span as the current span.+        app.add_middleware(_TraceIdResponseHeaderMiddleware)         excluded_urls: Final = (             os.environ.get("OTEL_PYTHON_FASTAPI_EXCLUDED_URLS")             if "OTEL_PYTHON_FASTAPI_EXCLUDED_URLS" in os.environlitellm/proxy/proxy_server.pyindex b51df22d07..ef0cb805c2 100644@@ -1906,7 +1906,7 @@ app = FastAPI( ## middleware after an application has started". See ## ``litellm.integrations.otel.mount`` for the full rationale; the call is a safe ## no-op when the gate is off or the instrumentation package is unavailable.-from litellm.integrations.otel.mount import instrument_fastapi_app+from litellm.integrations.otel.mount import instrument_fastapi_app, stamp_trace_id_header  instrument_fastapi_app(app) @@ -2223,6 +2223,12 @@ async def otel_request_validation_exception_handler(request: Request, exc: Reque async def otel_unhandled_exception_handler(request: Request, exc: Exception):     if isinstance(exc, (ProxyException, HTTPException, RequestValidationError)):         raise exc+    # Starlette sends this handler's response from ServerErrorMiddleware, outside the+    # trace-id header middleware, so the header is copied on here.+    return stamp_trace_id_header(request, await _unhandled_exception_response(request, exc))+++async def _unhandled_exception_response(request: Request, exc: Exception) -> Response:     if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(exc):         verbose_proxy_logger.warning("Database unavailable during request: %s", type(exc).__name__)         return await openai_exception_handler(tests/unit/integrations/otel/test_otel_v2_trace_id_header.pynew file mode 100644index 0000000000..303b6a2d6b@@ -0,0 +1,287 @@+"""``x-litellm-trace-id`` on proxy HTTP responses under OTel V2.++``instrument_fastapi_app`` mounts the trace-id header middleware next to the+FastAPI instrumentor; the proxy's real exception handlers are registered on the+test app so error responses take the same path they take in the proxy.+"""++import re+from typing import Final++import pytest++pytest.importorskip("opentelemetry")+pytest.importorskip("opentelemetry.instrumentation.fastapi")++from fastapi import FastAPI, HTTPException  # noqa: E402+from fastapi.exceptions import RequestValidationError  # noqa: E402+from fastapi.responses import JSONResponse, StreamingResponse  # noqa: E402+from fastapi.testclient import TestClient  # noqa: E402+from opentelemetry import trace  # noqa: E402+from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor  # noqa: E402+from opentelemetry.sdk.trace.export import SimpleSpanProcessor  # noqa: E402+from opentelemetry.sdk.trace.export.in_memory_span_exporter import (  # noqa: E402+    InMemorySpanExporter,+)+from opentelemetry.trace import SpanKind, format_trace_id  # noqa: E402+from pydantic import BaseModel  # noqa: E402++from litellm.integrations.otel.logger import OpenTelemetryV2  # noqa: E402+from litellm.integrations.otel.model.config import (  # noqa: E402+    OpenTelemetryV2Config,+    is_otel_v2_enabled,+)+from litellm.integrations.otel.mount import TRACE_ID_HEADER, instrument_fastapi_app  # noqa: E402+from litellm.proxy._types import ProxyException  # noqa: E402+from litellm.proxy.proxy_server import (  # noqa: E402+    openai_exception_handler,+    otel_request_validation_exception_handler,+    otel_unhandled_exception_handler,+    otlp_http_exception_handler,+)+from starlette.exceptions import HTTPException as StarletteHTTPException  # noqa: E402++CALL_ID: Final = "7ac48d62-40a1-44e6-8b50-19da641835a5"+INBOUND_TRACE_ID: Final = "af353cab815b907430a117c9b97b4d0e"+_TRACE_ID_SHAPE: Final = re.compile(r"^[0-9a-f]{32}$")+++@pytest.fixture(autouse=True)+def _clear_otel_v2_flag_cache():+    is_otel_v2_enabled.cache_clear()+    yield+    is_otel_v2_enabled.cache_clear()+++class _Body(BaseModel):+    model: str+++def _proxy_error(code: int, message: str) -> ProxyException:+    return ProxyException(+        message=message,+        type="invalid_request_error",+        param=None,+        code=code,+        headers={"x-litellm-call-id": CALL_ID},+    )+++def _app() -> FastAPI:+    app = FastAPI()+    app.add_exception_handler(ProxyException, openai_exception_handler)+    app.add_exception_handler(StarletteHTTPException, otlp_http_exception_handler)+    app.add_exception_handler(RequestValidationError, otel_request_validation_exception_handler)+    app.add_exception_handler(Exception, otel_unhandled_exception_handler)++    @app.post("/v1/chat/completions")+    async def chat():+        return JSONResponse({"ok": True}, headers={"x-litellm-call-id": CALL_ID})++    @app.post("/v1/stream")+    async def stream():+        async def chunks():+            yield b"data: one\n\n"+            yield b"data: [DONE]\n\n"++        return StreamingResponse(chunks(), media_type="text/event-stream", headers={"x-litellm-call-id": CALL_ID})++    @app.post("/v1/validate")+    async def validate(body: _Body):+        return {"model": body.model}++    @app.post("/v1/bad-model")+    async def bad_model():+        raise _proxy_error(400, "Invalid model name passed in model=missing-model")++    @app.post("/v1/auth")+    async def auth():+        raise _proxy_error(401, "Authentication Error")++    @app.post("/v1/forbidden")+    async def forbidden():+        raise HTTPException(status_code=403, detail="not allowed")++    @app.post("/v1/rate-limited")+    async def rate_limited():+        raise _proxy_error(429, "Rate limit exceeded")++    @app.post("/v1/provider-error")+    async def provider_error():+        raise _proxy_error(502, "upstream provider failed")++    @app.post("/v1/crash")+    async def crash():+        raise RuntimeError("kaboom")++    @app.get("/health/readiness")+    async def health():+        return {"status": "ok"}++    return app+++@pytest.fixture+def traced(monkeypatch):+    """An instrumented app plus the exporter its server spans land in."""+    monkeypatch.setenv("LITELLM_OTEL_V2", "1")+    monkeypatch.delenv("OTEL_PYTHON_FASTAPI_EXCLUDED_URLS", raising=False)+    logger = OpenTelemetryV2(config=OpenTelemetryV2Config(exporter="in_memory"))+    exporter = InMemorySpanExporter()+    logger._tracer_provider.add_span_processor(SimpleSpanProcessor(exporter))+    # instrument_fastapi_app binds to the OTel global, which set_tracer_provider only+    # sets once per process; place it directly and let monkeypatch undo it.+    monkeypatch.setattr(trace, "_TRACER_PROVIDER", logger._tracer_provider)+    app = _app()+    instrument_fastapi_app(app)+    try:+        yield TestClient(app, raise_server_exceptions=False), exporter+    finally:+        FastAPIInstrumentor.uninstrument_app(app)+++def _server_span(exporter: InMemorySpanExporter):+    spans = [s for s in exporter.get_finished_spans() if s.kind is SpanKind.SERVER]+    assert len(spans) == 1+    return spans[0]+++def test_successful_json_response_carries_the_server_span_trace_id(traced):+    client, exporter = traced++    response = client.post("/v1/chat/completions")++    assert response.status_code == 200+    span = _server_span(exporter)+    assert response.headers[TRACE_ID_HEADER] == format_trace_id(span.context.trace_id)+    assert _TRACE_ID_SHAPE.match(response.headers[TRACE_ID_HEADER])+    assert response.headers["x-litellm-call-id"] == CALL_ID+++def test_root_server_span_carries_the_response_call_id(traced):+    client, exporter = traced++    response = client.post("/v1/chat/completions")++    assert (_server_span(exporter).attributes or {})["litellm.call_id"] == response.headers["x-litellm-call-id"]+++def test_streaming_response_initial_headers_carry_the_trace_id(traced):+    client, exporter = traced++    with client.stream("POST", "/v1/stream") as response:+        trace_id = response.headers[TRACE_ID_HEADER]+        body = b"".join(response.iter_bytes())++    assert b"[DONE]" in body+    span = _server_span(exporter)+    assert trace_id == format_trace_id(span.context.trace_id)+    assert (span.attributes or {})["litellm.call_id"] == CALL_ID+++@pytest.mark.parametrize(+    ("path", "status"),+    [+        ("/v1/bad-model", 400),+        ("/v1/auth", 401),+        ("/v1/forbidden", 403),+        ("/v1/rate-limited", 429),+        ("/v1/provider-error", 502),+        ("/v1/crash", 500),+    ],+)+def test_error_responses_carry_the_trace_id(traced, path, status):+    client, exporter = traced++    response = client.post(path)++    assert response.status_code == status+    assert response.headers[TRACE_ID_HEADER] == format_trace_id(_server_span(exporter).context.trace_id)+++def test_validation_error_response_carries_the_trace_id(traced):+    client, exporter = traced++    response = client.post("/v1/validate", json={"not_model": 1})++    assert response.status_code == 422+    assert response.headers[TRACE_ID_HEADER] == format_trace_id(_server_span(exporter).context.trace_id)+++def test_proxy_exception_keeps_call_id_and_stamps_it_on_the_server_span(traced):+    client, exporter = traced++    response = client.post("/v1/bad-model")++    assert response.headers["x-litellm-call-id"] == CALL_ID+    assert (_server_span(exporter).attributes or {})["litellm.call_id"] == CALL_ID+++def test_inbound_traceparent_is_propagated_and_returned(traced):+    client, exporter = traced++    response = client.post(+        "/v1/chat/completions",+        headers={"traceparent": f"00-{INBOUND_TRACE_ID}-20cb45e2212a998e-01"},+    )++    assert response.headers[TRACE_ID_HEADER] == INBOUND_TRACE_ID+    assert format_trace_id(_server_span(exporter).context.trace_id) == INBOUND_TRACE_ID+++def test_upstream_trace_id_header_is_replaced_not_duplicated(traced):+    client, exporter = traced+    app = client.app++    @app.post("/v1/passthrough")+    async def passthrough():+        return JSONResponse({}, headers={TRACE_ID_HEADER: "0" * 32})++    response = client.post("/v1/passthrough")++    assert response.headers.get_list(TRACE_ID_HEADER) == [format_trace_id(_server_span(exporter).context.trace_id)]+++def test_excluded_route_without_a_server_span_gets_no_trace_id(traced):+    client, exporter = traced++    response = client.get("/health/readiness")++    assert response.status_code == 200+    assert TRACE_ID_HEADER not in response.headers+    assert not [s for s in exporter.get_finished_spans() if s.kind is SpanKind.SERVER]+++def test_no_recording_span_means_no_trace_id(monkeypatch):+    """Before a real provider is published the global is a no-op provider."""+    monkeypatch.setenv("LITELLM_OTEL_V2", "1")+    monkeypatch.setattr(trace, "_TRACER_PROVIDER", trace.NoOpTracerProvider())+    app = _app()+    instrument_fastapi_app(app)+    try:+        client = TestClient(app, raise_server_exceptions=False)+        ok = client.post("/v1/chat/completions")+        crashed = client.post("/v1/crash")+    finally:+        FastAPIInstrumentor.uninstrument_app(app)++    assert TRACE_ID_HEADER not in ok.headers+    assert ok.headers["x-litellm-call-id"] == CALL_ID+    assert crashed.status_code == 500+    assert TRACE_ID_HEADER not in crashed.headers+++def test_gate_off_means_no_trace_id(monkeypatch):+    monkeypatch.delenv("LITELLM_OTEL_V2", raising=False)+    app = _app()+    instrument_fastapi_app(app)+    client = TestClient(app, raise_server_exceptions=False)++    ok = client.post("/v1/chat/completions")+    failed = client.post("/v1/bad-model")+    crashed = client.post("/v1/crash")++    assert ok.headers["x-litellm-call-id"] == CALL_ID+    assert failed.headers["x-litellm-call-id"] == CALL_ID+    for response in (ok, failed, crashed):+        assert TRACE_ID_HEADER not in response.headers 
Merged fix · PR #42079+1183 −9
litellm/integrations/otel/README.mdindex 340bd71c983..7414a451038 100644@@ -194,7 +194,10 @@ matching the FastAPI server span); only a genuine error sets `ERROR`. - **Server spans** (one per HTTP route) are created by the   `opentelemetry-instrumentation-fastapi` package. It stamps `http.*` attributes   and extracts inbound `traceparent` headers. This package does **not** create-  or modify server spans — request routes never touch spans.+  server spans. It renames passthrough server spans, sets `litellm.call_id` on+  them, and returns each sampled server span to the client in a `traceresponse`+  response header (`trace_response.py`). A relayed upstream `traceresponse` is+  kept, and the server span links to it when that upstream restarted the trace. - **Gen-AI spans** (LLM calls, guardrails, internal service calls) are created   by this package from LiteLLM's logging callbacks. Request-level spans parent to   the server span via the captured anchor; DB/service spans parent to the activelitellm/integrations/otel/mount.pyindex 9340f6e9e15..cb52ad24c8e 100644@@ -12,6 +12,8 @@ when the feature gate is off. """  import os+from collections.abc import Mapping+from types import MappingProxyType from typing import TYPE_CHECKING, Final, Protocol  from litellm._logging import verbose_logger@@ -19,6 +21,7 @@ from litellm.integrations.otel.model.config import is_otel_v2_enabled  if TYPE_CHECKING:     from fastapi import FastAPI+    from opentelemetry.trace import Span, TracerProvider  # Routes excluded from server-span tracing by default: high-frequency pollers and # static UI/docs assets, none of which are LLM traffic. Entries are substring-matched@@ -76,7 +79,7 @@ class _RenameableSpan(Protocol):     def set_attribute(self, key: str, value: str) -> None: ...  -def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: dict) -> None:+def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: Mapping[str, object]) -> None:     """FastAPI ``server_request_hook``: give passthrough server spans a useful name.      The instrumentation matches the route at span creation, so both the span name@@ -89,8 +92,8 @@ def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: dict) ->     try:         if span is None or not span.is_recording():             return-        path: Final = scope.get("path") or ""-        method: Final = scope.get("method") or ""+        path: Final = str(scope.get("path") or "")+        method: Final = str(scope.get("method") or "")         first_segment: Final = path.lstrip("/").split("/", 1)[0]         if first_segment in PASSTHROUGH_PREFIXES:             span.update_name(f"{method} {path}".strip())@@ -99,7 +102,7 @@ def _passthrough_span_name_hook(span: "_RenameableSpan | None", scope: dict) ->         pass  -def instrument_fastapi_app(app: "FastAPI") -> None:+def instrument_fastapi_app(app: "FastAPI", tracer_provider: "TracerProvider | None" = None) -> None:     """Attach OTel server-span instrumentation to the proxy FastAPI app.      Safe no-op when the V2 gate is off or ``opentelemetry-instrumentation-fastapi``@@ -111,6 +114,14 @@ def instrument_fastapi_app(app: "FastAPI") -> None:     ``ProxyTracerProvider``; the proxy publishes the real provider as the global     after config load (see ``proxy_startup_event``), and the proxy delegates to it.     That way server spans and gen-ai spans share one provider and the same trace.++    Once instrumented, every sampled HTTP response this app sends carries a+    ``traceresponse`` header naming its server span, unless the response already+    relays an upstream's ``traceresponse`` or another response propagator is already+    registered. Other OTel-instrumented apps in the same process get no header. A relayed+    ``traceresponse`` from an upstream that restarted the trace becomes a link on the+    server span. The server span also gets ``litellm.call_id`` from the response's+    ``x-litellm-call-id`` header. ``tracer_provider`` defaults to the global provider.     """     try:         if not is_otel_v2_enabled():@@ -122,6 +133,20 @@ def instrument_fastapi_app(app: "FastAPI") -> None:         # ``proxy_server``'s unconditional ``import`` of this module crash when the         # package is absent, even with the gate off.         from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor+        from opentelemetry.instrumentation.propagators import (+            get_global_response_propagator,+            set_global_response_propagator,  # pyright: ignore[reportUnknownVariableType]  # upstream leaves its parameter unannotated+        )++        from litellm.integrations.otel.trace_response import (+            UpstreamFirstTraceResponsePropagator,+            claim_server_span,+        )++        def server_request_hook(span: "Span | None", scope: Mapping[str, object]) -> None:+            if span is not None:+                claim_server_span(span)+            _passthrough_span_name_hook(span, scope)          excluded_urls: Final = (             os.environ.get("OTEL_PYTHON_FASTAPI_EXCLUDED_URLS")@@ -130,12 +155,47 @@ def instrument_fastapi_app(app: "FastAPI") -> None:         )         FastAPIInstrumentor.instrument_app(             app,+            tracer_provider=tracer_provider,             excluded_urls=excluded_urls,-            server_request_hook=_passthrough_span_name_hook,+            server_request_hook=server_request_hook,             # Drop the ASGI "http receive"/"http send" lifecycle sub-spans: they             # are low-value noise and (for passthrough) carry the catch-all route             # template in their name, which can't be rewritten from a hook.             exclude_spans=["receive", "send"],         )+        if get_global_response_propagator() is None:+            set_global_response_propagator(UpstreamFirstTraceResponsePropagator())     except Exception as e:         verbose_logger.debug("Skipping OTel V2 FastAPI instrumentation: %s", e)+++def trace_response_expose_headers() -> tuple[str, ...]:+    return ("traceresponse",) if is_otel_v2_enabled() else ()+++def trace_response_headers(span: "Span | None") -> Mapping[str, str]:+    """Response headers naming ``span`` for a response built outside the FastAPI instrumentation.++    Starlette runs the ``Exception`` handler outside every middleware, so its 500 never reaches the response+    propagator and the handler has to add these itself.+    """+    if span is None or not is_otel_v2_enabled():+        return MappingProxyType({})+    try:+        from opentelemetry import trace+        from opentelemetry.instrumentation.asgi import (  # pyright: ignore[reportMissingTypeStubs]  # the ASGI instrumentation ships no py.typed marker+            asgi_getter,+            asgi_setter,+        )+        from opentelemetry.instrumentation.propagators import get_global_response_propagator++        from litellm.integrations.otel.trace_response import claim_server_span+    except ImportError:+        return MappingProxyType({})+    claim_server_span(span)+    propagator: Final = get_global_response_propagator()  # pyright: ignore[reportUnknownVariableType]  # upstream leaves inject unannotated+    if propagator is None:+        return MappingProxyType({})+    start: Final[dict[str, object]] = {"type": "http.response.start"}  # mutable-ok: the propagator writes into it+    propagator.inject(start, context=trace.set_span_in_context(span), setter=asgi_setter)  # pyright: ignore[reportUnknownMemberType]  # upstream leaves inject unannotated+    return MappingProxyType({key: ", ".join(asgi_getter.get(start, key) or ()) for key in asgi_getter.keys(start)})  # pyright: ignore[reportUnknownMemberType]  # the ASGI instrumentation ships no py.typed markerlitellm/integrations/otel/trace_response.pynew file mode 100644index 00000000000..d602e372788@@ -0,0 +1,74 @@+from contextvars import ContextVar+from typing import Final++from opentelemetry import trace+from opentelemetry.context import Context+from opentelemetry.instrumentation.asgi import (  # pyright: ignore[reportMissingTypeStubs]  # the ASGI instrumentation ships no py.typed marker+    asgi_getter,+)+from opentelemetry.instrumentation.propagators import ResponsePropagator+from opentelemetry.propagators.textmap import CarrierT, Setter, default_setter+from opentelemetry.trace import Span, format_span_id, format_trace_id+from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator++from litellm.integrations.otel.model.semconv import LiteLLM++_LITELLM_SERVER_SPAN: Final[ContextVar[Span | None]] = ContextVar(+    "litellm_otel_traceresponse_server_span", default=None+)+++def claim_server_span(span: Span) -> None:+    _LITELLM_SERVER_SPAN.set(span)+++def _response_header(message: dict[str, object], name: str) -> str | None:  # mutable-ok: asgi_getter takes a dict+    values: Final = asgi_getter.get(message, name)  # pyright: ignore[reportUnknownMemberType]  # the ASGI instrumentation ships no py.typed marker+    return values[0] if values else None+++def _http_response_start(message: object) -> dict[str, object] | None:  # mutable-ok: the propagator writes into it+    if isinstance(message, dict) and message.get("type") == "http.response.start":  # pyright: ignore[reportUnknownMemberType]  # an ASGI message is an untyped dict+        return message  # pyright: ignore[reportUnknownVariableType]  # an ASGI message is an untyped dict+    return None+++def _link_a_restarted_upstream_trace(span: Span, upstream_traceresponse: str) -> None:+    upstream: Final = trace.get_current_span(+        TraceContextTextMapPropagator().extract({"traceparent": upstream_traceresponse})+    ).get_span_context()+    if upstream.trace_id != span.get_span_context().trace_id:+        span.add_link(upstream)+++class UpstreamFirstTraceResponsePropagator(ResponsePropagator):+    def inject(+        self,+        carrier: CarrierT,+        context: Context | None = None,+        setter: Setter[CarrierT] = default_setter,+    ) -> None:+        start: Final = _http_response_start(carrier)+        if start is None:+            return+        span: Final = trace.get_current_span(context)+        if span is not _LITELLM_SERVER_SPAN.get():+            return+        call_id: Final = _response_header(start, "x-litellm-call-id")+        if call_id:+            span.set_attribute(LiteLLM.CALL_ID, call_id)+        upstream_traceresponse: Final = _response_header(start, "traceresponse")+        if upstream_traceresponse is not None:+            _link_a_restarted_upstream_trace(span, upstream_traceresponse)+            return+        own: Final = span.get_span_context()+        if not own.trace_flags.sampled:+            return+        sent: Final = start.get("headers")+        if isinstance(sent, tuple):+            start["headers"] = [*sent]+        setter.set(+            carrier,+            "traceresponse",+            f"00-{format_trace_id(own.trace_id)}-{format_span_id(own.span_id)}-{own.trace_flags:02x}",+        )litellm/proxy/proxy_server.pyindex b51df22d07a..72a10dd1ec9 100644@@ -1906,7 +1906,11 @@ app = FastAPI( ## middleware after an application has started". See ## ``litellm.integrations.otel.mount`` for the full rationale; the call is a safe ## no-op when the gate is off or the instrumentation package is unavailable.-from litellm.integrations.otel.mount import instrument_fastapi_app+from litellm.integrations.otel.mount import (+    instrument_fastapi_app,+    trace_response_expose_headers,+    trace_response_headers,+)  instrument_fastapi_app(app) @@ -2223,6 +2227,12 @@ async def otel_request_validation_exception_handler(request: Request, exc: Reque async def otel_unhandled_exception_handler(request: Request, exc: Exception):     if isinstance(exc, (ProxyException, HTTPException, RequestValidationError)):         raise exc+    response: Final = await _unhandled_exception_response(request, exc)+    response.headers.update(trace_response_headers(getattr(request.state, "parent_otel_span", None)))+    return response+++async def _unhandled_exception_response(request: Request, exc: Exception) -> Response:     if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(exc):         verbose_proxy_logger.warning("Database unavailable during request: %s", type(exc).__name__)         return await openai_exception_handler(@@ -2606,7 +2616,7 @@ app.add_middleware(     allow_credentials=allow_cors_credentials,     allow_methods=["*"],     allow_headers=["*"],-    expose_headers=LITELLM_UI_ALLOW_HEADERS,+    expose_headers=[*LITELLM_UI_ALLOW_HEADERS, *trace_response_expose_headers()], )  app.add_middleware(PrometheusAuthMiddleware)tests/integration/observability/test_otel_trace_response_header.pynew file mode 100644index 00000000000..e84aeab48cf@@ -0,0 +1,735 @@+import itertools+import json+import re+import threading+import uuid+from collections import deque+from collections.abc import Callable, Iterator, Mapping, Sequence+from concurrent.futures import ThreadPoolExecutor+from dataclasses import dataclass+from pathlib import Path+from types import MappingProxyType+from typing import Final+from urllib.parse import parse_qs, urlsplit++import anthropic+import httpx+import openai+import pytest+import yaml+from integration._support.client import Gateway, eventually, gateway_from_environment, list_value, object_value+from integration._support.database import read_rows+from integration._support.process import OwnedProxy, graceful_stop_seconds, owned_proxy_process+from integration._support.wire import Reply, Request, Wire, wire_server+from pydantic import JsonValue, TypeAdapter++MARKER: Final = re.compile(rb"traceresp-[0-9a-f]{32}")+TRACERESPONSE: Final = re.compile(r"00-([0-9a-f]{32})-([0-9a-f]{16})-01")+SERVER: Final = 2+CALL_ID: Final = "litellm.call_id"+ANSWER: Final = "trace ok"+JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue])+UPSTREAM_CALL_ID: Final = "upstream-chosen-call-id"+HOSTILE_UPSTREAM_VALUES: Final = MappingProxyType(+    {+        "empty": "",+        "numeric": "12345",+        "big": "00-" + "a" * 5000,+        "garbage": "not-a-traceparent",+        "joined": f"00-{'1' * 32}-{'2' * 16}-01, 00-{'3' * 32}-{'4' * 16}-01",+    }+)+RELAY_CASES: Final = ("none", "continued", "restarted", "callid", *HOSTILE_UPSTREAM_VALUES)+++def _marker() -> str:+    return "traceresp-" + uuid.uuid4().hex+++def _trace_id() -> str:+    return uuid.uuid4().hex+++def _span_id() -> str:+    return uuid.uuid4().hex[:16]+++def _chat_reply(identity: str, stream: bool) -> Reply:+    message: Final = {"role": "assistant", "content": ANSWER}+    usage: Final = {"prompt_tokens": 7, "completion_tokens": 2, "total_tokens": 9}+    if not stream:+        return Reply(+            body=json.dumps(+                {+                    "id": identity,+                    "object": "chat.completion",+                    "created": 1,+                    "model": "gpt-4o-mini",+                    "choices": [{"index": 0, "message": message, "finish_reason": "stop"}],+                    "usage": usage,+                }+            ).encode()+        )+    chunk: Final = {"id": identity, "object": "chat.completion.chunk", "created": 1, "model": "gpt-4o-mini"}+    frames: Final = (+        {**chunk, "choices": [{"index": 0, "delta": message}]},+        {**chunk, "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}], "usage": usage},+    )+    return Reply(+        content_type="text/event-stream",+        chunks=(*(b"data: " + json.dumps(frame).encode() + b"\n\n" for frame in frames), b"data: [DONE]\n\n"),+    )+++def _responses_reply(identity: str, stream: bool) -> Reply:+    response: Final = {+        "id": identity,+        "object": "response",+        "created_at": 1,+        "status": "completed",+        "model": "gpt-4o-mini",+        "output": [+            {+                "id": "msg_" + identity,+                "type": "message",+                "role": "assistant",+                "status": "completed",+                "content": [{"type": "output_text", "text": ANSWER, "annotations": []}],+            }+        ],+        "usage": {"input_tokens": 7, "output_tokens": 2, "total_tokens": 9},+    }+    if not stream:+        return Reply(body=json.dumps(response).encode())+    events: Final = (+        {"type": "response.created", "sequence_number": 0, "response": {**response, "status": "in_progress"}},+        {+            "type": "response.output_text.delta",+            "sequence_number": 1,+            "item_id": "msg_" + identity,+            "output_index": 0,+            "content_index": 0,+            "delta": ANSWER,+        },+        {"type": "response.completed", "sequence_number": 2, "response": response},+    )+    return Reply(+        content_type="text/event-stream",+        chunks=tuple(f"event: {event['type']}\ndata: {json.dumps(event)}\n\n".encode() for event in events),+    )+++def _provider(request: Request) -> Reply:+    found: Final = MARKER.search(request.body)+    if found is None:+        return Reply(status=404, body=b'{"error":"no marker"}')+    marker: Final = found.group(0).decode()+    stream: Final = JSON_OBJECT.validate_json(request.body).get("stream") is True+    if request.target.endswith("/responses"):+        return _responses_reply(f"resp_{marker}", stream)+    return _chat_reply(f"chatcmpl-{marker}", stream)+++def _query(request: Request) -> Mapping[str, str]:+    return MappingProxyType({name: values[0] for name, values in parse_qs(urlsplit(request.target).query).items()})+++def _continued_traceresponse(request: Request) -> Reply:+    incoming: Final = TRACERESPONSE.fullmatch(request.headers.get("traceparent", ""))+    if incoming is None:+        return Reply(status=400, body=b'{"error":"no sampled traceparent reached the upstream"}')+    return Reply(body=b'{"case":"continued"}', headers={"traceresponse": f"00-{incoming.group(1)}-{_span_id()}-01"})+++def _relay(request: Request) -> Reply:+    case: Final = urlsplit(request.target).path.rsplit("/", 1)[-1]+    query: Final = _query(request)+    body: Final = json.dumps({"case": case}).encode()+    if case == "continued":+        return _continued_traceresponse(request)+    if case == "restarted":+        return Reply(body=body, headers={"traceresponse": f"00-{query['trace']}-{query['span']}-01"})+    if case == "callid":+        return Reply(status=int(query.get("status", "200")), body=body, headers={"x-litellm-call-id": UPSTREAM_CALL_ID})+    if case in HOSTILE_UPSTREAM_VALUES:+        return Reply(body=body, headers={"traceresponse": HOSTILE_UPSTREAM_VALUES[case]})+    return Reply(body=body)+++@dataclass(frozen=True, slots=True)+class ExportedSpan:+    trace_id: str+    span_id: str+    parent_span_id: str+    kind: int+    attributes: Mapping[str, JsonValue]+    links: tuple[tuple[str, str], ...]+++def _attribute(value: JsonValue) -> JsonValue:+    typed: Final = object_value(value)+    return next((typed[key] for key in ("stringValue", "intValue", "boolValue", "doubleValue") if key in typed), None)+++def _link(value: JsonValue) -> tuple[str, str]:+    link: Final = object_value(value)+    return str(link["traceId"]), str(link["spanId"])+++def _exported(value: JsonValue) -> ExportedSpan:+    span: Final = object_value(value)+    attributes: Final = tuple(object_value(attribute) for attribute in list_value(span.get("attributes", [])))+    return ExportedSpan(+        trace_id=str(span["traceId"]),+        span_id=str(span["spanId"]),+        parent_span_id=str(span.get("parentSpanId", "")),+        kind=int(str(span.get("kind", 0))),+        attributes=MappingProxyType({str(item["key"]): _attribute(item["value"]) for item in attributes}),+        links=tuple(_link(link) for link in list_value(span.get("links", []))),+    )+++def _scope_spans(resource: JsonValue) -> Iterator[JsonValue]:+    return itertools.chain.from_iterable(+        list_value(object_value(scope).get("spans", [])) for scope in list_value(object_value(resource)["scopeSpans"])+    )+++def _batch_spans(body: bytes) -> tuple[ExportedSpan, ...]:+    resources: Final = list_value(JSON_OBJECT.validate_json(body).get("resourceSpans", []))+    return tuple(_exported(span) for span in itertools.chain.from_iterable(map(_scope_spans, resources)))+++@dataclass(frozen=True, slots=True)+class Collector:+    wire: Wire+    outage: threading.Event+    accepted: Sequence[Request]+    guard: threading.Lock++    def spans(self) -> tuple[ExportedSpan, ...]:+        with self.guard:+            batches: Final = tuple(self.accepted)+        return tuple(itertools.chain.from_iterable(_batch_spans(batch.body) for batch in batches))++    def with_span_id(self, span_id: str) -> tuple[ExportedSpan, ...]:+        return tuple(span for span in self.spans() if span.span_id == span_id)++    def span(self, span_id: str) -> ExportedSpan:+        eventually(lambda: self.with_span_id(span_id), lambda found: len(found) >= 1, seconds=30)+        found: Final = self.with_span_id(span_id)+        assert len(found) == 1, found+        return found[0]++    def server_spans_for_call(self, call_id: str) -> tuple[ExportedSpan, ...]:+        return tuple(span for span in self.spans() if span.kind == SERVER and span.attributes.get(CALL_ID) == call_id)++    def server_span_for_call(self, call_id: str) -> ExportedSpan:+        eventually(lambda: self.server_spans_for_call(call_id), lambda found: len(found) >= 1, seconds=30)+        found: Final = self.server_spans_for_call(call_id)+        assert len(found) == 1, found+        return found[0]+++@pytest.fixture(scope="module")+def collector() -> Iterator[Collector]:+    outage: Final = threading.Event()+    accepted: Final[deque[Request]] = deque()  # mutable-ok: the sink thread appends each accepted export batch+    guard: Final = threading.Lock()++    def sink(request: Request) -> Reply:+        if outage.is_set():+            return Reply(status=503, body=b'{"error":"sink down"}')+        with guard:+            accepted.append(request)+        return Reply()++    with wire_server(sink) as wire:+        yield Collector(wire, outage, accepted, guard)+++@pytest.fixture(scope="module")+def provider() -> Iterator[Wire]:+    with wire_server(_provider) as wire:+        yield wire+++@pytest.fixture(scope="module")+def relay() -> Iterator[Wire]:+    with wire_server(_relay) as wire:+        yield wire+++@dataclass(frozen=True, slots=True)+class Answer:+    headers: httpx.Headers+    text: str+++@dataclass(frozen=True, slots=True)+class Rig:+    proxy: Gateway+    process: OwnedProxy+    model: str+    upstream: Wire+    sink: Collector++    def base_url(self) -> str:+        return str(self.proxy.client.base_url)++    def openai_client(self) -> openai.OpenAI:+        return openai.OpenAI(base_url=self.base_url() + "/v1", api_key=self.proxy.key, max_retries=0)++    def anthropic_client(self) -> anthropic.Anthropic:+        return anthropic.Anthropic(base_url=self.base_url(), api_key=self.proxy.key, max_retries=0)++    def post(self, path: str, body: Mapping[str, JsonValue], headers: Sequence[tuple[str, str]] = ()) -> httpx.Response:+        return self.proxy.client.post(+            path, json=body, headers=[("Authorization", f"Bearer {self.proxy.key}"), *headers]+        )++    def chat(self, marker: str, *, headers: Sequence[tuple[str, str]] = ()) -> httpx.Response:+        return self.post(+            "/v1/chat/completions",+            {"model": self.model, "messages": [{"role": "user", "content": marker}], "cache": {"no-cache": True}},+            headers,+        )++    def upstream_count(self, marker: str) -> int:+        return sum(1 for request in self.upstream.drain() if marker.encode() in request.body)+++def _config(provider: Wire, relay: Wire, sink: Collector, model: str) -> dict[str, JsonValue]:+    config: Final = object_value(yaml.safe_load(Path("tests/integration/proxy_config.yaml").read_text()))+    return {+        **config,+        "model_list": [+            *list_value(config.get("model_list") or []),+            {+                "model_name": model,+                "litellm_params": {+                    "model": "openai/gpt-4o-mini",+                    "api_key": "integration-provider-key",+                    "api_base": provider.url + "/v1",+                },+            },+        ],+        "litellm_settings": {**object_value(config["litellm_settings"]), "callbacks": ["otel"]},+        "callback_settings": {"otel": {"exporter": "http/json", "endpoint": sink.wire.url}},+        "general_settings": {+            **object_value(config["general_settings"]),+            "pass_through_endpoints": [+                {"path": f"/relay/{case}", "target": f"{relay.url}/relay/{case}"} for case in RELAY_CASES+            ],+        },+    }+++@dataclass(frozen=True, slots=True)+class RigFactory:+    provider: Wire+    relay: Wire+    sink: Collector+    directory: Path+    otel_v2: bool++    def start(self) -> Iterator[Rig]:+        model: Final = "traceresp-model-" + uuid.uuid4().hex[:12]+        path: Final = self.directory / f"traceresponse-{uuid.uuid4().hex}.yaml"+        path.write_text(yaml.safe_dump(_config(self.provider, self.relay, self.sink, model)))+        gate: Final = {"LITELLM_OTEL_V2": "1"} if self.otel_v2 else {}+        with (+            gateway_from_environment() as gateway,+            owned_proxy_process(+                gateway,+                self.directory,+                {**gate, "OTEL_BSP_SCHEDULE_DELAY": "300"},+                config=path,+                remove_environment=() if self.otel_v2 else ("LITELLM_OTEL_V2",),+                workers=2,+            ) as owned,+        ):+            yield Rig(owned.gateway, owned, model, self.provider, self.sink)+++@pytest.fixture(scope="module")+def factory(+    provider: Wire, relay: Wire, collector: Collector, tmp_path_factory: pytest.TempPathFactory+) -> Callable[[bool], RigFactory]:+    return lambda otel_v2: RigFactory(provider, relay, collector, tmp_path_factory.mktemp("traceresponse"), otel_v2)+++@pytest.fixture(scope="module")+def rig(factory: Callable[[bool], RigFactory]) -> Iterator[Rig]:+    yield from factory(True).start()+++@pytest.fixture(scope="module")+def control(factory: Callable[[bool], RigFactory]) -> Iterator[Rig]:+    yield from factory(False).start()+++def _named(headers: httpx.Headers) -> tuple[str, str]:+    values: Final = headers.get_list("traceresponse")+    assert len(values) == 1, values+    match: Final = TRACERESPONSE.fullmatch(values[0])+    assert match is not None, values+    return match.group(1), match.group(2)+++def _assert_names_its_server_span(rig: Rig, headers: httpx.Headers) -> ExportedSpan:+    trace_id, span_id = _named(headers)+    span: Final = rig.sink.span(span_id)+    assert (span.kind, span.trace_id) == (SERVER, trace_id), span+    call_ids: Final = headers.get_list("x-litellm-call-id")+    assert len(call_ids) == 1 and span.attributes.get(CALL_ID) == call_ids[0], (call_ids, span)+    return span+++def _chat_sdk(rig: Rig, marker: str) -> Answer:+    completion: Final = rig.openai_client().chat.completions.with_raw_response.create(+        model=rig.model, messages=[{"role": "user", "content": marker}], extra_body={"cache": {"no-cache": True}}+    )+    return Answer(completion.headers, completion.parse().choices[0].message.content or "")+++def _chat_stream_sdk(rig: Rig, marker: str) -> Answer:+    with rig.openai_client().chat.completions.with_streaming_response.create(+        model=rig.model,+        messages=[{"role": "user", "content": marker}],+        stream=True,+        extra_body={"cache": {"no-cache": True}},+    ) as chunks:+        text: Final = "".join(chunk.choices[0].delta.content or "" for chunk in chunks.parse() if chunk.choices)+        return Answer(chunks.headers, text)+++def _messages_sdk(rig: Rig, marker: str) -> Answer:+    reply: Final = rig.anthropic_client().messages.with_raw_response.create(+        model=rig.model, max_tokens=16, messages=[{"role": "user", "content": marker}]+    )+    block: Final = reply.parse().content[0]+    assert isinstance(block, anthropic.types.TextBlock), block+    return Answer(reply.headers, block.text)+++def _messages_stream_sdk(rig: Rig, marker: str) -> Answer:+    with rig.anthropic_client().messages.with_streaming_response.create(+        model=rig.model, max_tokens=16, messages=[{"role": "user", "content": marker}], stream=True+    ) as streamed:+        text: Final = "".join(+            event.delta.text+            for event in streamed.parse()+            if event.type == "content_block_delta" and event.delta.type == "text_delta"+        )+        return Answer(streamed.headers, text)+++def _responses_sdk(rig: Rig, marker: str) -> Answer:+    created: Final = rig.openai_client().responses.with_raw_response.create(model=rig.model, input=marker)+    return Answer(created.headers, created.parse().output_text)+++def _responses_stream_sdk(rig: Rig, marker: str) -> Answer:+    with rig.openai_client().responses.with_streaming_response.create(+        model=rig.model, input=marker, stream=True+    ) as events:+        text: Final = "".join(event.delta for event in events.parse() if event.type == "response.output_text.delta")+        return Answer(events.headers, text)+++SDK_ROUTES: Final[Mapping[str, Callable[[Rig, str], Answer]]] = MappingProxyType(+    {+        "chat": _chat_sdk,+        "chat_stream": _chat_stream_sdk,+        "messages": _messages_sdk,+        "messages_stream": _messages_stream_sdk,+        "responses": _responses_sdk,+        "responses_stream": _responses_stream_sdk,+    }+)+++@pytest.mark.parametrize("route", tuple(SDK_ROUTES))+def test_sdk_response_names_the_server_span_that_carries_its_call_id(rig: Rig, route: str) -> None:+    marker: Final = _marker()+    answer: Final = SDK_ROUTES[route](rig, marker)+    assert answer.text == ANSWER, answer+    _assert_names_its_server_span(rig, answer.headers)+    assert rig.upstream_count(marker) == 1+++def _spend_rows(response_id: str) -> int:+    return len(read_rows('SELECT request_id FROM "LiteLLM_SpendLogs" WHERE request_id=%s', (response_id,)))+++@pytest.mark.parametrize("stream", (False, True), ids=("plain", "stream"))+def test_a_response_cache_hit_names_its_own_server_span(rig: Rig, stream: bool) -> None:+    marker: Final = _marker()+    body: Final[dict[str, JsonValue]] = {+        "model": rig.model,+        "messages": [{"role": "user", "content": marker}],+        "stream": stream,+    }+    first: Final = rig.post("/v1/chat/completions", body)+    assert first.status_code == 200, first.text+    first_span: Final = _assert_names_its_server_span(rig, first.headers)+    eventually(lambda: _spend_rows(f"chatcmpl-{marker}"), lambda rows: rows == 1, seconds=70)+    second: Final = rig.post("/v1/chat/completions", body)+    assert second.status_code == 200, second.text+    assert f"chatcmpl-{marker}" in second.text, second.text+    second_span: Final = _assert_names_its_server_span(rig, second.headers)+    assert second_span.span_id != first_span.span_id+    assert second.headers["x-litellm-call-id"] != first.headers["x-litellm-call-id"]+    assert rig.upstream_count(marker) == 1+++def test_a_sampled_caller_traceparent_is_continued_in_the_header(rig: Rig) -> None:+    trace_id: Final = _trace_id()+    parent: Final = _span_id()+    response: Final = rig.chat(_marker(), headers=(("traceparent", f"00-{trace_id}-{parent}-01"),))+    assert response.status_code == 200, response.text+    span: Final = _assert_names_its_server_span(rig, response.headers)+    assert (span.trace_id, span.parent_span_id) == (trace_id, parent), span+++def test_an_unsampled_caller_traceparent_gets_no_header(rig: Rig) -> None:+    response: Final = rig.chat(_marker(), headers=(("traceparent", f"00-{_trace_id()}-{_span_id()}-00"),))+    assert response.status_code == 200, response.text+    assert response.headers.get_list("traceresponse") == []+++@pytest.mark.parametrize(+    "traceparent",+    ("", "12345", "00-" + "f" * 5000, "not-a-traceparent", f"00-{'5' * 32}-{'6' * 16}-01, 00-{'7' * 32}-{'8' * 16}-01"),+    ids=("empty", "numeric", "five_kilobytes", "garbage", "list"),+)+def test_a_malformed_caller_traceparent_starts_a_fresh_trace_in_the_header(rig: Rig, traceparent: str) -> None:+    response: Final = rig.chat(_marker(), headers=(("traceparent", traceparent),))+    assert response.status_code == 200, response.text+    span: Final = _assert_names_its_server_span(rig, response.headers)+    assert span.parent_span_id == "", span+    assert span.trace_id not in traceparent, span+++def test_a_traceparent_sent_twice_is_continued_once(rig: Rig) -> None:+    trace_id: Final = _trace_id()+    parent: Final = _span_id()+    value: Final = f"00-{trace_id}-{parent}-01"+    response: Final = rig.chat(_marker(), headers=(("traceparent", value), ("traceparent", value)))+    assert response.status_code == 200, response.text+    span: Final = _assert_names_its_server_span(rig, response.headers)+    assert (span.trace_id, span.parent_span_id) == (trace_id, parent), span+++def _error_bad_key(rig: Rig) -> httpx.Response:+    return rig.proxy.client.post(+        "/v1/chat/completions",+        json={"model": rig.model, "messages": [{"role": "user", "content": _marker()}]},+        headers={"Authorization": "Bearer sk-not-a-real-key"},+    )+++def _error_unknown_model(rig: Rig) -> httpx.Response:+    return rig.post(+        "/v1/chat/completions",+        {"model": "no-such-model-" + uuid.uuid4().hex, "messages": [{"role": "user", "content": "x"}]},+    )+++def _error_invalid_body(rig: Rig) -> httpx.Response:+    return rig.post("/key/generate", {"models": "not-a-list"})+++def _error_unhandled(rig: Rig) -> httpx.Response:+    return rig.proxy.client.post(+        "/v1/chat/completions",+        content=b"[1]",+        headers={"Authorization": f"Bearer {rig.proxy.key}", "content-type": "application/json"},+    )+++ERRORS: Final[Mapping[str, tuple[int, Callable[[Rig], httpx.Response]]]] = MappingProxyType(+    {+        "bad_key": (401, _error_bad_key),+        "unknown_model": (400, _error_unknown_model),+        "invalid_body": (422, _error_invalid_body),+        "unhandled": (500, _error_unhandled),+    }+)+++@pytest.mark.parametrize("error", tuple(ERRORS))+def test_an_error_response_names_its_own_server_span(rig: Rig, error: str) -> None:+    status, send = ERRORS[error]+    response: Final = send(rig)+    assert response.status_code == status, response.text+    trace_id, span_id = _named(response.headers)+    span: Final = rig.sink.span(span_id)+    assert (span.kind, span.trace_id) == (SERVER, trace_id), span+++def test_an_excluded_health_route_gets_no_header(rig: Rig) -> None:+    response: Final = rig.proxy.client.get("/health/liveliness")+    assert response.status_code == 200, response.text+    assert response.headers.get_list("traceresponse") == []+++def test_a_pass_through_without_an_upstream_header_gets_the_gateway_header(rig: Rig) -> None:+    response: Final = rig.post("/relay/none", {"marker": _marker()})+    assert response.status_code == 200, response.text+    assert response.json() == {"case": "none"}+    span: Final = _assert_names_its_server_span(rig, response.headers)+    assert span.links == (), span+++def test_a_pass_through_upstream_header_in_the_same_trace_is_relayed_unchanged_without_a_link(rig: Rig) -> None:+    trace_id: Final = _trace_id()+    response: Final = rig.post(+        "/relay/continued", {"marker": _marker()}, (("traceparent", f"00-{trace_id}-{_span_id()}-01"),)+    )+    assert response.status_code == 200, response.text+    relayed: Final = response.headers.get_list("traceresponse")+    assert len(relayed) == 1 and relayed[0].startswith(f"00-{trace_id}-"), relayed+    span: Final = rig.sink.server_span_for_call(response.headers["x-litellm-call-id"])+    assert span.trace_id == trace_id, span+    assert span.span_id not in relayed[0], (span, relayed)+    assert span.links == (), span+++def test_a_pass_through_upstream_that_restarted_the_trace_is_relayed_and_linked(rig: Rig) -> None:+    upstream_trace: Final = _trace_id()+    upstream_span: Final = _span_id()+    response: Final = rig.proxy.client.post(+        "/relay/restarted",+        params={"trace": upstream_trace, "span": upstream_span},+        json={"marker": _marker()},+        headers={"Authorization": f"Bearer {rig.proxy.key}"},+    )+    assert response.status_code == 200, response.text+    assert response.headers.get_list("traceresponse") == [f"00-{upstream_trace}-{upstream_span}-01"]+    span: Final = rig.sink.server_span_for_call(response.headers["x-litellm-call-id"])+    assert span.trace_id != upstream_trace, span+    assert span.links == ((upstream_trace, upstream_span),), span+++@pytest.mark.parametrize("case", tuple(HOSTILE_UPSTREAM_VALUES))+def test_a_hostile_pass_through_upstream_header_is_relayed_once_unchanged_and_never_linked(rig: Rig, case: str) -> None:+    response: Final = rig.post(f"/relay/{case}", {"marker": _marker()})+    assert response.status_code == 200, response.text+    assert response.headers.get_list("traceresponse") == [HOSTILE_UPSTREAM_VALUES[case]]+    span: Final = rig.sink.server_span_for_call(response.headers["x-litellm-call-id"])+    assert span.links == (), span+++@pytest.mark.parametrize("status", (200, 404, 500))+def test_a_pass_through_upstream_call_id_never_replaces_the_gateway_call_id(rig: Rig, status: int) -> None:+    response: Final = rig.proxy.client.post(+        "/relay/callid",+        params={"status": str(status)},+        json={"marker": _marker()},+        headers={"Authorization": f"Bearer {rig.proxy.key}"},+    )+    assert response.status_code == status, response.text+    call_ids: Final = response.headers.get_list("x-litellm-call-id")+    assert len(call_ids) == 1 and call_ids[0] != UPSTREAM_CALL_ID, call_ids+    _assert_names_its_server_span(rig, response.headers)+    assert rig.sink.server_spans_for_call(UPSTREAM_CALL_ID) == ()+++def _exposed(response: httpx.Response) -> frozenset[str]:+    listed: Final = response.headers.get("access-control-expose-headers", "")+    return frozenset(name.strip().lower() for name in listed.split(",") if name.strip())+++def test_a_browser_origin_can_read_the_header(rig: Rig) -> None:+    response: Final = rig.proxy.request("GET", "/v1/models", headers={"Origin": "http://dashboard.example"})+    assert response.status_code == 200, response.text+    assert "traceresponse" in _exposed(response), response.headers+    _named(response.headers)+++def test_with_the_v2_gate_off_no_response_carries_or_exposes_the_header(control: Rig) -> None:+    marker: Final = _marker()+    chat: Final = control.chat(marker)+    assert chat.status_code == 200, chat.text+    assert chat.headers.get_list("traceresponse") == []+    assert control.upstream_count(marker) == 1+    models: Final = control.proxy.request("GET", "/v1/models", headers={"Origin": "http://dashboard.example"})+    assert models.status_code == 200, models.text+    assert _exposed(models), models.headers+    assert "traceresponse" not in _exposed(models), models.headers+    assert models.headers.get_list("traceresponse") == []+++def _raw_send(rig: Rig, index: int) -> httpx.Response:+    marker: Final = _marker()+    stream: Final = index % 2 == 0+    route: Final = index % 3+    if route == 0:+        return rig.post(+            "/v1/chat/completions",+            {"model": rig.model, "messages": [{"role": "user", "content": marker}], "stream": stream},+        )+    if route == 1:+        return rig.post("/v1/responses", {"model": rig.model, "input": marker, "stream": stream})+    return rig.post(+        "/v1/messages",+        {"model": rig.model, "max_tokens": 16, "messages": [{"role": "user", "content": marker}], "stream": stream},+    )+++def _burst(rig: Rig, count: int) -> tuple[httpx.Response, ...]:+    with ThreadPoolExecutor(max_workers=8) as pool:+        return tuple(pool.map(lambda index: _raw_send(rig, index), range(count)))+++def _assert_each_names_its_own_span(rig: Rig, responses: Sequence[httpx.Response]) -> None:+    assert [response.status_code for response in responses] == [200] * len(responses), [+        response.text for response in responses+    ]+    spans: Final = tuple(_assert_names_its_server_span(rig, response.headers) for response in responses)+    assert len({span.span_id for span in spans}) == len(responses), spans+    assert len({span.trace_id for span in spans}) == len(responses), spans+++def test_concurrent_mixed_requests_each_name_their_own_server_span(rig: Rig) -> None:+    _assert_each_names_its_own_span(rig, _burst(rig, 24))+++def test_a_sink_outage_keeps_every_header_and_the_spans_land_after_recovery(rig: Rig) -> None:+    rig.sink.outage.set()+    try:+        responses: Final = _burst(rig, 20)+        assert all(len(TRACERESPONSE.findall(response.headers.get("traceresponse", ""))) == 1 for response in responses)+        eventually(lambda: any(True for _ in rig.sink.wire.drain()), lambda seen: seen, seconds=30)+    finally:+        rig.sink.outage.clear()+    _assert_each_names_its_own_span(rig, responses)+++def _fresh_chat(rig: Rig) -> httpx.Response:+    with httpx.Client(base_url=rig.base_url(), timeout=30, trust_env=False) as client:+        return client.post(+            "/v1/chat/completions",+            json={+                "model": rig.model,+                "messages": [{"role": "user", "content": _marker()}],+                "cache": {"no-cache": True},+            },+            headers={"Authorization": f"Bearer {rig.proxy.key}"},+        )+++@pytest.mark.timeout(2 * graceful_stop_seconds() + 120)+def test_replacement_workers_still_name_their_server_spans(factory: Callable[[bool], RigFactory]) -> None:+    started: Final = factory(True).start()+    fresh: Final = next(started)+    try:+        originals: Final = eventually(fresh.process.worker_pids, lambda found: len(found) == 2, seconds=30)+        replacements: Final = frozenset(fresh.process.kill_worker(pid) for pid in sorted(originals))+        assert not replacements & originals, (originals, replacements)+        assert fresh.process.process.poll() is None, "Proxy root exited after its workers were replaced"+        _assert_each_names_its_own_span(fresh, tuple(_fresh_chat(fresh) for _ in range(8)))+    finally:+        started.close()tests/unit/integrations/otel/test_otel_v2_mount.pyindex e2007486a40..3a27c3ec4e4 100644@@ -5,6 +5,7 @@ surface and the server-span + shared-provider behavior it produces. """  +import contextvars from datetime import datetime, timezone  import pytest@@ -14,15 +15,28 @@ pytest.importorskip("opentelemetry") pytest.importorskip("opentelemetry.instrumentation.fastapi") fastapi = pytest.importorskip("fastapi") +from fastapi.middleware.cors import CORSMiddleware  # noqa: E402+from fastapi.responses import StreamingResponse  # noqa: E402 from fastapi.testclient import TestClient  # noqa: E402+from starlette.routing import Route  # noqa: E402+import httpx  # noqa: E402 from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor  # noqa: E402+from opentelemetry.instrumentation.propagators import (  # noqa: E402+    ResponsePropagator,+    get_global_response_propagator,+    set_global_response_propagator,+)+from opentelemetry.sdk.trace import TracerProvider  # noqa: E402 from opentelemetry.sdk.trace.export import SimpleSpanProcessor  # noqa: E402 from opentelemetry.sdk.trace.export.in_memory_span_exporter import (  # noqa: E402     InMemorySpanExporter, ) from opentelemetry import trace  # noqa: E402+from opentelemetry.sdk.trace.sampling import ALWAYS_OFF  # noqa: E402 from opentelemetry.trace import SpanKind  # noqa: E402 +from litellm.constants import LITELLM_UI_ALLOW_HEADERS  # noqa: E402+ from litellm.integrations.otel.model.config import (  # noqa: E402     OpenTelemetryV2Config,     is_otel_v2_enabled,@@ -32,6 +46,11 @@ from litellm.integrations.otel.mount import (  # noqa: E402     PASSTHROUGH_PREFIXES,     _passthrough_span_name_hook,     instrument_fastapi_app,+    trace_response_expose_headers,+    trace_response_headers,+)+from litellm.proxy.pass_through_endpoints.pass_through_endpoints import (  # noqa: E402+    HttpPassThroughEndpointHelpers, ) from litellm.integrations.otel.plumbing.context import (  # noqa: E402     request_root_http_route,@@ -52,6 +71,13 @@ def _reset_request_root_span():     _otel_context._mcp_message_transport_span.set(None)  +@pytest.fixture(autouse=True)+def _reset_response_propagator():+    set_global_response_propagator(None)+    yield+    set_global_response_propagator(None)++ @pytest.fixture(autouse=True) def _clear_otel_v2_flag_cache():     is_otel_v2_enabled.cache_clear()@@ -227,6 +253,20 @@ def test_blank_route_on_the_server_span_is_omitted():     assert request_root_http_route() is None  +def test_span_hook_does_not_break_requests_when_the_span_provider_fails():+    class _FailingSpan:+        def is_recording(self) -> bool:+            raise RuntimeError("span provider unavailable")++        def update_name(self, name: str) -> None:+            raise AssertionError(name)++        def set_attribute(self, key: str, value: str) -> None:+            raise AssertionError((key, value))++    assert _passthrough_span_name_hook(_FailingSpan(), {}) is None++ def test_known_passthrough_prefixes_present():     """Guard the prefix set against accidental edits."""     assert {"openai", "anthropic", "vertex_ai", "bedrock"} <= PASSTHROUGH_PREFIXES@@ -254,3 +294,255 @@ def test_instrument_fastapi_app_attaches_when_gate_on(monkeypatch):         assert getattr(app, "_is_instrumented_by_opentelemetry", False) is True     finally:         FastAPIInstrumentor.uninstrument_app(app)+++_UPSTREAM_TRACERESPONSE = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"+++class _TupleHeadersApp:+    async def __call__(self, scope, receive, send):+        await send({"type": "http.response.start", "status": 200, "headers": ((b"x-litellm-call-id", b"call-tuple"),)})+        await send({"type": "http.response.body", "body": b"ok"})+++def _traced_app(monkeypatch, tracer_provider):+    from litellm.proxy.proxy_server import otel_unhandled_exception_handler++    monkeypatch.setenv("LITELLM_OTEL_V2", "1")+    is_otel_v2_enabled.cache_clear()+    app = fastapi.FastAPI()+    app.add_exception_handler(Exception, otel_unhandled_exception_handler)++    @app.get("/success")+    async def success():+        return fastapi.Response(headers={"x-litellm-call-id": "call-success"})++    @app.get("/failure")+    async def failure():+        raise fastapi.HTTPException(status_code=400, headers={"x-litellm-call-id": "call-failure"})++    async def chunks():+        yield b"data: first\n\n"+        yield b"data: [DONE]\n\n"++    @app.get("/stream")+    async def stream():+        return StreamingResponse(chunks(), headers={"x-litellm-call-id": "call-stream"})++    async def relay_upstream():+        upstream = httpx.Headers({"traceresponse": _UPSTREAM_TRACERESPONSE})+        return fastapi.Response(headers=HttpPassThroughEndpointHelpers.get_response_headers(upstream, "call-relayed"))++    app.add_api_route("/relayed", relay_upstream)+    app.add_api_route("/relayed/health", relay_upstream)++    @app.get("/relayed-untraced")+    async def relay_untraced_upstream():+        upstream = httpx.Headers({"access-control-expose-headers": "x-upstream-request-id"})+        return fastapi.Response(headers=HttpPassThroughEndpointHelpers.get_response_headers(upstream, "call-untraced"))++    app.router.routes.append(Route("/tuple-headers", _TupleHeadersApp()))++    @app.get("/crash")+    async def crash(request: fastapi.Request):+        request.state.parent_otel_span = trace.get_current_span()+        raise RuntimeError("unhandled")++    @app.websocket("/ws")+    async def websocket_route(websocket: fastapi.WebSocket):+        await websocket.accept()+        await websocket.send_text("ok")+        await websocket.close()++    instrument_fastapi_app(app, tracer_provider=tracer_provider)+    return app+++def _exporting_provider():+    logger = OpenTelemetryV2(config=OpenTelemetryV2Config(exporter="in_memory"))+    exporter = InMemorySpanExporter()+    logger._tracer_provider.add_span_processor(SimpleSpanProcessor(exporter))+    return logger._tracer_provider, exporter+++def _server_spans_by_route(exporter):+    return {(s.attributes or {})["http.route"]: s for s in exporter.get_finished_spans() if s.kind is SpanKind.SERVER}+++def test_every_traced_response_names_its_own_server_span(monkeypatch):+    provider, exporter = _exporting_provider()+    client = TestClient(_traced_app(monkeypatch, provider), raise_server_exceptions=False)++    responses = {path: client.get(path) for path in ("/success", "/failure", "/stream", "/tuple-headers", "/crash")}++    server_spans = _server_spans_by_route(exporter)+    assert responses["/failure"].status_code == 400+    assert responses["/tuple-headers"].status_code == 200+    assert responses["/crash"].status_code == 500+    assert responses["/stream"].text == "data: first\n\ndata: [DONE]\n\n"+    for path, response in responses.items():+        context = server_spans[path].context+        assert response.headers.get_list("traceresponse") == [f"00-{context.trace_id:032x}-{context.span_id:016x}-01"]+++@pytest.mark.parametrize("path", ["/relayed", "/relayed/health"])+def test_a_relayed_upstream_traceresponse_is_returned_once_and_unchanged(monkeypatch, path):+    provider, _ = _exporting_provider()++    response = TestClient(_traced_app(monkeypatch, provider)).get(path)++    assert response.headers.get_list("traceresponse") == [_UPSTREAM_TRACERESPONSE]+++@pytest.mark.parametrize(+    ("caller_headers", "linked"),+    [({}, True), ({"traceparent": f"00-{_UPSTREAM_TRACERESPONSE.split('-')[1]}-00f067aa0ba902b7-01"}, False)],+)+def test_a_relayed_traceresponse_from_a_restarted_trace_is_linked(monkeypatch, caller_headers, linked):+    provider, exporter = _exporting_provider()++    TestClient(_traced_app(monkeypatch, provider)).get("/relayed", headers=caller_headers)++    links = _server_spans_by_route(exporter)["/relayed"].links+    upstream_ids = [f"00-{link.context.trace_id:032x}-{link.context.span_id:016x}-01" for link in links]+    assert upstream_ids == ([_UPSTREAM_TRACERESPONSE] if linked else [])+++def test_traceresponse_leaves_access_control_expose_headers_to_cors(monkeypatch):+    provider, _ = _exporting_provider()+    client = TestClient(_traced_app(monkeypatch, provider), raise_server_exceptions=False)++    responses = [client.get(path) for path in ("/success", "/crash", "/relayed-untraced")]++    assert all(response.headers.get_list("traceresponse") for response in responses)+    exposed = [response.headers.get_list("access-control-expose-headers") for response in responses]+    assert exposed == [[], [], ["x-upstream-request-id"]]+++def test_server_span_carries_the_call_id_its_response_returns(monkeypatch):+    provider, exporter = _exporting_provider()+    client = TestClient(_traced_app(monkeypatch, provider))++    responses = {path: client.get(path) for path in ("/success", "/failure", "/stream", "/relayed", "/tuple-headers")}++    server_spans = _server_spans_by_route(exporter)+    for path, response in responses.items():+        assert (server_spans[path].attributes or {})["litellm.call_id"] == response.headers["x-litellm-call-id"]+++def test_traceresponse_keeps_the_caller_trace(monkeypatch):+    provider, _ = _exporting_provider()+    caller_trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"++    response = TestClient(_traced_app(monkeypatch, provider)).get(+        "/success", headers={"traceparent": f"00-{caller_trace_id}-00f067aa0ba902b7-01"}+    )++    assert response.headers["traceresponse"].split("-")[1] == caller_trace_id+++def test_unsampled_requests_get_no_traceresponse(monkeypatch):+    client = TestClient(_traced_app(monkeypatch, TracerProvider(sampler=ALWAYS_OFF)), raise_server_exceptions=False)++    responses = [client.get(path) for path in ("/success", "/crash")]++    assert [response.headers.get_list("traceresponse") for response in responses] == [[], []]+++def test_websocket_handshakes_carry_no_traceresponse(monkeypatch):+    provider, _ = _exporting_provider()++    with TestClient(_traced_app(monkeypatch, provider)).websocket_connect("/ws") as websocket:+        assert websocket.receive_text() == "ok"+        handshake_header_names = [name for name, _ in websocket.extra_headers or ()]++    assert b"traceresponse" not in handshake_header_names+++def test_gate_off_responses_carry_no_traceresponse(monkeypatch):+    monkeypatch.delenv("LITELLM_OTEL_V2", raising=False)+    app = fastapi.FastAPI()++    @app.get("/ping")+    def ping():+        return {"ok": True}++    instrument_fastapi_app(app)++    assert "traceresponse" not in TestClient(app).get("/ping").headers+++def test_failed_instrumentation_registers_no_response_propagator(monkeypatch):+    monkeypatch.setenv("LITELLM_OTEL_V2", "1")+    app = fastapi.FastAPI()++    with TestClient(app):+        instrument_fastapi_app(app)++    assert get_global_response_propagator() is None+++def test_an_operator_response_propagator_is_kept(monkeypatch):+    class _OperatorPropagator(ResponsePropagator):+        def inject(self, carrier, context=None, setter=None):+            setter.set(carrier, "x-operator-trace", "kept")++    set_global_response_propagator(_OperatorPropagator())+    provider, _ = _exporting_provider()++    response = TestClient(_traced_app(monkeypatch, provider)).get("/relayed")++    assert response.headers["x-operator-trace"] == "kept"+    assert response.headers["traceresponse"] == _UPSTREAM_TRACERESPONSE+++def test_browsers_can_read_traceresponse_through_cors(monkeypatch):+    provider, _ = _exporting_provider()+    app = _traced_app(monkeypatch, provider)+    app.add_middleware(+        CORSMiddleware, allow_origins=["*"], expose_headers=[*LITELLM_UI_ALLOW_HEADERS, *trace_response_expose_headers()]+    )++    response = TestClient(app).get("/success", headers={"origin": "https://ui.example.com"})++    exposed = {name.strip() for name in response.headers["access-control-expose-headers"].split(",")}+    assert "traceresponse" in exposed+    assert response.headers["traceresponse"]+++def test_cors_leaves_traceresponse_unexposed_with_the_gate_off(monkeypatch):+    monkeypatch.delenv("LITELLM_OTEL_V2", raising=False)+    is_otel_v2_enabled.cache_clear()++    assert "traceresponse" not in [*LITELLM_UI_ALLOW_HEADERS, *trace_response_expose_headers()]+++def test_other_instrumented_apps_in_the_process_get_no_traceresponse(monkeypatch):+    provider, _ = _exporting_provider()+    litellm_client = TestClient(_traced_app(monkeypatch, provider))+    operator_app = fastapi.FastAPI()++    @operator_app.get("/operator")+    async def operator_route():+        return {"ok": True}++    FastAPIInstrumentor.instrument_app(operator_app, tracer_provider=provider)+    try:+        operator_response = TestClient(operator_app).get("/operator")+        litellm_response = litellm_client.get("/success")+    finally:+        FastAPIInstrumentor.uninstrument_app(operator_app)++    assert "traceresponse" not in operator_response.headers+    assert litellm_response.headers["traceresponse"]+++def test_the_unhandled_error_helper_names_the_span_it_is_given_in_any_context(monkeypatch):+    provider, _ = _exporting_provider()+    _traced_app(monkeypatch, provider)+    span = provider.get_tracer(__name__).start_span("POST /v1/chat/completions", kind=SpanKind.SERVER)+    context = span.get_span_context()++    headers = contextvars.Context().run(trace_response_headers, span)++    assert headers["traceresponse"] == f"00-{context.trace_id:032x}-{context.span_id:016x}-01"tests/unit/proxy/test_proxy_server_endpoints_and_startup.pyindex b981e8282bf..09646baa07f 100644@@ -102,7 +102,7 @@ def test_cors_exposes_cache_key_header_to_browser_js():     from litellm.constants import LITELLM_UI_ALLOW_HEADERS      cors_middleware = next(m for m in app.user_middleware if m.cls is CORSMiddleware)-    assert cors_middleware.kwargs["expose_headers"] is LITELLM_UI_ALLOW_HEADERS+    assert set(LITELLM_UI_ALLOW_HEADERS) <= set(cors_middleware.kwargs["expose_headers"])     assert "x-litellm-cache-key" in cors_middleware.kwargs["expose_headers"]