Backtestslitellmrun-20261111-220245#48304
auth failure handler 500s on any non-object JSON body (AttributeError: 'list' object has no attribute 'get')
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches stop the AttributeError at its source (the handler now works on a dict-or-empty view of the body) and both make the surrogate-repair fallback in read_request_body stop handing a raw list/scalar to callers, which is the path that still 500s on current main (the independent tests show only the NaN/Infinity/1e400 cases fail on base, i.e. the json.loads repair branch). All independent tests pass on both. The difference is scope and risk. Patch A is a tight fix: an isinstance guard in the handler, a small _json_object_body helper applied to both parse paths so a non-object body consistently yields {}, and tests at three layers (handler with every body shape from the issue including BudgetExceededError, end-to-end user_api_key_auth with raw bytes including the repair-fallback inputs, and read_request_body itself). Patch B goes much further: it turns pydantic ValidationError into a 400 inside read_request_body and then bolts on a route-sniffing carve-out (importing pass-through and lens modules into http_parsing_utils, inspecting request.scope['route'], checking APIRoute.body_field.is_required() and whether the stripped body equals b'null'). That heuristic is fragile and coupled: it had to add path_params to an unrelated vertex pass-through test to keep it working, it rewrites the protobuf and lens buffering tests, and any non-pass-through endpoint that legitimately accepts a list body would now get a 400 from the auth dependency with no carve-out. It also changes _parse_binary_body behaviour. None of that is required by the issue, which explicitly accepts 'the normal 401', and it adds surface for regressions that A does not. B's handler-side change (as_str_mapping, widening the parameter type to object) is fine, and its parametrized classification test is good, but the parsing-layer redesign counts against it. A resolves the issue at the root with less risk and adequate, non-tautological tests.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
27m 34s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
127vs 253 merged
Changed lines, ours vs merged
The issue
View on GitHubBug Report
Describe the bug
Any request whose JSON body is valid JSON but not an object ([], null, "s", 1, true) causes an
unhandled AttributeError and a 500 on unauthenticated endpoints. The auth failure handler assumes the
parsed body is a dict and calls .get() on it.
To Reproduce
No API key needed:
curl -i -X POST http://localhost:4000/v1/chat/completions \
-H 'Content-Type: application/json' -d '[]'
Result: 500 Internal Server Error. {} correctly gives 401, so the difference is purely the body type.
All of these 500: [], [1], [1,2], null, "s", 1, true, [[]], [[1]].
Traceback
File "litellm/proxy/auth/user_api_key_auth.py", line 2280, in user_api_key_auth_builder
return await UserAPIKeyAuthExceptionHandler.handle_authentication_error(
File "litellm/proxy/auth/auth_exception_handler.py", line 222, in handle_authentication_error
model=request_data.get("model"),
AttributeError: 'list' object has no attribute 'get'
Root cause
litellm/proxy/auth/auth_exception_handler.py:
seed_request_identity(
user_api_key_dict,
model=request_data.get("model"),
)
and again a few lines later:
budget_model: Final = request_data.get("model")
request_data is whatever json.loads produced, so it is only a dict when the client sent an object. There is
no isinstance(request_data, dict) guard.
Expected behavior
A non-object body on an unauthenticated request should give the normal 401 (or a 4xx validation error), not a
500 with a traceback.
Your Environment
- LiteLLM 1.106.3 (
litellm/litellm:latest); the same unguarded lines are present on currentmain - Linux, aarch64, official container image
- Default config, no database
Impact
Unauthenticated 500 + traceback per request, trivially triggerable by any client. Low severity (DoS-ish /
log noise), but it is on the auth-failure path, so it fires for every malformed client that sends an array.
The two patches
litellm/proxy/auth/auth_exception_handler.pyindex f6637a1fdf..aafbb08105 100644@@ -196,6 +196,9 @@ class UserAPIKeyAuthExceptionHandler: use_x_forwarded_for=general_settings.get("use_x_forwarded_for") is True, ) user_agent: Final = _get_user_agent(request)+ # Callers pass the decoded body through unchecked; a client that sent a JSON+ # array or scalar must still get its auth error, not a 500 from `.get()`.+ request_body: Final[dict[str, object]] = request_data if isinstance(request_data, dict) else {} # Log authentication failures before identity seeding and callbacks, so the log # survives a raising callback pipeline. Classify and route malformed virtual-key@@ -229,12 +232,12 @@ class UserAPIKeyAuthExceptionHandler: # without this the failed trace would carry no team/key attributes. seed_request_identity( user_api_key_dict,- model=request_data.get("model"),+ model=request_body.get("model"), ) # Budget checks live in tenant-scoped helpers (key / team / org / tag) # that don't see the request model, so the BudgetExceededError they- # raise carries `llm_provider=""`. Resolve it here off `request_data`+ # raise carries `llm_provider=""`. Resolve it here off `request_body` # so custom-callback consumers reading StandardLoggingPayload get # the same `llm_provider` attribution as for RPM/TPM 429s. if isinstance(e, litellm.BudgetExceededError) and not e.llm_provider:@@ -242,14 +245,14 @@ class UserAPIKeyAuthExceptionHandler: resolve_llm_provider_for_rate_limit, ) - budget_model: Final = request_data.get("model")+ budget_model: Final = request_body.get("model") _, e.llm_provider = resolve_llm_provider_for_rate_limit( budget_model if isinstance(budget_model, str) else None ) # Allow callbacks to transform the error response transformed_exception: Final = await proxy_logging_obj.post_call_failure_hook(- request_data=_with_client_context(request_data, requester_ip, user_agent),+ request_data=_with_client_context(request_body, requester_ip, user_agent), original_exception=e, user_api_key_dict=user_api_key_dict, error_type=ProxyErrorTypes.auth_error,litellm/proxy/common_utils/http_parsing_utils.pyindex a38769ac41..56deb5611d 100644@@ -173,6 +173,11 @@ def _parse_binary_body(body: bytes) -> dict: return {} +def _json_object_body(decoded: object) -> dict:+ """Only a JSON object carries request params; a top-level array, scalar or null parses to {}."""+ return dict(_JSON_REQUEST_BODY.validate_python(decoded)) if isinstance(decoded, dict) else {}++ def _declared_content_length(headers: Mapping[str, str]) -> int | None: declared: Final = headers.get("content-length") return int(declared) if isinstance(declared, str) and declared.isdigit() else None@@ -273,7 +278,7 @@ async def read_request_body(request: Request | None) -> dict: parsed_body = {} else: try:- parsed_body = dict(_JSON_REQUEST_BODY.validate_python(orjson.loads(body)))+ parsed_body = _json_object_body(orjson.loads(body)) except orjson.JSONDecodeError as e: # The surrogate-repair fallback below runs two full-body re.sub # passes, which block the event loop on multi-MB malformed bodies.@@ -299,8 +304,8 @@ async def read_request_body(request: Request | None) -> dict: body_str = re.sub(r"(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]", "", body_str) try:- parsed_body = json.loads(body_str)- json.dumps(parsed_body, ensure_ascii=False).encode("utf-8")+ repaired_body: Final = json.loads(body_str)+ json.dumps(repaired_body, ensure_ascii=False).encode("utf-8") except (json.JSONDecodeError, UnicodeEncodeError): # json.loads accepts lone surrogate escapes that no provider can encode verbose_proxy_logger.error("Invalid JSON payload received: %s", e)@@ -310,6 +315,7 @@ async def read_request_body(request: Request | None) -> dict: param="request_body", code=status.HTTP_400_BAD_REQUEST, )+ parsed_body = _json_object_body(repaired_body) # Cache the parsed result safe_set_request_parsed_body(request=request, parsed_body=parsed_body)tests/unit/proxy/auth/test_auth_exception_handler.pyindex d83a8aa2fc..237321bff4 100644@@ -1,5 +1,6 @@ import asyncio import json+from typing import Final from unittest.mock import AsyncMock, MagicMock, patch import httpx@@ -1070,6 +1071,54 @@ async def test_auth_failure_without_headers_scope_still_raises_original_error() assert "user_agent" not in mock_hook.call_args[1]["request_data"].get("metadata", {}) +_NON_OBJECT_JSON_BODIES: Final = [[], [1], [1, 2], [[]], [[1]], None, "s", 1, True]+++@pytest.mark.asyncio+@pytest.mark.parametrize("request_data", _NON_OBJECT_JSON_BODIES)+async def test_auth_failure_with_a_non_object_json_body_still_raises_the_401(request_data: object) -> None:+ with (+ patch( # test-quality-ok: handler reads proxy_server globals at call time+ "litellm.proxy.proxy_server.proxy_logging_obj.post_call_failure_hook",+ new_callable=AsyncMock,+ return_value=None,+ ) as mock_hook,+ patch( # test-quality-ok: handler reads proxy_server globals at call time+ "litellm.proxy.proxy_server.general_settings",+ {"allow_requests_on_db_unavailable": False},+ ),+ ):+ with pytest.raises(ProxyException) as exc_info:+ await UserAPIKeyAuthExceptionHandler.handle_authentication_error(+ Exception("Invalid proxy server token passed"),+ Request(scope={"type": "http", "headers": [(b"user-agent", b"array-client/1.0")]}),+ request_data, # type: ignore[arg-type] # the decoded body of a client that sent no JSON object+ "/v1/chat/completions",+ None,+ "",+ )++ assert int(exc_info.value.code) == status.HTTP_401_UNAUTHORIZED, exc_info.value.message+ assert mock_hook.call_args.kwargs["request_data"] == {"metadata": {"user_agent": "array-client/1.0"}}+++@pytest.mark.asyncio+@pytest.mark.parametrize("request_data", _NON_OBJECT_JSON_BODIES)+async def test_budget_exceeded_with_a_non_object_json_body_still_raises_the_budget_error(request_data: object) -> None:+ with pytest.raises(ProxyException) as exc_info:+ await UserAPIKeyAuthExceptionHandler.handle_authentication_error(+ BudgetExceededError(message="Budget exceeded", current_cost=100, max_budget=100),+ MagicMock(),+ request_data, # type: ignore[arg-type] # the decoded body of a client that sent no JSON object+ "/v1/chat/completions",+ None,+ "sk-over-budget",+ )++ assert exc_info.value.type == ProxyErrorTypes.budget_exceeded+ assert int(exc_info.value.code) == status.HTTP_422_UNPROCESSABLE_CONTENT++ def _marked_malformed_key_error() -> HTTPException: """Build the malformed-key 401 as its raise site does: marker stamped on it.""" error = HTTPException(status_code=401, detail="LiteLLM Virtual Key expected. Received=test")tests/unit/proxy/auth/test_user_api_key_auth.pyindex 60a8dd73a3..7364fda0a8 100644@@ -1310,6 +1310,40 @@ async def test_auth_rejects_wrong_or_empty_present_custom_header( assert exc_info.value.code == "401" +@pytest.mark.asyncio+@pytest.mark.parametrize(+ "body",+ [b"[]", b"[1]", b"[1,2]", b"[[]]", b"[[1]]", b"null", b'"s"', b"1", b"true", b"[NaN]", b"NaN"],+)+async def test_unauthenticated_request_with_a_non_object_json_body_is_rejected_with_401(+ body: bytes, monkeypatch: pytest.MonkeyPatch+):+ from fastapi import Request+ from starlette.datastructures import URL++ from litellm.proxy import proxy_server+ from litellm.proxy._types import ProxyException++ monkeypatch.setattr(proxy_server, "master_key", "sk-lit9211-master")+ monkeypatch.setattr(proxy_server, "general_settings", {})++ request = Request(+ scope={+ "type": "http",+ "method": "POST",+ "path": "/v1/chat/completions",+ "headers": [(b"content-type", b"application/json")],+ },+ receive=AsyncMock(return_value={"type": "http.request", "body": body, "more_body": False}),+ )+ request._url = URL(url="/v1/chat/completions")++ with pytest.raises(ProxyException) as exc_info:+ await user_api_key_auth(request=request, api_key="")++ assert exc_info.value.code == "401", exc_info.value.message++ @pytest.mark.asyncio async def test_user_api_key_from_query_param(): """Ensure user_api_key_auth reads API key from `key` query parameter."""tests/unit/proxy/common_utils/test_http_parsing_utils.pyindex cd5820fa2f..f538981657 100644@@ -643,6 +643,27 @@ async def test_lone_surrogate_escape_is_rejected_with_400(content: bytes): assert parsed["messages"][0]["content"] == "say ok \U0001f600" +@pytest.mark.asyncio+@pytest.mark.parametrize(+ "body",+ [+ pytest.param(b"[]", id="array"),+ pytest.param(b"[[1]]", id="nested-array"),+ pytest.param(b"null", id="null"),+ pytest.param(b'"s"', id="string"),+ pytest.param(b"1", id="number"),+ pytest.param(b"true", id="bool"),+ pytest.param(b"[NaN]", id="array-via-repair-fallback"),+ pytest.param(b"NaN", id="scalar-via-repair-fallback"),+ ],+)+async def test_non_object_json_body_parses_to_an_empty_dict(body: bytes):+ request = _make_json_request(body)+ parsed = await read_request_body(request)+ assert parsed == {}+ assert await read_request_body(request) == {}, "a non-object body must be cached as the same empty dict"++ @pytest.mark.asyncio @pytest.mark.parametrize("media_type", ["application/x-protobuf", "application/protobuf", "application/octet-stream"]) async def test_json_body_under_a_binary_content_type_is_still_parsed(media_type: str):
litellm/proxy/auth/auth_exception_handler.pyindex f6637a1fdf3..2eb25cc6105 100644@@ -12,7 +12,7 @@ import litellm from litellm._logging import verbose_proxy_logger, verbose_proxy_stdout_logger from litellm.constants import EMPTY_MAPPING from litellm.integrations.otel.runtime import seed_request_identity-from litellm.litellm_core_utils.core_helpers import is_expected_client_error+from litellm.litellm_core_utils.core_helpers import as_str_mapping, is_expected_client_error from litellm.proxy._types import ( LitellmUserRoles, ModelAccessDeniedProxyException,@@ -139,7 +139,7 @@ class UserAPIKeyAuthExceptionHandler: async def handle_authentication_error( e: Exception, request: Request,- request_data: dict[str, object],+ request_data: object, route: str, parent_otel_span: Span | None, api_key: str,@@ -190,6 +190,8 @@ class UserAPIKeyAuthExceptionHandler: request_route=route, ) else:+ request_mapping: Final = as_str_mapping(request_data)+ request_context: Final[dict[str, object]] = dict(request_mapping) if request_mapping is not None else {} # raise the exception to the caller requester_ip: Final = get_request_ip_address( request=request,@@ -229,7 +231,7 @@ class UserAPIKeyAuthExceptionHandler: # without this the failed trace would carry no team/key attributes. seed_request_identity( user_api_key_dict,- model=request_data.get("model"),+ model=request_context.get("model"), ) # Budget checks live in tenant-scoped helpers (key / team / org / tag)@@ -242,14 +244,14 @@ class UserAPIKeyAuthExceptionHandler: resolve_llm_provider_for_rate_limit, ) - budget_model: Final = request_data.get("model")+ budget_model: Final = request_context.get("model") _, e.llm_provider = resolve_llm_provider_for_rate_limit( budget_model if isinstance(budget_model, str) else None ) # Allow callbacks to transform the error response transformed_exception: Final = await proxy_logging_obj.post_call_failure_hook(- request_data=_with_client_context(request_data, requester_ip, user_agent),+ request_data=_with_client_context(request_context, requester_ip, user_agent), original_exception=e, user_api_key_dict=user_api_key_dict, error_type=ProxyErrorTypes.auth_error,litellm/proxy/common_utils/http_parsing_utils.pyindex a38769ac41b..9f7fed40ab3 100644@@ -6,7 +6,7 @@ from typing import Annotated, Any, Final, Literal, Union, get_args, get_origin import orjson from fastapi import Request, UploadFile, status-from pydantic import TypeAdapter+from pydantic import TypeAdapter, ValidationError from starlette._utils import get_route_path from starlette.requests import HTTPConnection from typing_extensions import NotRequired, ReadOnly, Required, assert_never@@ -162,13 +162,11 @@ def coerce_numeric_form_fields( } -def _parse_binary_body(body: bytes) -> dict:+def _parse_binary_body(body: bytes) -> dict[str, object]: """JSON sent under a binary content type still parses; real binary (protobuf) carries no params -> {}.""" try:- parsed: Final = _DECODED_JSON.validate_python(orjson.loads(body))- if isinstance(parsed, dict):- return parsed- except orjson.JSONDecodeError:+ return dict(_JSON_REQUEST_BODY.validate_python(orjson.loads(body)))+ except (orjson.JSONDecodeError, ValidationError): pass return {} @@ -299,8 +297,9 @@ async def read_request_body(request: Request | None) -> dict: body_str = re.sub(r"(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]", "", body_str) try:- parsed_body = json.loads(body_str)- json.dumps(parsed_body, ensure_ascii=False).encode("utf-8")+ decoded_body: Final = _DECODED_JSON.validate_python(json.loads(body_str))+ json.dumps(decoded_body, ensure_ascii=False).encode("utf-8")+ parsed_body = dict(_JSON_REQUEST_BODY.validate_python(decoded_body)) except (json.JSONDecodeError, UnicodeEncodeError): # json.loads accepts lone surrogate escapes that no provider can encode verbose_proxy_logger.error("Invalid JSON payload received: %s", e)@@ -315,6 +314,34 @@ async def read_request_body(request: Request | None) -> dict: safe_set_request_parsed_body(request=request, parsed_body=parsed_body) return parsed_body + except ValidationError as e:+ from fastapi.routing import APIRoute++ from litellm.proxy.auth.auth_utils import (+ request_dispatched_to_pass_through_endpoint,+ request_dispatched_to_provider_pass_through,+ )+ from litellm.proxy.lens.buffering import LensRoute++ route: Final[object] = request.scope.get("route") if request is not None else None+ if request is not None and (+ request_dispatched_to_pass_through_endpoint(request)+ or request_dispatched_to_provider_pass_through(request)+ or isinstance(route, LensRoute)+ or (+ isinstance(route, APIRoute)+ and route.body_field is not None+ and not route.body_field.field_info.is_required()+ and (await request.body()).strip() == b"null"+ )+ ):+ return {}+ raise ProxyException(+ message="Invalid JSON payload: expected an object",+ type="invalid_request_error",+ param="request_body",+ code=status.HTTP_400_BAD_REQUEST,+ ) from e except (json.JSONDecodeError, orjson.JSONDecodeError, ProxyException) as e: # Re-raise ProxyException as-is verbose_proxy_logger.error("Invalid JSON payload received: %s", e)tests/unit/proxy/auth/test_auth_exception_handler.pyindex d83a8aa2fcd..e21491fd996 100644@@ -1,5 +1,7 @@ import asyncio import json+from collections.abc import Mapping+from typing import Final from unittest.mock import AsyncMock, MagicMock, patch import httpx@@ -29,6 +31,7 @@ import litellm from litellm._logging import verbose_proxy_logger from litellm.constants import INVALID_VIRTUAL_KEY_ERROR_MARKER from litellm.exceptions import BudgetExceededError+from litellm.integrations.custom_logger import CustomLogger from litellm.proxy._types import ( ModelAccessDeniedProxyException, ProxyErrorTypes,@@ -812,6 +815,65 @@ def _http_request(client_host: str | None = "10.1.2.3", headers: dict[str, str] ) +class _AuthFailureRecorder(CustomLogger):+ def __init__(self) -> None:+ super().__init__()+ self.data: dict[str, object] | None = None+ self.error: Exception | None = None++ async def async_post_call_failure_hook(+ self,+ request_data: dict[str, object],+ original_exception: Exception,+ user_api_key_dict: UserAPIKeyAuth,+ traceback_str: str | None = None,+ ) -> HTTPException | None:+ self.data = dict(request_data)+ self.error = original_exception+ return None+++@pytest.mark.asyncio+@pytest.mark.parametrize("request_data", [[], None, "metadata", 1, True])+@pytest.mark.parametrize("error_kind", ["auth", "budget", "database"])+async def test_auth_error_preserves_classification_with_non_object_context(+ monkeypatch: pytest.MonkeyPatch, request_data: object, error_kind: str+) -> None:+ import litellm.proxy.proxy_server as proxy_server++ monkeypatch.setattr(litellm, "budget_exceeded_status_code", 429)+ errors: Final = {+ "auth": HTTPException(status_code=401, detail="Invalid API key"),+ "budget": BudgetExceededError(current_cost=2, max_budget=1),+ "database": httpx.ConnectError("Database unavailable"),+ }+ error: Final = errors[error_kind]+ expected: Final = _as_proxy_exception(error)+ recorder: Final = _AuthFailureRecorder()+ monkeypatch.setattr(litellm, "callbacks", [recorder])+ monkeypatch.setattr(proxy_server, "general_settings", {"allow_requests_on_db_unavailable": False})++ with pytest.raises(ProxyException) as exc_info:+ await UserAPIKeyAuthExceptionHandler.handle_authentication_error(+ error,+ _http_request(headers={"user-agent": "shape-regression"}),+ request_data,+ "/key/info",+ None,+ "sk-test",+ )++ assert exc_info.value.code == expected.code+ assert exc_info.value.type == expected.type+ assert exc_info.value.message == expected.message+ assert recorder.error is error+ assert recorder.data is not None+ metadata: Final = recorder.data["metadata"]+ assert isinstance(metadata, Mapping)+ assert metadata["requester_ip_address"] == "10.1.2.3"+ assert metadata["user_agent"] == "shape-regression"++ @pytest.mark.asyncio @pytest.mark.parametrize( "auth_error, general_settings, request_kwargs, expected_ip",tests/unit/proxy/auth/test_user_api_key_auth_request_flow.pyindex 108f7ce2983..2d9c781b0d1 100644@@ -7724,7 +7724,8 @@ async def test_user_api_key_auth_does_not_overwrite_end_user_id_set_by_builder() @pytest.mark.asyncio-async def test_user_api_key_auth_authenticates_before_raising_malformed_body_error():+@pytest.mark.parametrize("body", [b'{}{"model": "gpt-4o"}', b"[]", b"null", b"[1e400]"])+async def test_user_api_key_auth_authenticates_before_raising_malformed_body_error(body: bytes) -> None: """Regression (LIT-4780): a body that fails to parse must still be authenticated first, so the rejected request's trace carries the caller's key / team / user identity instead of an anonymous root span. The parse error is re-raised@@ -7742,10 +7743,11 @@ async def test_user_api_key_auth_authenticates_before_raising_malformed_body_err "headers": [(b"content-type", b"application/json")], "method": "POST", "path": "/chat/completions",+ "query_string": b"probe=1", } ) request._url = URL(url="/chat/completions")- request._body = b'{}{"model": "gpt-4o"}'+ request._body = body attrs = _proxy_attrs_for_centralized_checks(user_custom_auth=None) originals = {a: getattr(_proxy_server_mod, a, None) for a in attrs}tests/unit/proxy/common_utils/test_http_parsing_utils.pyindex cd5820fa2fd..354feb0bab1 100644@@ -64,6 +64,100 @@ def _starlette_request( return Request(scope, receive) +@pytest.mark.asyncio+@pytest.mark.parametrize(+ "body",+ [b"[]", b'[["model", "test"]]', b"null", b'"s"', b"1", b"true", b"false"],+)+@pytest.mark.parametrize("path", ["/v1/messages", "/vertex_ai/v1/messages", "/lens/datasets"])+async def test_read_request_body_rejects_non_object_json(body: bytes, path: str) -> None:+ request: Final = _starlette_request(body, "application/json", path)++ for _ in range(2):+ with pytest.raises(ProxyException) as exc_info:+ await read_request_body(request)+ assert exc_info.value.code == "400"+ assert exc_info.value.type == "invalid_request_error"+ assert exc_info.value.param == "request_body"+ assert exc_info.value.message == "Invalid JSON payload: expected an object"+ assert _safe_get_request_parsed_body(request) is None+ assert await request.body() == body+++@pytest.mark.asyncio+@pytest.mark.parametrize("body", [b"[1e400]", b"1e400", b"[NaN]"])+async def test_read_request_body_rejects_non_objects_from_json_fallback(body: bytes) -> None:+ with pytest.raises(ProxyException) as exc_info:+ await read_request_body(_starlette_request(body, "application/json"))+ assert exc_info.value.code == "400"+ assert exc_info.value.message == "Invalid JSON payload: expected an object"+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ "body, invalid_unicode",+ [+ (b"[]", False),+ (b"null", False),+ (b'"value"', False),+ (b"[1e400]", False),+ (b'"\\ud800"', True),+ (b'["\\ud800"]', True),+ (b'{"value":"\\ud800"}', True),+ ],+)+@pytest.mark.parametrize("dispatch", ["custom", "provider", "lens"])+async def test_pass_through_body_validation_preserves_bytes(dispatch: str, body: bytes, invalid_unicode: bool) -> None:+ from fastapi.routing import APIRoute+ from litellm.proxy.lens.adapter import lens_request+ from litellm.proxy.lens.buffering import LensRoute+ from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import vertex_proxy_route+ from litellm.proxy.pass_through_endpoints.pass_through_endpoints import create_pass_through_route++ path: Final = {"custom": "/v1/messages", "provider": "/vertex_ai/predict", "lens": "/lens/datasets"}[dispatch]+ route: Final = {+ "custom": APIRoute(path, create_pass_through_route(path, "https://upstream.test"), methods=["POST"]),+ "provider": APIRoute("/vertex_ai/{endpoint:path}", vertex_proxy_route, methods=["POST"]),+ "lens": LensRoute("/lens/{path:path}", lens_request, methods=["POST"]),+ }[dispatch]+ incoming: Final = _starlette_request(body, "application/json", path)+ _, dispatched_scope = route.matches(incoming.scope)+ request: Final = Request({**incoming.scope, **dispatched_scope}, incoming.receive)++ for _ in range(2):+ if invalid_unicode:+ with pytest.raises(ProxyException) as exc_info:+ await read_request_body(request)+ assert exc_info.value.code == "400"+ else:+ assert await read_request_body(request) == {}+ assert await request.body() == body+++@pytest.mark.asyncio+@pytest.mark.parametrize("optional", [False, True])+@pytest.mark.parametrize("body", [b"null", b" \nnull\t", b"[]", b'"value"'])+async def test_null_body_respects_the_dispatched_body_requirement(optional: bool, body: bytes) -> None:+ from fastapi.routing import APIRoute+ from litellm.proxy.management_endpoints.key_management_endpoints import generate_key_fn+ from litellm.proxy.spend_tracking.spend_management_endpoints import global_spend_end_users++ endpoint: Final = global_spend_end_users if optional else generate_key_fn+ route: Final = APIRoute("/audit", endpoint, methods=["POST"])+ incoming: Final = _starlette_request(body, "application/json", "/audit")+ _, dispatched_scope = route.matches(incoming.scope)+ request: Final = Request({**incoming.scope, **dispatched_scope}, incoming.receive)++ if optional and body.strip() == b"null":+ assert await read_request_body(request) == {}+ assert await request.json() is None+ else:+ with pytest.raises(ProxyException) as exc_info:+ await read_request_body(request)+ assert exc_info.value.code == "400"+ assert await request.body() == body++ @pytest.mark.asyncio async def test_read_request_body_marks_body_received_once_with_its_size(monkeypatch: pytest.MonkeyPatch): events: list[tuple[str, dict[str, str | int]]] = [] # mutable-ok: recorder for the injected phase_event double@@ -131,14 +225,20 @@ async def test_read_raw_json_body_is_none_for_form_bodies(): @pytest.mark.asyncio @pytest.mark.parametrize("content_type", ["application/x-protobuf", "application/protobuf; charset=binary"])-async def test_protobuf_body_is_not_parsed_as_json(content_type):- # OTLP trace exports (POST /v1/traces) are binary protobuf; arbitrary bytes like these- # used to hit the JSON surrogate-repair path and fail auth with a 400.- body = b"\n\xa2\x01\n\x1c\n\x0cservice.name\x12\x0c\n\nswarm\xed\xa0\x80\xff"- request = _starlette_request(body, content_type)+@pytest.mark.parametrize(+ "body",+ [+ b"\n\xa2\x01\n\x1c\n\x0cservice.name\x12\x0c\n\nswarm\xed\xa0\x80\xff",+ pytest.param(b"100000000", id="protobuf-also-json-number"),+ ],+)+async def test_protobuf_body_is_not_parsed_as_json(content_type: str, body: bytes) -> None:+ request: Final = _starlette_request(body, content_type) assert await read_request_body(request) == {}- assert await request.body() == body # body is still readable by the endpoint+ assert await read_request_body(request) == {}+ assert await read_raw_json_body(request) == body+ assert await request.body() == body @pytest.mark.asynciotests/unit/proxy/lens/test_buffering.pyindex b544b70d36b..2b868e72096 100644@@ -249,7 +249,10 @@ async def test_admission_releases_upload_on_disconnect_or_cancellation(cancel: b @pytest.mark.asyncio @pytest.mark.parametrize("reject_auth", [False, True])-async def test_admitted_bytes_survive_auth_parsing_and_remain_reserved_until_sent(reject_auth: bool) -> None:+@pytest.mark.parametrize("body", [b" {} ", b"[]", b"null", b'"s"'])+async def test_admitted_bytes_survive_auth_parsing_and_remain_reserved_until_sent(+ reject_auth: bool, body: bytes+) -> None: app: Final = FastAPI() router: Final = APIRouter(route_class=LensRoute)@@ -264,9 +267,11 @@ async def test_admitted_bytes_survive_auth_parsing_and_remain_reserved_until_sen if reject_auth: raise HTTPException(401, "Invalid key") - async with httpx.AsyncClient(- transport=httpx.MockTransport(lambda _: httpx.Response(200, content=b"result"))- ) as upstream:+ def upstream_response(incoming: httpx.Request) -> httpx.Response:+ assert incoming.content == body+ return httpx.Response(200, content=b"result")++ async with httpx.AsyncClient(transport=httpx.MockTransport(upstream_response)) as upstream: @router.post("/lens/datasets") async def endpoint(request: Request, auth: Annotated[None, Depends(auth_reader)]) -> Response:@@ -284,14 +289,14 @@ async def test_admitted_bytes_survive_auth_parsing_and_remain_reserved_until_sen sending.set() await finish.wait() - incoming: Final = request(b" {} ")+ incoming: Final = request(body) task: Final = asyncio.create_task(app(incoming.scope, incoming.receive, send)) try: await asyncio.wait_for(sending.wait(), 2) async with httpx.AsyncClient( transport=httpx.ASGITransport(app=app), base_url="http://gateway.test" ) as client:- blocked: Final = await client.post("/lens/datasets", content=b" {} " * 2)+ blocked: Final = await client.post("/lens/datasets", content=b" " * (11 - len(body))) unrelated: Final = await client.post("/ordinary", content=b"ordinary body") assert blocked.status_code == 503 assert unrelated.content == b"ordinary body"tests/unit/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.pyindex 0186a33dbc4..60043daa402 100644@@ -4357,6 +4357,7 @@ class TestVertexRawPredictStreamingClassification: "type": "http", "method": "POST", "path": f"/vertex_ai/{endpoint}",+ "path_params": {"endpoint": endpoint}, "headers": [ (b"content-type", b"application/json"), (b"x-litellm-api-key", b"test-key"),