Backtestslitellmrun-20261018-182052#41421
[Bug]: LiteLLM Proxy 1.100.0 serializes image-edit uploads as strings for OpenAI GPT Image models
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the actual defect in the same way: the raw `image[]` / `mask[]` form parts remain in the parsed body after FastAPI binds them to the `image_array`/`mask_array` parameters, so they are forwarded as extra provider params and stringified into text multipart fields. A pops them from `data`; B filters them out in a dict comprehension. Both pass all three independent tests that fail on base, so correctness on the reported scenario is equal. The differences are in tests and extra hardening. Patch A's regression test is materially stronger: it drives the request through the proxy endpoint and then through `litellm.aimage_edit` with an httpx MockTransport, and asserts on the actual multipart body sent to OpenAI (image[] and mask parts are file parts with the original bytes, no `UploadFile`/`BytesIO` repr anywhere), parameterized over both `image[]` and `image` spellings. That directly mirrors the failure the issue reports at the wire level. Patch B's tests only inspect the kwargs captured at `route_request`; they do fail on base (since `image[]` would be present) and cover mask[] and the canonical fields, but stop short of the provider serialization that actually produced the OpenAI error. Patch A also makes `_form_field_value` raise a TypeError for bytes/file-like values instead of silently shipping a repr string. This is scope beyond the ask and carries a small risk of turning a previously-garbage request into an exception for any other caller, but on base such a request was already broken, so failing loudly is a reasonable guard against recurrence and is covered by unit tests. Patch B, by contrast, includes unrelated formatting churn in existing tests (reflowing `monkeypatch.setattr` lines and function signatures), which is neutral-to-slightly-negative noise. Craft is comparable: B's named constants and comprehension are tidy; A's comment explains why the pop is needed. On balance A wins on the stronger, end-to-end regression test that targets the exact symptom in the issue, with the hardening being acceptable and tested.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
35m 50s
Wall-clock time in the container
Full test suite
No new failures
424 passed · 0 skipped · 424 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
85vs 88 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
Image edits through the LiteLLM Proxy fail in version 1.100.0 for OpenAI GPT Image models. The uploaded image file is being converted to a string representation of a Python BytesIO object ("<_io.BytesIO object at 0x7f0d0bc73600>") before being sent to OpenAI.
OpenAI rejects this with:
{
"error": {
"message": "Invalid type for 'image[0]': expected a file, but got a string instead.",
"type": "invalid_request_error",
"param": "image[0]",
"code": "invalid_type"
}
}
The same request succeeds in LiteLLM version 1.95.0.
User Flow
Before a fix: A user sends an image-edit request through LiteLLM Proxy 1.100.0 using a multipart/form-data request to POST /v1/images/edits with an uploaded image file and a prompt (e.g., "Make this image photorealistic"). LiteLLM receives the UploadFile correctly but serializes it to a string before forwarding to OpenAI. OpenAI rejects the request because it expects a file object, not a string representation.
- User sends:
curl -X POST "http://localhost:4000/v1/images/edits" -F "model=openai/gpt-image-2" -F "prompt=Make this image photorealistic" -F "image[]=@./input.png;type=image/png" - LiteLLM receives UploadFile with filename, size, and content-type headers
- LiteLLM converts the UploadFile's BytesIO object to a string
- LiteLLM forwards stringified image to OpenAI
- OpenAI rejects with invalid_type error
After a fix: The same user flow succeeds. LiteLLM preserves the UploadFile/BytesIO object and forwards it to OpenAI as a multipart file, and OpenAI processes it successfully.
Proof the bug occurs
Config / setup the proxy ran with:
Default LiteLLM Proxy setup with OpenAI API key configured
Version or commit:
1.100.0 (regression; works in 1.95.0)
Commands and their full output:
curl -X POST "http://localhost:4000/v1/images/edits" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-F "model=openai/gpt-image-2" \
-F "prompt=Make this image photorealistic" \
-F "image[]=@./input.png;type=image/png"
LiteLLM request logging shows:
- Receives:
image[]: UploadFile(filename='...png', size=2014638, headers=Headers({'content-type': 'image/png', 'content-disposition': 'form-data; name="image[]"; filename="...png"'})) - Sends to OpenAI:
{"image": ["<_io.BytesIO object at 0x7f0d0bc73600>"], "model": "openai/gpt-image-2", "prompt": "..."}
OpenAI error response:
{
"error": {
"message": "Invalid type for 'image[0]': expected a file, but got a string instead.",
"type": "invalid_request_error",
"param": "image[0]",
"code": "invalid_type"
}
}
Stack trace passes through: litellm/proxy/image_endpoints/endpoints.py:313 and litellm/images/main.py:1038
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
1.100.0
Twitter / LinkedIn details
_No response_
The two patches
litellm/litellm_core_utils/llm_request_utils.pyindex 04824a5bf3..2ab16af124 100644@@ -6,6 +6,9 @@ from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH def _form_field_value(value: object) -> str:+ if isinstance(value, (bytes, bytearray, memoryview)) or callable(getattr(value, "read", None)):+ # str() of a file would ship its repr (e.g. "<_io.BytesIO object at ...>") as text+ raise TypeError(f"file-like value of type {type(value).__name__} cannot be sent as a text form field") if value is True: return "true" if value is False:litellm/proxy/image_endpoints/endpoints.pyindex 5b90c0ff83..9e1ac7e0da 100644@@ -300,6 +300,11 @@ async def image_edit_api( numeric_fields=IMAGE_EDIT_NUMERIC_FORM_FIELDS, ) )+ # The raw ``image[]`` / ``mask[]`` form parts are normalized into ``image`` / ``mask``+ # below; left in ``data`` they would reach the provider as extra params and be sent as+ # text fields carrying the UploadFile's repr instead of the file.+ for _array_field in ("image[]", "mask[]"):+ data.pop(_array_field, None) image_files: Final = await batch_to_bytesio(image) mask_files: Final = await batch_to_bytesio(mask) if image_files:tests/test_litellm/litellm_core_utils/test_llm_request_utils.pyindex 765c47547c..3f75e1242c 100644@@ -1,3 +1,5 @@+import io+ import httpx import pytest @@ -105,3 +107,18 @@ def test_flatten_form_field_values_rejects_over_deep_nesting(): assert isinstance(nested, dict) with pytest.raises(ValueError, match="max depth"): flatten_form_field_values(nested)+++@pytest.mark.parametrize(+ "file_value",+ [io.BytesIO(b"\x89PNG"), b"\x89PNG", [io.BytesIO(b"\x89PNG")]],+ ids=["bytesio", "bytes", "list-of-bytesio"],+)+def test_flatten_form_field_values_refuses_to_stringify_a_file(file_value):+ with pytest.raises(TypeError, match="file-like value"):+ flatten_form_field_values({"image[]": file_value})+++def test_serialize_multipart_form_fields_refuses_to_stringify_a_file():+ with pytest.raises(TypeError, match="file-like value"):+ serialize_multipart_form_fields({"input_reference": io.BytesIO(b"\x89PNG")})tests/test_litellm/proxy/image_endpoints/test_endpoints.pyindex 31b87530c9..a223597d16 100644@@ -328,3 +328,63 @@ async def test_failure_before_the_provider_call_bills_the_callers_litellm_call_i assert raised.value.headers["x-litellm-call-id"] == call_id assert [data["litellm_call_id"] for data in hook_request_data] == [call_id]+++def _multipart_parts(content_type: str, body: bytes) -> list[tuple[str, bool, bytes]]:+ """(field name, is a file part, payload) for each part of a multipart body."""+ boundary = content_type.split("boundary=")[1].encode()+ parts = []+ for part in body.split(b"--" + boundary):+ if b'name="' not in part:+ continue+ headers, _, payload = part.partition(b"\r\n\r\n")+ name = headers.split(b'name="')[1].split(b'"')[0].decode()+ parts.append((name, b"filename=" in headers, payload.rstrip(b"\r\n-")))+ return parts+++@pytest.mark.parametrize("image_field", ["image[]", "image"])+def test_image_edit_upload_reaches_openai_as_a_file_not_a_string(monkeypatch, image_field):+ """+ Regression: an ``image[]`` upload was left in the request data as a raw UploadFile, forwarded as+ an extra provider param and sent to OpenAI as a text ``image[]`` part, which OpenAI rejects with+ "Invalid type for 'image[0]': expected a file, but got a string instead".+ """+ import httpx++ import litellm+ from litellm.llms.custom_httpx.http_handler import AsyncHTTPHandler++ png = b"\x89PNG\r\n\x1a\nimage-bytes"+ mask_png = b"\x89PNG\r\n\x1a\nmask-bytes"+ mask_field = "mask[]" if image_field == "image[]" else "mask"+ captured: Dict[str, Any] = {}+ response = _image_edit_client(monkeypatch, captured).post(+ "/v1/images/edits",+ files=[+ (image_field, ("input.png", png, "image/png")),+ (mask_field, ("mask.png", mask_png, "image/png")),+ ],+ data={"model": "openai/gpt-image-2", "prompt": "Make this image photorealistic"},+ )+ assert response.status_code == 200+ assert "image[]" not in captured and "mask[]" not in captured++ sent: Dict[str, Any] = {}++ def respond(request: httpx.Request) -> httpx.Response:+ sent["content_type"] = request.headers["content-type"]+ sent["body"] = request.content+ return httpx.Response(200, json={"created": 1, "data": [{"b64_json": "aW1n"}]})++ client = AsyncHTTPHandler()+ client.client = httpx.AsyncClient(transport=httpx.MockTransport(respond))+ asyncio.run(+ litellm.aimage_edit(**captured, api_key="sk-test", api_base="https://openai.example/v1", client=client)+ )++ parts = _multipart_parts(sent["content_type"], sent["body"])+ assert [(name, payload) for name, is_file, payload in parts if name == "image[]"] == [("image[]", png)]+ assert all(is_file for name, is_file, _ in parts if name in ("image[]", "mask"))+ assert [payload for name, _, payload in parts if name == "mask"] == [mask_png]+ assert not any(b"UploadFile" in payload or b"BytesIO" in payload for _, _, payload in parts)
litellm/proxy/image_endpoints/endpoints.pyindex 5b90c0ff830..b9580ba3948 100644@@ -36,6 +36,10 @@ router: Final = APIRouter() IMAGE_EDIT_NUMERIC_FORM_FIELDS: Final = numeric_form_fields(get_type_hints(ImageEditRequestParams)) +IMAGE_ARRAY_FIELD: Final = "image[]"+MASK_ARRAY_FIELD: Final = "mask[]"+BRACKETED_FILE_FIELDS: Final = frozenset({IMAGE_ARRAY_FIELD, MASK_ARRAY_FIELD})+ async def uploadfile_to_bytesio(upload: UploadFile) -> io.BytesIO: """@@ -244,9 +248,9 @@ async def image_edit_api( fastapi_response: Response, user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth), image: list[UploadFile] | None = File(None),- image_array: list[UploadFile] | None = File(None, alias="image[]"),+ image_array: list[UploadFile] | None = File(None, alias=IMAGE_ARRAY_FIELD), mask: list[UploadFile] | None = File(None),- mask_array: list[UploadFile] | None = File(None, alias="mask[]"),+ mask_array: list[UploadFile] | None = File(None, alias=MASK_ARRAY_FIELD), model: str | None = None, ): """@@ -294,12 +298,14 @@ async def image_edit_api( ######################################################### # Read request body and convert UploadFiles to BytesIO #########################################################- data: Final = dict(- coerce_numeric_form_fields(+ data: Final = {+ key: value+ for key, value in coerce_numeric_form_fields( parsed_body=await _read_request_body(request=request), numeric_fields=IMAGE_EDIT_NUMERIC_FORM_FIELDS,- )- )+ ).items()+ if key not in BRACKETED_FILE_FIELDS+ } image_files: Final = await batch_to_bytesio(image) mask_files: Final = await batch_to_bytesio(mask) if image_files:tests/test_litellm/proxy/image_endpoints/test_endpoints.pyindex 31b87530c94..ad0901e9eee 100644@@ -95,18 +95,14 @@ async def test_image_generation_prompt_rerouting(monkeypatch): monkeypatch.setattr("litellm.proxy.proxy_server.general_settings", {}) monkeypatch.setattr("litellm.proxy.proxy_server.llm_router", None) monkeypatch.setattr("litellm.proxy.proxy_server.proxy_config", {})- monkeypatch.setattr(- "litellm.proxy.proxy_server.proxy_logging_obj", fake_proxy_logger- )+ monkeypatch.setattr("litellm.proxy.proxy_server.proxy_logging_obj", fake_proxy_logger) monkeypatch.setattr("litellm.proxy.proxy_server.user_model", None) monkeypatch.setattr("litellm.proxy.proxy_server.version", "test-version") monkeypatch.setattr( "litellm.proxy.common_request_processing.ProxyBaseLLMRequestProcessing.get_custom_headers", classmethod(lambda *args, **kwargs: {}), )- monkeypatch.setattr(- "litellm.proxy.image_endpoints.endpoints.route_request", fake_route_request- )+ monkeypatch.setattr("litellm.proxy.image_endpoints.endpoints.route_request", fake_route_request) result = await endpoints.image_generation( request=request,@@ -141,6 +137,60 @@ def _image_edit_client(monkeypatch, captured: Dict[str, Any]) -> TestClient: return TestClient(app) +def test_image_edit_image_array_alias_is_not_forwarded(monkeypatch):+ """The documented `image[]` alias must reach the provider only as `image`."""+ captured: Dict[str, Any] = {}++ response = _image_edit_client(monkeypatch, captured).post(+ "/v1/images/edits",+ files={"image[]": ("tree.png", b"\x89PNG\r\n\x1a\ntree", "image/png")},+ data={"model": "gpt-image-1", "prompt": "add a hat"},+ )++ assert response.status_code == 200+ assert "image[]" not in captured+ assert [buffer.getvalue() for buffer in captured["image"]] == [b"\x89PNG\r\n\x1a\ntree"]+ assert [buffer.name for buffer in captured["image"]] == ["tree.png"]+++def test_image_edit_mask_array_alias_is_not_forwarded(monkeypatch):+ """`mask[]` has the same shape as `image[]` and must be dropped the same way."""+ captured: Dict[str, Any] = {}++ response = _image_edit_client(monkeypatch, captured).post(+ "/v1/images/edits",+ files={+ "image": ("tree.png", b"\x89PNG\r\n\x1a\ntree", "image/png"),+ "mask[]": ("mask.png", b"\x89PNG\r\n\x1a\nmask", "image/png"),+ },+ data={"model": "gpt-image-1", "prompt": "add a hat"},+ )++ assert response.status_code == 200+ assert "mask[]" not in captured+ assert [buffer.getvalue() for buffer in captured["mask"]] == [b"\x89PNG\r\n\x1a\nmask"]+ assert [buffer.getvalue() for buffer in captured["image"]] == [b"\x89PNG\r\n\x1a\ntree"]+++def test_image_edit_canonical_file_fields_still_reach_the_provider(monkeypatch):+ """Dropping the bracketed aliases must not touch the canonical fields."""+ captured: Dict[str, Any] = {}++ response = _image_edit_client(monkeypatch, captured).post(+ "/v1/images/edits",+ files={+ "image": ("tree.png", b"\x89PNG\r\n\x1a\ntree", "image/png"),+ "mask": ("mask.png", b"\x89PNG\r\n\x1a\nmask", "image/png"),+ },+ data={"model": "gpt-image-1", "prompt": "add a hat"},+ )++ assert response.status_code == 200+ assert [buffer.getvalue() for buffer in captured["image"]] == [b"\x89PNG\r\n\x1a\ntree"]+ assert [buffer.getvalue() for buffer in captured["mask"]] == [b"\x89PNG\r\n\x1a\nmask"]+ assert captured["prompt"] == "add a hat"++ def test_image_edit_multipart_n_reaches_the_provider_as_an_int(monkeypatch): """A multipart `n` must not arrive as the string Starlette parsed it into.""" captured: Dict[str, Any] = {}@@ -180,7 +230,9 @@ async def test_a_model_the_router_cannot_serve_answers_an_openai_typed_error(mon async def fake_add_litellm_data_to_request(**kwargs: object) -> object: return kwargs["data"] - async def fake_pre_call_hook(*, user_api_key_dict: UserAPIKeyAuth, data: dict[str, object], call_type: str) -> dict[str, object]:+ async def fake_pre_call_hook(+ *, user_api_key_dict: UserAPIKeyAuth, data: dict[str, object], call_type: str+ ) -> dict[str, object]: return data async def fake_post_call_failure_hook(**_: object) -> None:@@ -211,7 +263,9 @@ async def test_a_model_the_router_cannot_serve_answers_an_openai_typed_error(mon request = Request({"type": "http", "method": "POST", "path": "/v1/images/generations", "headers": []}, receive) with pytest.raises(ProxyException) as raised:- await endpoints.image_generation(request=request, fastapi_response=Response(), user_api_key_dict=UserAPIKeyAuth())+ await endpoints.image_generation(+ request=request, fastapi_response=Response(), user_api_key_dict=UserAPIKeyAuth()+ ) assert (raised.value.type, raised.value.param, raised.value.code) == ("invalid_request_error", None, "404")