Backtestslitellmrun-20261019-033420#25985
[Bug]: Ollama chat transformation fails with JSONDecodeError 'Unterminated string' when parsing tool call arguments
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches resolve the reported crash: the independent tests show both now raise an error containing the raw arguments and log them. Beyond that, B is the stronger fix. Patch A's test diff is defective: it replaces the `def test_tools_passed_directly_without_capability_check` line with the new tests, leaving the old test's docstring and body dangling inside `test_transform_request_valid_tool_call_arguments_passes`, so an existing regression test silently stops running as its own test (the "removing 1 line" from test files confirms this). A also raises `BadRequestError` with a hardcoded `model="ollama"` rather than the actual model, and only fixes the `ollama_chat` path while the identical `json.loads(call["function"]["arguments"])` in `ollama_pt` (used by `ollama/` models) is left unguarded. Patch B centralizes parsing in a shared helper used by both `ollama_chat` transformation and `ollama_pt`, handles None/empty/dict/non-string arguments, repairs trivially truncated JSON (the issue's optional item 3), preserves the tool name and raw arguments in a 400 `OllamaError`, and adds exception mapping so the error surfaces as a non-retryable `BadRequestError` rather than being re-matched by substring checks on the embedded raw output (a real hazard it explicitly tests). B's tests are extensive and meaningful: unit-level helper cases, transform_request cases, and `litellm.completion`-level tests for both `ollama/` and `ollama_chat/` with `max_retries=0`. B's larger scope is justified by covering the same root cause in the sibling code path; its string-prefix match in exception mapping is somewhat brittle, and auto-repair carries a small risk of silently forwarding partial data, but these are minor relative to A's broken test file and incomplete coverage.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
17m 33s
Wall-clock time in the container
Full test suite
No new failures
860 passed · 8 skipped · 868 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
211vs 93 merged
Changed lines, ours vs merged
The issue
View on GitHubBug Description
When using LiteLLM proxy with an Ollama-compatible provider (custom model glm-5), the request transformation fails with a JSONDecodeError: Unterminated string error when parsing tool call arguments. The error occurs in litellm/llms/ollama/chat/transformation.py at line 274.
Steps to Reproduce
- Configure LiteLLM proxy with an Ollama-compatible provider (e.g., custom local model server)
- Send a chat completion request that includes tool/function definitions
- The upstream model returns a tool call with arguments
- LiteLLM attempts to transform the request and fails to parse the tool arguments JSON
Expected Behavior
LiteLLM should gracefully handle tool call arguments, either:
- Successfully parse valid JSON arguments
- Handle malformed/incomplete JSON with proper error handling (catch exception, log the actual arguments string for debugging, and fail with a more informative error message)
Actual Behavior
LiteLLM crashes with:
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)
The error is raised at litellm/llms/ollama/chat/transformation.py:274:
arguments = json.loads(typed_tool["function"]["arguments"])
Root Cause Analysis
The code assumes typed_tool["function"]["arguments"] is always valid JSON. However, the upstream model may return:
- Incomplete/truncated JSON (e.g., due to streaming issues or token limits)
- Malformed JSON with unescaped characters
- Partial tool call arguments
The current implementation does not handle these edge cases.
Relevant Log Output
2026-04-17 18:28:21.669 | 23:28:21 - LiteLLM Proxy:ERROR: common_request_processing.py:1522 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.APIConnectionError: Unterminated string starting at: line 1 column 10 (char 9)
2026-04-17 18:28:21.669 | Traceback (most recent call last):
2026-04-17 18:28:21.669 | File "/usr/lib/python3.13/site-packages/litellm/main.py", line 4015, in completion
2026-04-17 18:28:21.669 | response = base_llm_http_handler.completion(
2026-04-17 18:28:21.669 | model=model,
2026-04-17 18:28:21.669 | ...<14 lines>...
2026-04-17 18:28:21.669 | client=client,
2026-04-17 18:28:21.669 | )
2026-04-17 18:28:21.669 | File "/usr/lib/python3.13/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 410, in completion
2026-04-17 18:28:21.669 | data = provider_config.transform_request(
2026-04-17 18:28:21.669 | model=model,
2026-04-17 18:28:21.669 | ...<3 lines>...
2026-04-17 18:28:21.669 | headers=headers,
2026-04-17 18:28:21.669 | )
2026-04-17 18:28:21.669 | File "/usr/lib/python3.13/site-packages/litellm/llms/ollama/chat/transformation.py", line 274, in transform_request
2026-04-17 18:28:21.669 | arguments = json.loads(typed_tool["function"]["arguments"])
2026-04-17 18:28:21.669 | File "/usr/lib/python3.13/json/__init__.py", line 352, in loads
2026-04-17 18:28:21.669 | return _default_decoder.decode(s)
2026-04-17 18:28:21.669 | ~~~~~~~~~~~~~~~~~~~~~^^^
2026-04-17 18:28:21.669 | File "/usr/lib/python3.13/json/decoder.py", line 345, in decode
2026-04-17 18:28:21.669 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2026-04-17 18:28:21.669 | ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
2026-04-17 18:28:21.669 | File "/usr/lib/python3.13/json/decoder.py", line 361, in raw_decode
2026-04-17 18:28:21.669 | obj, end = self.scan_once(s, idx)
2026-04-17 18:28:21.669 | ~~~~~~~~~~~~~~^^^^^^^^
2026-04-17 18:28:21.669 | json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 10 (char 9)
2026-04-17 18:28:21.669 | . Received Model Group=glm-5
2026-04-17 18:28:21.669 | Available Model Group Fallbacks=None LiteLLM Retried: 3 times, LiteLLM Max Retries: 3
Environment
- LiteLLM Proxy running via Docker
- Python 3.13
- Model: glm-5 (custom local provider via Ollama-compatible endpoint)
Suggested Fix
Add try/except around the JSON parsing in transformation.py to:
- Log the actual arguments string that failed to parse (for debugging)
- Return a more informative error message including the raw arguments string
- Optionally, attempt to repair common JSON issues (truncated strings, missing closing braces)
try:
arguments = json.loads(typed_tool["function"]["arguments"])
except json.JSONDecodeError as e:
# Log the malformed arguments for debugging
verbose_logger.error(f"Failed to parse tool call arguments: {typed_tool['function']['arguments']!r}")
raise litellm.InternalServerError(
f"Model returned malformed tool call arguments: {e.msg}. Raw arguments: {typed_tool['function']['arguments']!r}"
)
Related Issues
- #13823 - Similar JSONDecodeError in Ollama completion transformation (response side)
- #11267 - APIConnectionError parsing Tool call response from Ollama
- #18920 - BadRequestError with Unterminated string (Anthropic)
The two patches
litellm/litellm_core_utils/exception_mapping_utils.pyindex ef062ff47a..fa4b716897 100644@@ -2036,7 +2036,17 @@ def exception_type( # type: ignore # noqa: PLR0915 error_str = original_exception.get("error", "") else: error_str = str(original_exception)- if "no such file or directory" in error_str:+ if error_str.startswith("Failed to parse tool call arguments"):+ # checked first: the message embeds raw model output, which+ # could otherwise match the substring checks below+ exception_mapping_worked = True+ raise BadRequestError(+ message=f"OllamaException: {original_exception}",+ llm_provider="ollama",+ model=model,+ response=getattr(original_exception, "response", None),+ )+ elif "no such file or directory" in error_str: exception_mapping_worked = True raise BadRequestError( message=f"OllamaException: Invalid Model/Model not loaded - {original_exception}",litellm/litellm_core_utils/prompt_templates/factory.pyindex a037360c87..df529e886e 100644@@ -204,6 +204,8 @@ def ollama_pt( ) -> Union[ str, OllamaVisionModelObject ]: # https://github.com/ollama/ollama/blob/af4cf55884ac54b9e637cd71dadfe9b7a5685877/docs/modelfile.md#template+ from litellm.llms.ollama.common_utils import parse_ollama_tool_call_arguments+ user_message_types = {"user", "tool", "function"} msg_i = 0 images = []@@ -250,7 +252,9 @@ def ollama_pt( for call in tool_calls: call_id: str = call["id"] function_name: str = call["function"]["name"]- arguments = json.loads(call["function"]["arguments"])+ arguments = parse_ollama_tool_call_arguments(+ call["function"].get("arguments"), tool_name=function_name+ ) ollama_tool_calls.append( {litellm/llms/ollama/chat/transformation.pyindex c990cc2e09..4cfdb349b4 100644@@ -36,7 +36,7 @@ from litellm.types.llms.openai import ( ) from litellm.types.utils import ModelResponse, ModelResponseStream -from ..common_utils import OllamaError+from ..common_utils import OllamaError, parse_ollama_tool_call_arguments if TYPE_CHECKING: from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj@@ -272,7 +272,10 @@ class OllamaChatConfig(BaseConfig): if typed_tool["type"] == "function": arguments = {} if "arguments" in typed_tool["function"]:- arguments = json.loads(typed_tool["function"]["arguments"])+ arguments = parse_ollama_tool_call_arguments(+ typed_tool["function"]["arguments"],+ tool_name=typed_tool["function"].get("name"),+ ) ollama_tool_call = OllamaToolCall( function=OllamaToolCallFunction( name=typed_tool["function"].get("name") or "",litellm/llms/ollama/common_utils.pyindex 8aedd9b350..1e5873014e 100644@@ -1,4 +1,4 @@-from typing import List, Optional, Union+from typing import Any, List, Optional, Union import httpx @@ -13,6 +13,42 @@ class OllamaError(BaseLLMException): super().__init__(status_code=status_code, message=message, headers=headers) +def parse_ollama_tool_call_arguments(+ arguments: Union[str, dict, None], tool_name: Optional[str] = None+) -> Any:+ """+ Parse OpenAI-format tool call `arguments` (a JSON string) for an Ollama request.++ - dict arguments are passed through unchanged+ - empty / None arguments become {}+ - truncated JSON is repaired when possible (missing closing braces/brackets)+ - unparseable JSON raises an OllamaError (400) that includes the raw arguments+ """+ from litellm.litellm_core_utils.prompt_templates.common_utils import (+ parse_tool_call_arguments,+ )++ if isinstance(arguments, dict):+ return arguments+ if arguments is not None and not isinstance(arguments, str):+ raise OllamaError(+ status_code=400,+ message=f"Failed to parse tool call arguments for tool '{tool_name or '<unknown>'}' (Ollama request). Expected a JSON string or object, got {type(arguments).__name__}: {arguments!r}",+ headers={},+ )+ try:+ return parse_tool_call_arguments(+ arguments, tool_name=tool_name, context="Ollama request"+ )+ except ValueError as e:+ verbose_logger.error(+ "Failed to parse tool call arguments for tool '%s': %r",+ tool_name or "<unknown>",+ arguments,+ )+ raise OllamaError(status_code=400, message=str(e), headers={}) from e++ def _convert_image(image): """ Convert image to base64 encoded image if not already in base64 formattests/test_litellm/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_factory.pyindex 30b47a853e..172c20efd4 100644@@ -2339,3 +2339,48 @@ def test_anthropic_messages_pt_file_block_preserves_cache_control(): assert text_block["type"] == "text" assert "cache_control" in text_block assert text_block["cache_control"]["type"] == "ephemeral"+++def _ollama_pt_tool_call_messages(arguments):+ return [+ {"role": "user", "content": "Weather in Paris?"},+ {+ "role": "assistant",+ "content": "",+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {"name": "get_weather", "arguments": arguments},+ }+ ],+ },+ ]+++@pytest.mark.parametrize(+ "arguments, expected",+ [+ ('{"city": "Paris"}', {"city": "Paris"}),+ ("", {}),+ ({"city": "Paris"}, {"city": "Paris"}),+ ('{"city": "Paris"', {"city": "Paris"}),+ ],+)+def test_ollama_pt_tool_call_arguments(arguments, expected):+ result = ollama_pt(+ model="llama2", messages=_ollama_pt_tool_call_messages(arguments)+ )+ tool_calls_json = result["prompt"].split("Tool Calls: ", 1)[1].split("\n\n")[0]+ assert json.loads(tool_calls_json)[0]["function"]["arguments"] == expected+++def test_ollama_pt_malformed_tool_call_arguments_raises_informative_error():+ from litellm.llms.ollama.common_utils import OllamaError++ raw = '{"city": "Par'+ with pytest.raises(OllamaError) as exc_info:+ ollama_pt(model="llama2", messages=_ollama_pt_tool_call_messages(raw))+ assert exc_info.value.status_code == 400+ assert "Unterminated string" in exc_info.value.message+ assert raw in exc_info.value.messagetests/test_litellm/llms/ollama/test_ollama_chat_transformation.pyindex a1afdd3a36..fd6077fbd1 100644@@ -735,3 +735,106 @@ class TestOllamaReasoningContentStreaming: assert result.choices[0].finish_reason == "stop" +++class TestOllamaChatToolCallArgumentsParsing:+ """+ Regression tests for https://github.com/BerriAI/litellm/issues - malformed+ tool call arguments in assistant messages crashed transform_request with a raw+ JSONDecodeError ('Unterminated string'), surfaced as APIConnectionError.+ """++ @staticmethod+ def _messages(arguments):+ return [+ {"role": "user", "content": "What's the weather in Paris?"},+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_1",+ "type": "function",+ "function": {"name": "get_weather", "arguments": arguments},+ }+ ],+ },+ {"role": "tool", "tool_call_id": "call_1", "content": "sunny"},+ ]++ def _transform(self, arguments):+ return OllamaChatConfig().transform_request(+ model="glm-5",+ messages=cast(list, self._messages(arguments)),+ optional_params={},+ litellm_params={},+ headers={},+ )++ @pytest.mark.parametrize(+ "arguments, expected",+ [+ ('{"city": "Paris"}', {"city": "Paris"}),+ ("", {}),+ (None, {}),+ ({"city": "Paris"}, {"city": "Paris"}),+ ('{"city": "Paris"', {"city": "Paris"}), # truncated -> repaired+ ],+ )+ def test_tool_call_arguments_parsed(self, arguments, expected):+ from litellm.llms.ollama.common_utils import parse_ollama_tool_call_arguments++ assert (+ parse_ollama_tool_call_arguments(arguments, tool_name="get_weather")+ == expected+ )+ data = self._transform(arguments)+ assert [m["role"] for m in data["messages"]] == ["user", "assistant", "tool"]++ def test_unterminated_string_raises_informative_error(self):+ from litellm.llms.ollama.common_utils import OllamaError++ raw = '{"city": "Par'+ with pytest.raises(OllamaError) as exc_info:+ self._transform(raw)+ assert exc_info.value.status_code == 400+ assert "get_weather" in exc_info.value.message+ assert "Unterminated string" in exc_info.value.message+ assert raw in exc_info.value.message++ @pytest.mark.parametrize("model", ["ollama_chat/glm-5", "ollama/glm-5"])+ def test_completion_raises_bad_request_for_malformed_arguments(self, model):+ raw = '{"city": "Par'+ with pytest.raises(litellm.BadRequestError) as exc_info:+ litellm.completion(+ model=model,+ messages=self._messages(raw),+ api_base="http://127.0.0.1:9",+ max_retries=0,+ )+ assert "Unterminated string" in str(exc_info.value)+ assert raw in str(exc_info.value)++ @pytest.mark.parametrize("model", ["ollama_chat/glm-5", "ollama/glm-5"])+ def test_malformed_arguments_mentioning_connection_error_still_bad_request(+ self, model+ ):+ # raw arguments are embedded in the error message; they must not trip the+ # connection-error mapping (-> retryable ServiceUnavailableError)+ raw = '{"log": "Failed to establish a new connection to db'+ with pytest.raises(litellm.BadRequestError):+ litellm.completion(+ model=model,+ messages=self._messages(raw),+ api_base="http://127.0.0.1:9",+ max_retries=0,+ )++ @pytest.mark.parametrize("arguments", [["Paris"], 42])+ def test_non_string_arguments_raise_informative_error(self, arguments):+ from litellm.llms.ollama.common_utils import OllamaError++ with pytest.raises(OllamaError) as exc_info:+ self._transform(arguments)+ assert exc_info.value.status_code == 400+ assert "get_weather" in exc_info.value.message
litellm/llms/ollama/chat/transformation.pyindex c990cc2e093..31e694ce798 100644@@ -1,6 +1,7 @@ import json import time from litellm._uuid import uuid+from litellm._logging import verbose_logger from typing import ( TYPE_CHECKING, Any,@@ -272,7 +273,18 @@ class OllamaChatConfig(BaseConfig): if typed_tool["type"] == "function": arguments = {} if "arguments" in typed_tool["function"]:- arguments = json.loads(typed_tool["function"]["arguments"])+ raw_args = typed_tool["function"]["arguments"]+ try:+ arguments = json.loads(raw_args)+ except json.JSONDecodeError as e:+ verbose_logger.error(+ f"Failed to parse tool call arguments as JSON: {raw_args!r}. Error: {e}"+ )+ raise litellm.BadRequestError(+ message=f"Tool call arguments contain malformed JSON: {e.msg}. Raw arguments: {raw_args!r}",+ model="ollama",+ llm_provider="ollama",+ ) ollama_tool_call = OllamaToolCall( function=OllamaToolCallFunction( name=typed_tool["function"].get("name") or "",tests/test_litellm/llms/ollama/test_ollama_chat_transformation.pyindex a1afdd3a369..6cd3e1a4dd9 100644@@ -338,7 +338,84 @@ class TestOllamaToolCalling: Issue: https://github.com/BerriAI/litellm/issues/18922 """ - def test_tools_passed_directly_without_capability_check(self):+ def test_transform_request_malformed_tool_call_arguments_raises_bad_request(self):+ """Test that malformed JSON in tool call arguments raises BadRequestError.++ Regression: json.JSONDecodeError was previously raised directly, wrapping it+ in litellm.BadRequestError gives callers an actionable error with context.++ Issue: https://github.com/BerriAI/litellm/issues/25985+ """+ config = OllamaChatConfig()++ messages = cast(+ list[AllMessageValues],+ [+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_123",+ "type": "function",+ "function": {+ "name": "get_weather",+ "arguments": '{"city": "Toky', # truncated JSON+ },+ }+ ],+ }+ ],+ )++ import litellm++ with pytest.raises(litellm.BadRequestError) as exc_info:+ config.transform_request(+ model="qwen3:14b",+ messages=messages,+ optional_params={},+ litellm_params={},+ headers={},+ )++ assert "malformed JSON" in str(exc_info.value)++ def test_transform_request_valid_tool_call_arguments_passes(self):+ """Test that valid JSON tool call arguments are parsed without raising an exception."""+ config = OllamaChatConfig()++ messages = cast(+ list[AllMessageValues],+ [+ {+ "role": "assistant",+ "content": None,+ "tool_calls": [+ {+ "id": "call_123",+ "type": "function",+ "function": {+ "name": "get_weather",+ "arguments": '{"city": "Tokyo"}',+ },+ }+ ],+ }+ ],+ )++ # Should not raise any exception+ result = config.transform_request(+ model="qwen3:14b",+ messages=messages,+ optional_params={},+ litellm_params={},+ headers={},+ )+ assert "messages" in result++ """Test that tools are passed directly to Ollama without model capability checks. Previously, the code called litellm.get_model_info() which could fail