Backtestslitellmrun-20261111-220245#46050
[Bug]: Reranker forcefully sets truncate to true for TEI
The judges split evenly between our patch and the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches make the same core fix: they stop hard-coding `truncate: False` in transform_rerank_request so TEI's server-side --auto-truncate governs by default, and they add `truncate`/`truncation_direction` as supported Cohere-style params that are forwarded when a caller sets them. All three independent tests (sync default, async default, explicit truncate=True) fail on base and pass on both, so neither patch leaves the reported behaviour broken. The differences are at the margins. Patch A's production change is minimal and surgical: two new supported-param entries, two new mapping branches, one removed default. Patch B's production diff additionally deletes the `elif k == "documents": optional_rerank_params["texts"] = v` branch from map_cohere_rerank_params without any explanation. The independent tests still see `texts` in the outbound body on B, so that branch is evidently redundant with another path, but silently removing an unrelated mapping line in the same hunk is unexplained churn that adds risk and obscures the fix; it is exactly the kind of out-of-scope edit the rubric says to count against a patch. B's ValueError validation for non-bool truncate and non-Left/Right direction is reasonable hardening but beyond the issue's ask and earns no credit on its own; A's `isinstance(v, bool)` guard quietly drops malformed values, which is a slightly weaker edge but consistent with the permissive style of the surrounding branches. On tests, both update the existing assertions that pinned `truncate: False` and both add tests that fail before and pass after (default omits truncate; explicit True/False and Left are forwarded). B's unit tests are tidy and parametrized; A adds an equivalent unit test using an httpx MockTransport plus an integration wire test that depends on an `integration._support` harness whose availability cannot be confirmed here, and A also sprinkles `Final` annotations through existing tests as cosmetic churn. Those are minor negatives for A's test hygiene, but they live in test files, whereas B's unexplained deletion is in production code. Weighing correctness and scope first, A is the cleaner, lower-risk fix.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
6m 46s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
86vs 172 merged
Changed lines, ours vs merged
The issue
View on GitHubDescription
https://github.com/BerriAI/litellm/pull/45682
We tried to set --auto-truncate to true in the running TEI process, however requests through LiteLLM were not being truncated. We were using the Hugging Face transformer, which appears to unconditionally set truncate to false.
Config
Not able to copy configuration for confidential reasons
LiteLLM Version
v100.4
Steps to Repro
hitting the reranking endpoint (litellm.svc..../rerank) which routed to a self hosted TEI reranking model.
Which part of LiteLLM is this about?
LLM translation: a specific provider's request or response
How are you deploying?
Helm chart, componentized (recommended)
The two patches
litellm/llms/huggingface/rerank/transformation.pyindex 1f130f8428..d5d7063872 100644@@ -78,6 +78,8 @@ class HuggingFaceRerankConfig(BaseRerankConfig): "documents", "top_n", "return_documents",+ "truncate",+ "truncation_direction", ] def map_cohere_rerank_params(@@ -104,10 +106,18 @@ class HuggingFaceRerankConfig(BaseRerankConfig): optional_rerank_params["return_text"] = v elif k == "top_n" and v is not None: optional_rerank_params["top_n"] = v- elif k == "documents" and v is not None:- optional_rerank_params["texts"] = v elif k == "query" and v is not None: optional_rerank_params["query"] = v+ elif k == "truncate" and v is not None:+ if not isinstance(v, bool):+ raise ValueError(f"HuggingFace rerank 'truncate' must be a bool, got {v!r}")+ optional_rerank_params["truncate"] = v+ elif k == "truncation_direction" and v is not None:+ if v not in ("Left", "Right"):+ raise ValueError(+ f"HuggingFace rerank 'truncation_direction' must be 'Left' or 'Right', got {v!r}"+ )+ optional_rerank_params["truncation_direction"] = v return OptionalRerankParams(**optional_rerank_params) @@ -147,11 +157,10 @@ class HuggingFaceRerankConfig(BaseRerankConfig): raise ValueError("query is required for HuggingFace rerank") if "texts" not in optional_rerank_params: raise ValueError("Cohere 'documents' param is required for HuggingFace rerank")- # Ensure return_text is a boolean value- # HuggingFace API expects return_text parameter, corresponding to our return_documents parameter+ # HuggingFace API expects return_text parameter, corresponding to our return_documents parameter.+ # `truncate` is only sent when the caller sets it, so TEI's server-side --auto-truncate applies otherwise. request_body: Final = { "raw_scores": False,- "truncate": False, "truncation_direction": "Right", } tests/unit/llms/huggingface/rerank/test_huggingface_rerank_transformation.pyindex 6fd2b006fe..c73e6f0c0c 100644@@ -275,7 +275,7 @@ def test_request_transformation(): assert request_body["top_n"] == 2 assert request_body["return_text"] is True assert request_body["raw_scores"] is False- assert request_body["truncate"] is False+ assert "truncate" not in request_body, request_body assert request_body["truncation_direction"] == "Right" @@ -385,13 +385,72 @@ def test_huggingface_rerank_request_payload(mock_post): "texts": ["hello", "world"], "raw_scores": False, "return_text": True,- "truncate": False, "truncation_direction": "Right", "top_n": 2, } - for key, value in expected_request.items():- assert request_data[key] == value+ assert request_data == expected_request, request_data assert response.results is not None assert len(response.results) == 2+++def _mock_tei_rerank_response() -> MagicMock:+ mock_response = MagicMock()+ mock_response.json = lambda: [{"index": 0, "score": 0.9}, {"index": 1, "score": 0.1}]+ mock_response.status_code = 200+ mock_response.headers = {"content-type": "application/json"}+ return mock_response+++@patch("litellm.llms.custom_httpx.http_handler.HTTPHandler.post")+def test_huggingface_rerank_omits_truncate_so_tei_auto_truncate_applies(mock_post):+ mock_post.return_value = _mock_tei_rerank_response()++ litellm.rerank(+ model="huggingface/BAAI/bge-reranker-base",+ query="hello",+ documents=["hello", "world"],+ api_key="test_api_key",+ api_base="http://tei.local",+ )++ request_data = json.loads(mock_post.call_args.kwargs["data"])+ assert "truncate" not in request_data, f"truncate must be left to TEI's --auto-truncate: {request_data}"+++@pytest.mark.parametrize("truncate", [True, False])+@patch("litellm.llms.custom_httpx.http_handler.HTTPHandler.post")+def test_huggingface_rerank_forwards_caller_truncate(mock_post, truncate):+ mock_post.return_value = _mock_tei_rerank_response()++ litellm.rerank(+ model="huggingface/BAAI/bge-reranker-base",+ query="hello",+ documents=["hello", "world"],+ api_key="test_api_key",+ api_base="http://tei.local",+ truncate=truncate,+ truncation_direction="Left",+ )++ request_data = json.loads(mock_post.call_args.kwargs["data"])+ assert request_data["truncate"] is truncate, request_data+ assert request_data["truncation_direction"] == "Left", request_data+++@pytest.mark.parametrize(+ "bad_params",+ [{"truncate": "yes"}, {"truncation_direction": "middle"}],+)+def test_huggingface_rerank_rejects_invalid_truncation_params(bad_params):+ from litellm.llms.huggingface.rerank.transformation import HuggingFaceRerankConfig++ with pytest.raises(ValueError):+ HuggingFaceRerankConfig().map_cohere_rerank_params(+ non_default_params={"query": "hello", "documents": ["hello"], **bad_params},+ model="test",+ drop_params=False,+ query="hello",+ documents=["hello"],+ )
litellm/llms/huggingface/rerank/transformation.pyindex 1f130f84283..476e22fc088 100644@@ -78,6 +78,8 @@ class HuggingFaceRerankConfig(BaseRerankConfig): "documents", "top_n", "return_documents",+ "truncate",+ "truncation_direction", ] def map_cohere_rerank_params(@@ -104,6 +106,10 @@ class HuggingFaceRerankConfig(BaseRerankConfig): optional_rerank_params["return_text"] = v elif k == "top_n" and v is not None: optional_rerank_params["top_n"] = v+ elif k == "truncate" and isinstance(v, bool):+ optional_rerank_params["truncate"] = v+ elif k == "truncation_direction" and v is not None:+ optional_rerank_params["truncation_direction"] = v elif k == "documents" and v is not None: optional_rerank_params["texts"] = v elif k == "query" and v is not None:@@ -151,7 +157,6 @@ class HuggingFaceRerankConfig(BaseRerankConfig): # HuggingFace API expects return_text parameter, corresponding to our return_documents parameter request_body: Final = { "raw_scores": False,- "truncate": False, "truncation_direction": "Right", } tests/integration/providers/test_huggingface_rerank_truncate_wire.pynew file mode 100644index 00000000000..0d29dbf0ded@@ -0,0 +1,74 @@+import json+from collections.abc import Mapping+from types import MappingProxyType+from typing import Final++import pytest+from integration._support.client import JSON_OBJECT, Gateway+from integration._support.wire import Reply, Request, wire_server++MODEL: Final = "huggingface/BAAI/bge-reranker-base"+QUERY: Final = "what is a fox"+TEXT: Final = "a short doc about foxes"+RESPONSE: Final = json.dumps([{"index": 0, "score": 0.82}]).encode()+TRUNCATION_CASES: Final = (+ pytest.param(+ MappingProxyType({}),+ MappingProxyType({"truncation_direction": "Right"}),+ id="omitted-truncate-uses-server-default",+ ),+ pytest.param(+ MappingProxyType({"truncate": True, "truncation_direction": "Left"}),+ MappingProxyType({"truncate": True, "truncation_direction": "Left"}),+ id="explicit-true-and-left-are-forwarded",+ ),+ pytest.param(+ MappingProxyType({"truncate": False}),+ MappingProxyType({"truncate": False, "truncation_direction": "Right"}),+ id="explicit-false-is-forwarded",+ ),+)+++def rerank_peer(request: Request, expected_body: Mapping[str, object]) -> Reply:+ assert request.method == "POST" and request.target == "/rerank", request.target+ assert request.headers["authorization"] == "Bearer integration-provider-key"+ body: Final = JSON_OBJECT.validate_json(request.body)+ assert body == expected_body, body+ return Reply(body=RESPONSE)+++@pytest.mark.parametrize(("request_params", "expected_truncation_params"), TRUNCATION_CASES)+def test_huggingface_rerank_forwards_truncation_options(+ gateway: Gateway,+ request_params: Mapping[str, bool | str],+ expected_truncation_params: Mapping[str, bool | str],+) -> None:+ expected_body: Final = {+ "raw_scores": False,+ "truncation_direction": "Right",+ "query": QUERY,+ "texts": [TEXT],+ "return_text": False,+ **expected_truncation_params,+ }+ with wire_server(lambda request: rerank_peer(request, expected_body)) as wire, gateway.scenario() as scenario:+ model: Final = scenario.model(+ model=MODEL,+ api_base=wire.url,+ model_info={"mode": "rerank"},+ )+ response: Final = gateway.request(+ "POST",+ "/v1/rerank",+ {+ "model": model,+ "query": QUERY,+ "documents": [TEXT],+ "return_documents": False,+ **request_params,+ },+ )+ assert response.status_code == 200, response.text+ assert response.json()["results"] == [{"index": 0, "relevance_score": 0.82}], response.text+ assert len(wire.drain()) == 1, "Expected exactly one provider rerank call"tests/unit/llms/huggingface/rerank/test_huggingface_rerank_transformation.pyindex 6fd2b006fef..00a3cbfdb7f 100644@@ -5,11 +5,14 @@ Based on the test patterns from other rerank providers and the current HuggingFa import asyncio import json+from typing import Final from unittest.mock import AsyncMock, MagicMock, patch +import httpx import pytest import litellm+from litellm.llms.custom_httpx.http_handler import HTTPHandler def assert_response_shape(response, custom_llm_provider):@@ -219,11 +222,11 @@ def test_huggingface_rerank_return_documents(mock_post): assert "text" in result["document"] -def test_huggingface_rerank_config():+def test_huggingface_rerank_config() -> None: """Test HuggingFaceRerankConfig class functionality.""" from litellm.llms.huggingface.rerank.transformation import HuggingFaceRerankConfig - config = HuggingFaceRerankConfig()+ config: Final = HuggingFaceRerankConfig() # Test complete URL generation assert config.get_complete_url(None, "test") == "https://api-inference.huggingface.co/rerank"@@ -233,14 +236,16 @@ def test_huggingface_rerank_config(): assert custom_url == "https://custom.huggingface.co/rerank" # Test supported parameters- supported_params = config.get_supported_cohere_rerank_params("test")+ supported_params: Final = config.get_supported_cohere_rerank_params("test") assert "query" in supported_params assert "documents" in supported_params assert "top_n" in supported_params assert "return_documents" in supported_params+ assert "truncate" in supported_params+ assert "truncation_direction" in supported_params # Test parameter mapping- params = config.map_cohere_rerank_params(+ params: Final = config.map_cohere_rerank_params( non_default_params={ "query": "hello", "documents": ["hello", "world"],@@ -259,24 +264,28 @@ def test_huggingface_rerank_config(): assert params["return_text"] is True -def test_request_transformation():+def test_request_transformation() -> None: """Test request transformation logic.""" from litellm.llms.huggingface.rerank.transformation import HuggingFaceRerankConfig from litellm.types.rerank import OptionalRerankParams - config = HuggingFaceRerankConfig()+ config: Final = HuggingFaceRerankConfig() - optional_params = OptionalRerankParams(query="hello", texts=["hello", "world"], top_n=2, return_text=True)+ optional_params: Final = OptionalRerankParams(query="hello", texts=["hello", "world"], top_n=2, return_text=True) - request_body = config.transform_rerank_request(model="test", optional_rerank_params=optional_params, headers={})+ request_body: Final = config.transform_rerank_request(+ model="test", optional_rerank_params=optional_params, headers={}+ ) - assert request_body["query"] == "hello"- assert request_body["texts"] == ["hello", "world"]- assert request_body["top_n"] == 2- assert request_body["return_text"] is True- assert request_body["raw_scores"] is False- assert request_body["truncate"] is False- assert request_body["truncation_direction"] == "Right"+ expected_request: Final = {+ "query": "hello",+ "texts": ["hello", "world"],+ "top_n": 2,+ "return_text": True,+ "raw_scores": False,+ "truncation_direction": "Right",+ }+ assert request_body == expected_request def test_response_transformation():@@ -345,20 +354,20 @@ def test_validate_environment(): @patch("litellm.llms.custom_httpx.http_handler.HTTPHandler.post")-def test_huggingface_rerank_request_payload(mock_post):+def test_huggingface_rerank_request_payload(mock_post: MagicMock) -> None: """Test that the request payload is correctly formatted for HuggingFace API."""- mock_response_data = [{"index": 0, "score": 0.9}, {"index": 1, "score": 0.1}]+ mock_response_data: Final = [{"index": 0, "score": 0.9}, {"index": 1, "score": 0.1}] def return_val(): return mock_response_data - mock_response = MagicMock()+ mock_response: Final = MagicMock() mock_response.json = return_val mock_response.status_code = 200 mock_response.headers = {"content-type": "application/json"} mock_post.return_value = mock_response - response = litellm.rerank(+ response: Final = litellm.rerank( model="huggingface/BAAI/bge-reranker-base", query="hello", documents=["hello", "world"],@@ -370,28 +379,58 @@ def test_huggingface_rerank_request_payload(mock_post): mock_post.assert_called_once() # Verify URL- call_url = mock_post.call_args.kwargs["url"]+ call_url: Final = mock_post.call_args.kwargs["url"] assert call_url == "https://api-inference.huggingface.co/rerank" # Verify headers- headers = mock_post.call_args.kwargs["headers"]+ headers: Final = mock_post.call_args.kwargs["headers"] assert "Bearer test_api_key" in headers["Authorization"] assert headers["content-type"] == "application/json" # Verify request body- request_data = json.loads(mock_post.call_args.kwargs["data"])- expected_request = {+ request_data: Final = json.loads(mock_post.call_args.kwargs["data"])+ expected_request: Final = { "query": "hello", "texts": ["hello", "world"], "raw_scores": False, "return_text": True,- "truncate": False, "truncation_direction": "Right", "top_n": 2, } - for key, value in expected_request.items():- assert request_data[key] == value+ assert request_data == expected_request assert response.results is not None assert len(response.results) == 2+++def test_huggingface_rerank_forwards_truncate_options() -> None:+ expected_request: Final = {+ "query": "hello",+ "texts": ["hello", "world"],+ "raw_scores": False,+ "return_text": False,+ "truncation_direction": "Left",+ "truncate": True,+ }++ def respond(request: httpx.Request) -> httpx.Response:+ assert request.url == "https://api-inference.huggingface.co/rerank"+ assert request.headers["Authorization"] == "Bearer test_api_key"+ assert json.loads(request.content) == expected_request+ return httpx.Response(200, json=[{"index": 0, "score": 0.9}])++ with httpx.Client(transport=httpx.MockTransport(respond)) as http_client:+ response: Final = litellm.rerank(+ model="huggingface/BAAI/bge-reranker-base",+ query="hello",+ documents=["hello", "world"],+ api_key="test_api_key",+ return_documents=False,+ truncate=True,+ truncation_direction="Left",+ client=HTTPHandler(client=http_client),+ )++ assert response.results is not None+ assert response.results[0]["relevance_score"] == 0.9