Backtestslitellmrun-20261111-220245#46050

Issue #46050 Merged fix PR #48706 Run 11 Nov 2026 · run-20261111-220245 BerriAI/litellm

[Bug]: Reranker forcefully sets truncate to true for TEI

Tie

The judges split evenly between our patch and the merged fix.

In this judgment, Patch B = our agent, Patch A = the merged fix.

Both patches make the same core fix: they stop hard-coding `truncate: False` in transform_rerank_request so TEI's server-side --auto-truncate governs by default, and they add `truncate`/`truncation_direction` as supported Cohere-style params that are forwarded when a caller sets them. All three independent tests (sync default, async default, explicit truncate=True) fail on base and pass on both, so neither patch leaves the reported behaviour broken.

The differences are at the margins. Patch A's production change is minimal and surgical: two new supported-param entries, two new mapping branches, one removed default. Patch B's production diff additionally deletes the `elif k == "documents": optional_rerank_params["texts"] = v` branch from map_cohere_rerank_params without any explanation. The independent tests still see `texts` in the outbound body on B, so that branch is evidently redundant with another path, but silently removing an unrelated mapping line in the same hunk is unexplained churn that adds risk and obscures the fix; it is exactly the kind of out-of-scope edit the rubric says to count against a patch. B's ValueError validation for non-bool truncate and non-Left/Right direction is reasonable hardening but beyond the issue's ask and earns no credit on its own; A's `isinstance(v, bool)` guard quietly drops malformed values, which is a slightly weaker edge but consistent with the permissive style of the surrounding branches.

On tests, both update the existing assertions that pinned `truncate: False` and both add tests that fail before and pass after (default omits truncate; explicit True/False and Left are forwarded). B's unit tests are tidy and parametrized; A adds an equivalent unit test using an httpx MockTransport plus an integration wire test that depends on an `integration._support` harness whose availability cannot be confirmed here, and A also sprinkles `Final` annotations through existing tests as cosmetic churn. Those are minor negatives for A's test hygiene, but they live in test files, whereas B's unexplained deletion is in production code. Weighing correctness and scope first, A is the cleaner, lower-risk fix.

Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.

Solve

6m 46s

Wall-clock time in the container

Full test suite

Not run

The harness did not reach the test stage

Contamination audit

Clean

No sign the agent saw the real fix

Patch size

86vs 172 merged

Changed lines, ours vs merged

Description

https://github.com/BerriAI/litellm/pull/45682

We tried to set --auto-truncate to true in the running TEI process, however requests through LiteLLM were not being truncated. We were using the Hugging Face transformer, which appears to unconditionally set truncate to false.

Config

Not able to copy configuration for confidential reasons

LiteLLM Version

v100.4

Steps to Repro

hitting the reranking endpoint (litellm.svc..../rerank) which routed to a self hosted TEI reranking model.

Which part of LiteLLM is this about?

LLM translation: a specific provider's request or response

How are you deploying?

Helm chart, componentized (recommended)

The two patches

Our patch+77 −9
litellm/llms/huggingface/rerank/transformation.pyindex 1f130f8428..d5d7063872 100644@@ -78,6 +78,8 @@ class HuggingFaceRerankConfig(BaseRerankConfig):             "documents",             "top_n",             "return_documents",+            "truncate",+            "truncation_direction",         ]      def map_cohere_rerank_params(@@ -104,10 +106,18 @@ class HuggingFaceRerankConfig(BaseRerankConfig):                     optional_rerank_params["return_text"] = v                 elif k == "top_n" and v is not None:                     optional_rerank_params["top_n"] = v-                elif k == "documents" and v is not None:-                    optional_rerank_params["texts"] = v                 elif k == "query" and v is not None:                     optional_rerank_params["query"] = v+                elif k == "truncate" and v is not None:+                    if not isinstance(v, bool):+                        raise ValueError(f"HuggingFace rerank 'truncate' must be a bool, got {v!r}")+                    optional_rerank_params["truncate"] = v+                elif k == "truncation_direction" and v is not None:+                    if v not in ("Left", "Right"):+                        raise ValueError(+                            f"HuggingFace rerank 'truncation_direction' must be 'Left' or 'Right', got {v!r}"+                        )+                    optional_rerank_params["truncation_direction"] = v          return OptionalRerankParams(**optional_rerank_params) @@ -147,11 +157,10 @@ class HuggingFaceRerankConfig(BaseRerankConfig):             raise ValueError("query is required for HuggingFace rerank")         if "texts" not in optional_rerank_params:             raise ValueError("Cohere 'documents' param is required for HuggingFace rerank")-        # Ensure return_text is a boolean value-        # HuggingFace API expects return_text parameter, corresponding to our return_documents parameter+        # HuggingFace API expects return_text parameter, corresponding to our return_documents parameter.+        # `truncate` is only sent when the caller sets it, so TEI's server-side --auto-truncate applies otherwise.         request_body: Final = {             "raw_scores": False,-            "truncate": False,             "truncation_direction": "Right",         } tests/unit/llms/huggingface/rerank/test_huggingface_rerank_transformation.pyindex 6fd2b006fe..c73e6f0c0c 100644@@ -275,7 +275,7 @@ def test_request_transformation():     assert request_body["top_n"] == 2     assert request_body["return_text"] is True     assert request_body["raw_scores"] is False-    assert request_body["truncate"] is False+    assert "truncate" not in request_body, request_body     assert request_body["truncation_direction"] == "Right"  @@ -385,13 +385,72 @@ def test_huggingface_rerank_request_payload(mock_post):         "texts": ["hello", "world"],         "raw_scores": False,         "return_text": True,-        "truncate": False,         "truncation_direction": "Right",         "top_n": 2,     } -    for key, value in expected_request.items():-        assert request_data[key] == value+    assert request_data == expected_request, request_data      assert response.results is not None     assert len(response.results) == 2+++def _mock_tei_rerank_response() -> MagicMock:+    mock_response = MagicMock()+    mock_response.json = lambda: [{"index": 0, "score": 0.9}, {"index": 1, "score": 0.1}]+    mock_response.status_code = 200+    mock_response.headers = {"content-type": "application/json"}+    return mock_response+++@patch("litellm.llms.custom_httpx.http_handler.HTTPHandler.post")+def test_huggingface_rerank_omits_truncate_so_tei_auto_truncate_applies(mock_post):+    mock_post.return_value = _mock_tei_rerank_response()++    litellm.rerank(+        model="huggingface/BAAI/bge-reranker-base",+        query="hello",+        documents=["hello", "world"],+        api_key="test_api_key",+        api_base="http://tei.local",+    )++    request_data = json.loads(mock_post.call_args.kwargs["data"])+    assert "truncate" not in request_data, f"truncate must be left to TEI's --auto-truncate: {request_data}"+++@pytest.mark.parametrize("truncate", [True, False])+@patch("litellm.llms.custom_httpx.http_handler.HTTPHandler.post")+def test_huggingface_rerank_forwards_caller_truncate(mock_post, truncate):+    mock_post.return_value = _mock_tei_rerank_response()++    litellm.rerank(+        model="huggingface/BAAI/bge-reranker-base",+        query="hello",+        documents=["hello", "world"],+        api_key="test_api_key",+        api_base="http://tei.local",+        truncate=truncate,+        truncation_direction="Left",+    )++    request_data = json.loads(mock_post.call_args.kwargs["data"])+    assert request_data["truncate"] is truncate, request_data+    assert request_data["truncation_direction"] == "Left", request_data+++@pytest.mark.parametrize(+    "bad_params",+    [{"truncate": "yes"}, {"truncation_direction": "middle"}],+)+def test_huggingface_rerank_rejects_invalid_truncation_params(bad_params):+    from litellm.llms.huggingface.rerank.transformation import HuggingFaceRerankConfig++    with pytest.raises(ValueError):+        HuggingFaceRerankConfig().map_cohere_rerank_params(+            non_default_params={"query": "hello", "documents": ["hello"], **bad_params},+            model="test",+            drop_params=False,+            query="hello",+            documents=["hello"],+        ) 
Merged fix · PR #48706+145 −27
litellm/llms/huggingface/rerank/transformation.pyindex 1f130f84283..476e22fc088 100644@@ -78,6 +78,8 @@ class HuggingFaceRerankConfig(BaseRerankConfig):             "documents",             "top_n",             "return_documents",+            "truncate",+            "truncation_direction",         ]      def map_cohere_rerank_params(@@ -104,6 +106,10 @@ class HuggingFaceRerankConfig(BaseRerankConfig):                     optional_rerank_params["return_text"] = v                 elif k == "top_n" and v is not None:                     optional_rerank_params["top_n"] = v+                elif k == "truncate" and isinstance(v, bool):+                    optional_rerank_params["truncate"] = v+                elif k == "truncation_direction" and v is not None:+                    optional_rerank_params["truncation_direction"] = v                 elif k == "documents" and v is not None:                     optional_rerank_params["texts"] = v                 elif k == "query" and v is not None:@@ -151,7 +157,6 @@ class HuggingFaceRerankConfig(BaseRerankConfig):         # HuggingFace API expects return_text parameter, corresponding to our return_documents parameter         request_body: Final = {             "raw_scores": False,-            "truncate": False,             "truncation_direction": "Right",         } tests/integration/providers/test_huggingface_rerank_truncate_wire.pynew file mode 100644index 00000000000..0d29dbf0ded@@ -0,0 +1,74 @@+import json+from collections.abc import Mapping+from types import MappingProxyType+from typing import Final++import pytest+from integration._support.client import JSON_OBJECT, Gateway+from integration._support.wire import Reply, Request, wire_server++MODEL: Final = "huggingface/BAAI/bge-reranker-base"+QUERY: Final = "what is a fox"+TEXT: Final = "a short doc about foxes"+RESPONSE: Final = json.dumps([{"index": 0, "score": 0.82}]).encode()+TRUNCATION_CASES: Final = (+    pytest.param(+        MappingProxyType({}),+        MappingProxyType({"truncation_direction": "Right"}),+        id="omitted-truncate-uses-server-default",+    ),+    pytest.param(+        MappingProxyType({"truncate": True, "truncation_direction": "Left"}),+        MappingProxyType({"truncate": True, "truncation_direction": "Left"}),+        id="explicit-true-and-left-are-forwarded",+    ),+    pytest.param(+        MappingProxyType({"truncate": False}),+        MappingProxyType({"truncate": False, "truncation_direction": "Right"}),+        id="explicit-false-is-forwarded",+    ),+)+++def rerank_peer(request: Request, expected_body: Mapping[str, object]) -> Reply:+    assert request.method == "POST" and request.target == "/rerank", request.target+    assert request.headers["authorization"] == "Bearer integration-provider-key"+    body: Final = JSON_OBJECT.validate_json(request.body)+    assert body == expected_body, body+    return Reply(body=RESPONSE)+++@pytest.mark.parametrize(("request_params", "expected_truncation_params"), TRUNCATION_CASES)+def test_huggingface_rerank_forwards_truncation_options(+    gateway: Gateway,+    request_params: Mapping[str, bool | str],+    expected_truncation_params: Mapping[str, bool | str],+) -> None:+    expected_body: Final = {+        "raw_scores": False,+        "truncation_direction": "Right",+        "query": QUERY,+        "texts": [TEXT],+        "return_text": False,+        **expected_truncation_params,+    }+    with wire_server(lambda request: rerank_peer(request, expected_body)) as wire, gateway.scenario() as scenario:+        model: Final = scenario.model(+            model=MODEL,+            api_base=wire.url,+            model_info={"mode": "rerank"},+        )+        response: Final = gateway.request(+            "POST",+            "/v1/rerank",+            {+                "model": model,+                "query": QUERY,+                "documents": [TEXT],+                "return_documents": False,+                **request_params,+            },+        )+        assert response.status_code == 200, response.text+        assert response.json()["results"] == [{"index": 0, "relevance_score": 0.82}], response.text+        assert len(wire.drain()) == 1, "Expected exactly one provider rerank call"tests/unit/llms/huggingface/rerank/test_huggingface_rerank_transformation.pyindex 6fd2b006fef..00a3cbfdb7f 100644@@ -5,11 +5,14 @@ Based on the test patterns from other rerank providers and the current HuggingFa  import asyncio import json+from typing import Final from unittest.mock import AsyncMock, MagicMock, patch +import httpx import pytest  import litellm+from litellm.llms.custom_httpx.http_handler import HTTPHandler   def assert_response_shape(response, custom_llm_provider):@@ -219,11 +222,11 @@ def test_huggingface_rerank_return_documents(mock_post):             assert "text" in result["document"]  -def test_huggingface_rerank_config():+def test_huggingface_rerank_config() -> None:     """Test HuggingFaceRerankConfig class functionality."""     from litellm.llms.huggingface.rerank.transformation import HuggingFaceRerankConfig -    config = HuggingFaceRerankConfig()+    config: Final = HuggingFaceRerankConfig()      # Test complete URL generation     assert config.get_complete_url(None, "test") == "https://api-inference.huggingface.co/rerank"@@ -233,14 +236,16 @@ def test_huggingface_rerank_config():     assert custom_url == "https://custom.huggingface.co/rerank"      # Test supported parameters-    supported_params = config.get_supported_cohere_rerank_params("test")+    supported_params: Final = config.get_supported_cohere_rerank_params("test")     assert "query" in supported_params     assert "documents" in supported_params     assert "top_n" in supported_params     assert "return_documents" in supported_params+    assert "truncate" in supported_params+    assert "truncation_direction" in supported_params      # Test parameter mapping-    params = config.map_cohere_rerank_params(+    params: Final = config.map_cohere_rerank_params(         non_default_params={             "query": "hello",             "documents": ["hello", "world"],@@ -259,24 +264,28 @@ def test_huggingface_rerank_config():     assert params["return_text"] is True  -def test_request_transformation():+def test_request_transformation() -> None:     """Test request transformation logic."""     from litellm.llms.huggingface.rerank.transformation import HuggingFaceRerankConfig     from litellm.types.rerank import OptionalRerankParams -    config = HuggingFaceRerankConfig()+    config: Final = HuggingFaceRerankConfig() -    optional_params = OptionalRerankParams(query="hello", texts=["hello", "world"], top_n=2, return_text=True)+    optional_params: Final = OptionalRerankParams(query="hello", texts=["hello", "world"], top_n=2, return_text=True) -    request_body = config.transform_rerank_request(model="test", optional_rerank_params=optional_params, headers={})+    request_body: Final = config.transform_rerank_request(+        model="test", optional_rerank_params=optional_params, headers={}+    ) -    assert request_body["query"] == "hello"-    assert request_body["texts"] == ["hello", "world"]-    assert request_body["top_n"] == 2-    assert request_body["return_text"] is True-    assert request_body["raw_scores"] is False-    assert request_body["truncate"] is False-    assert request_body["truncation_direction"] == "Right"+    expected_request: Final = {+        "query": "hello",+        "texts": ["hello", "world"],+        "top_n": 2,+        "return_text": True,+        "raw_scores": False,+        "truncation_direction": "Right",+    }+    assert request_body == expected_request   def test_response_transformation():@@ -345,20 +354,20 @@ def test_validate_environment():   @patch("litellm.llms.custom_httpx.http_handler.HTTPHandler.post")-def test_huggingface_rerank_request_payload(mock_post):+def test_huggingface_rerank_request_payload(mock_post: MagicMock) -> None:     """Test that the request payload is correctly formatted for HuggingFace API."""-    mock_response_data = [{"index": 0, "score": 0.9}, {"index": 1, "score": 0.1}]+    mock_response_data: Final = [{"index": 0, "score": 0.9}, {"index": 1, "score": 0.1}]      def return_val():         return mock_response_data -    mock_response = MagicMock()+    mock_response: Final = MagicMock()     mock_response.json = return_val     mock_response.status_code = 200     mock_response.headers = {"content-type": "application/json"}     mock_post.return_value = mock_response -    response = litellm.rerank(+    response: Final = litellm.rerank(         model="huggingface/BAAI/bge-reranker-base",         query="hello",         documents=["hello", "world"],@@ -370,28 +379,58 @@ def test_huggingface_rerank_request_payload(mock_post):     mock_post.assert_called_once()      # Verify URL-    call_url = mock_post.call_args.kwargs["url"]+    call_url: Final = mock_post.call_args.kwargs["url"]     assert call_url == "https://api-inference.huggingface.co/rerank"      # Verify headers-    headers = mock_post.call_args.kwargs["headers"]+    headers: Final = mock_post.call_args.kwargs["headers"]     assert "Bearer test_api_key" in headers["Authorization"]     assert headers["content-type"] == "application/json"      # Verify request body-    request_data = json.loads(mock_post.call_args.kwargs["data"])-    expected_request = {+    request_data: Final = json.loads(mock_post.call_args.kwargs["data"])+    expected_request: Final = {         "query": "hello",         "texts": ["hello", "world"],         "raw_scores": False,         "return_text": True,-        "truncate": False,         "truncation_direction": "Right",         "top_n": 2,     } -    for key, value in expected_request.items():-        assert request_data[key] == value+    assert request_data == expected_request      assert response.results is not None     assert len(response.results) == 2+++def test_huggingface_rerank_forwards_truncate_options() -> None:+    expected_request: Final = {+        "query": "hello",+        "texts": ["hello", "world"],+        "raw_scores": False,+        "return_text": False,+        "truncation_direction": "Left",+        "truncate": True,+    }++    def respond(request: httpx.Request) -> httpx.Response:+        assert request.url == "https://api-inference.huggingface.co/rerank"+        assert request.headers["Authorization"] == "Bearer test_api_key"+        assert json.loads(request.content) == expected_request+        return httpx.Response(200, json=[{"index": 0, "score": 0.9}])++    with httpx.Client(transport=httpx.MockTransport(respond)) as http_client:+        response: Final = litellm.rerank(+            model="huggingface/BAAI/bge-reranker-base",+            query="hello",+            documents=["hello", "world"],+            api_key="test_api_key",+            return_documents=False,+            truncate=True,+            truncation_direction="Left",+            client=HTTPHandler(client=http_client),+        )++    assert response.results is not None+    assert response.results[0]["relevance_score"] == 0.9