Backtestslitellmrun-20261111-220245#36774
[Bug]: default_priority not being used when no priority provided
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches fix the root cause the same way: acompletion resolves the explicit priority or the router's default_priority (handling 0 correctly, unlike the original `or`), pops it from kwargs, and runs the request through `_schedule_factory` with `async_function_with_fallbacks`, while `schedule_acompletion` delegates to acompletion so the request is queued exactly once. Both cover prompt-management models, atext_completion, and the /queue route, and both pass all seven independent tests. The difference is scope and risk. Patch A is a focused fix with targeted unit tests (default used when absent, explicit 0 overrides default, schedule_acompletion queues once, prompt-management and text-completion paths) that would fail before and pass after. Patch B wraps the same fix in a large redesign: a new request_priority module with pydantic validation, a new behavior that rejects non-integer priorities with a 400 (reversing an existing integration test that previously asserted such values bypass the scheduler and are forwarded), drop_params resolution across request/router/deployment levels, default_priority parsing with warnings, a change to `_wait_for_scheduler_turn` for pinned deployments, queue-route error remapping, and hundreds of lines of integration tests including one explicitly skipped as a known unfixed bug. None of that was asked for by the issue, it changes observable client behavior, and it makes the actual fix much harder to review and more likely to regress something adjacent. On correctness they are equivalent; on tests A's are sufficient and direct; on scope A clearly wins; on craft A is consistent with the surrounding code without introducing a parallel validation layer.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
27m 11s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
171vs 1,468 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
I'm using litellm-helm to deploy my litellm, I've configured it to have a default_priority as this :
proxy_config:
router_settings:
default_priority: 10
When using my client, it does not send a priority and so I get the following error :
Error: API error from litellm (model: code): LLM backend error [litellm]
status: 500 Internal Server Error
reason: Internal Server Error
request_id: N/A
endpoint: <proxy>/chat/completions
model: code
provider_message: Router.schedule_acompletion() missing 1 required positional argument: 'priority'
body_excerpt: {"error":{"message":"Router.schedule_acompletion() missing 1 required positional argument: 'priority'","type":"None","param":"None","code":"500"}}
payload_summary: {"model":"code","message_count":4,"approx_chars":13499,"temperature":0.2,"has_tools":true,"tool_choice":"auto"}
Run /retry [additional instructions] to continue the interrupted response.
As I have seen in the following code (router.py):
request_priority: Final = kwargs.get("priority") or self.default_priority
start_time: Final = time.time()
_is_prompt_management_model: Final = self._is_prompt_management_model(model)
if _is_prompt_management_model:
return await self._prompt_management_factory(
model=model,
messages=messages,
kwargs=kwargs,
)
if request_priority is not None and isinstance(request_priority, int):
response = await self.schedule_acompletion(**kwargs)
else:
response = await self.async_function_with_fallbacks(**kwargs)
the variable request_priority which get the potential default value isn't used in the call to the schedule_acompletion method hence the error.
User Flow
Before a (hypothetical) fix: a user calling chat/completion without a priority configured in the body.
- They send POST https://<proxy>/v1/chat/completions with no priority in the body.
- The HTTP response is a 500 Internal server Error with the message : "Router.schedule_acompletion() missing 1 required positional argument: 'priority'"
After a (hypothetical) fix: the same HTTP call chat/completion without a priority configured in the body.
- They send the same POST https://<proxy>/v1/chat/completions with no priority in the body.
- The HTTP response is 200 and the default_priority configured is used.
Proof the bug occurs
The code is still present in the latest versions 1.96.0, 1.96.2 and branches main, litellm_internal_staging.
Method acompletion :
async def acompletion(
self,
model: str,
messages: list[AllMessageValues],
stream: bool = False,
**kwargs,
):
try:
kwargs["model"] = model
kwargs["messages"] = messages
kwargs["stream"] = stream
kwargs["original_function"] = self._acompletion
self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)
request_priority: Final = kwargs.get("priority") or self.default_priority
start_time: Final = time.time()
_is_prompt_management_model: Final = self._is_prompt_management_model(model)
if _is_prompt_management_model:
return await self._prompt_management_factory(
model=model,
messages=messages,
kwargs=kwargs,
)
if request_priority is not None and isinstance(request_priority, int):
response = await self.schedule_acompletion(**kwargs)
else:
response = await self.async_function_with_fallbacks(**kwargs)
end_time: Final = time.time()
_duration: Final = end_time - start_time
asyncio.create_task(
self.service_logger_obj.async_service_success_hook(
service=ServiceTypes.ROUTER,
duration=_duration,
call_type="acompletion",
start_time=start_time,
end_time=end_time,
parent_otel_span=_get_parent_otel_span_from_kwargs(kwargs),
)
)
return response
except Exception as e:
asyncio.create_task(
send_llm_exception_alert(
litellm_router_instance=self,
request_kwargs=kwargs,
error_traceback_str=traceback.format_exc(),
original_exception=e,
)
)
raise e
Setup this configuration :
proxy_config:
router_settings:
default_priority: 10
Execute the curl request :
curl -X POST "https://<proxy>/v1/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <api_key>" \
-d '{
"model": "code",
"messages": [
{
"role": "system",
"content": "Say Hello"
},
{
"role": "user",
"content": "Hi"
}
],
"temperature": 0.2,
"stream": true,
"thinking": {
"type": "enabled",
"budget_tokens": 8192
}
}'
and you will get the following error :
`
{"error":{"message":"Router.schedule_acompletion() missing 1 required positional argument: 'priority'","type":"None","param":"None","code":"500"}}`
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
v1.96.0
Twitter / LinkedIn details
_No response_
The two patches
litellm/proxy/proxy_server.pyindex 1840586a08..e79c1da770 100644@@ -18281,6 +18281,8 @@ async def async_queue_request( if llm_router is None: raise HTTPException(status_code=500, detail={"error": CommonProxyErrors.no_llm_router.value}) router: Final = llm_router+ if data.get("priority") is None and router.default_priority is not None:+ data["priority"] = router.default_priority if "stream" in data and data["stream"] is True: # use generate_responses to stream responses litellm/router.pyindex e3c1e56b7e..24c90eb0c6 100644@@ -1219,7 +1219,7 @@ class Router: caching_groups (Optional[List[tuple]]): List of model groups for caching across model groups. Defaults to None. client_ttl (int): Time-to-live for cached clients in seconds. Defaults to 3600. polling_interval: (Optional[float]): frequency of polling queue. Only for '.scheduler_acompletion()'. Default is 3ms.- default_priority: (Optional[int]): the default priority for a request. Only for '.scheduler_acompletion()'. Default is None.+ default_priority: (Optional[int]): priority for scheduled requests that send none. Default is None. num_retries (Optional[int]): Number of retries for failed requests. Defaults to 2. timeout (Optional[float]): Timeout for requests. Defaults to None. default_litellm_params (dict): Default parameters for Router.chat.completion.create. Defaults to {}.@@ -3188,11 +3188,27 @@ class Router: kwargs["original_function"] = self._acompletion self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)- request_priority: Final = kwargs.get("priority") or self.default_priority+ explicit_priority: Final = kwargs.get("priority")+ request_priority: Final = explicit_priority if explicit_priority is not None else self.default_priority start_time: Final = time.time() _is_prompt_management_model: Final = self._is_prompt_management_model(model) if _is_prompt_management_model:+ if request_priority is not None and isinstance(request_priority, int):+ kwargs.pop("priority", None)++ async def _scheduled_prompt_management(**scheduled_kwargs: Any):+ return await self._prompt_management_factory(+ model=model, messages=messages, kwargs=scheduled_kwargs+ )++ return await self._schedule_factory(+ model=model,+ priority=request_priority,+ original_function=_scheduled_prompt_management,+ args=(),+ kwargs=kwargs,+ ) return await self._prompt_management_factory( model=model, messages=messages,@@ -3201,7 +3217,15 @@ class Router: controls: Final = per_request_fallback_controls(kwargs) kwargs[MID_STREAM_FALLBACK_CONTROLS_KEY] = controls # rebind-ok: forwarded to every hop if request_priority is not None and isinstance(request_priority, int):- response = await self.schedule_acompletion(**kwargs)+ # the scheduler consumes the priority; it is not forwarded to the deployment call+ kwargs.pop("priority", None)+ response = await self._schedule_factory(+ model=model,+ priority=request_priority,+ original_function=self.async_function_with_fallbacks,+ args=(),+ kwargs=kwargs,+ ) else: response = await self.async_function_with_fallbacks(**kwargs) end_time: Final = time.time()@@ -5067,24 +5091,9 @@ class Router: stream=False, **kwargs, ):- await self._wait_for_scheduler_turn(- model=model,- priority=priority,- parent_otel_span=get_parent_otel_span_from_kwargs(kwargs),- request_kwargs=cast(Mapping[str, object], kwargs), # cast-ok: kwargs keys are param names- )- try:- _response: Final = await self.acompletion(model=model, messages=messages, stream=stream, **kwargs)- response_hidden_params: Final = get_hidden_params(_response)- if response_hidden_params is not None:- additional_headers: Final = cast( # cast-ok: router headers are stored as a mutable mapping- dict[str, object], response_hidden_params.setdefault("additional_headers", {})- )- additional_headers.update({"x-litellm-request-prioritization-used": True})- return _response- except Exception as e:- setattr(e, "priority", priority)- raise e+ # acompletion waits for the scheduler turn exactly once for this priority; awaiting the+ # turn here and then calling acompletion would queue the request again under default_priority+ return await self.acompletion(model=model, messages=messages, stream=stream, priority=priority, **kwargs) async def _schedule_factory( self,@@ -5646,20 +5655,23 @@ class Router: is_async: bool | None = False, **kwargs, ):- if kwargs.get("priority", None) is not None:- return await self._schedule_factory(- model=model,- priority=kwargs.pop("priority"),- original_function=self.atext_completion,- args=(model, prompt),- kwargs=kwargs,- ) try:+ # the scheduler consumes the priority; it is not forwarded to the deployment call+ explicit_priority: Final = kwargs.pop("priority", None)+ request_priority: Final = explicit_priority if explicit_priority is not None else self.default_priority kwargs["model"] = model kwargs["prompt"] = prompt kwargs["original_function"] = self._atext_completion self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)+ if request_priority is not None:+ return await self._schedule_factory(+ model=model,+ priority=request_priority,+ original_function=self.async_function_with_fallbacks,+ args=(),+ kwargs=kwargs,+ ) response: Final = await self.async_function_with_fallbacks(**kwargs) return responsetests/unit/proxy/proxy_server/test_routes_queue.pyindex 27cc630071..0f841bb571 100644@@ -82,6 +82,17 @@ def test_queue_chat_completions_happy(client, auth_as, patched_queue): } +def test_queue_chat_completions_without_priority_uses_router_default_priority(client, auth_as, patched_queue):+ """``POST /queue/chat/completions`` with no priority is queued under the router's default_priority."""+ patched_queue.default_priority = 10+ payload = {"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}]}+ with auth_as():+ response = client.post("/queue/chat/completions", json=payload)+ assert response.status_code == 200+ assert response.headers["x-litellm-priority"] == "10"+ assert patched_queue.schedule_acompletion.call_args.kwargs["priority"] == 10++ def test_queue_chat_completions_no_router_error(client, auth_as, queue_no_router): """Pins ``POST /queue/chat/completions`` (error: no llm_router).""" payload = {"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}]}tests/unit/test_router/test_router_helper_utils.pyindex e3061294be..875329179b 100644@@ -241,6 +241,94 @@ async def test_router_schedule_acompletion(model_list): assert response["choices"][0]["message"]["content"] == "I'm fine, thank you!" +@pytest.mark.asyncio+async def test_router_acompletion_without_priority_is_scheduled_with_default_priority(model_list):+ """A request with no priority is queued under the router's default_priority instead of failing."""+ router = Router(model_list=model_list, default_priority=10)+ with patch.object(router, "_wait_for_scheduler_turn", wraps=router._wait_for_scheduler_turn) as wait_for_turn:+ response = await router.acompletion(+ model="gpt-5-mini",+ messages=[{"role": "user", "content": "Hello, how are you?"}],+ mock_response="I'm fine, thank you!",+ )+ assert response["choices"][0]["message"]["content"] == "I'm fine, thank you!"+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+ wait_for_turn.assert_awaited_once()+ assert wait_for_turn.call_args.kwargs["priority"] == 10+++@pytest.mark.asyncio+async def test_router_acompletion_explicit_zero_priority_overrides_default_priority(model_list):+ router = Router(model_list=model_list, default_priority=10)+ with patch.object(router, "_wait_for_scheduler_turn", wraps=router._wait_for_scheduler_turn) as wait_for_turn:+ response = await router.acompletion(+ model="gpt-5-mini",+ messages=[{"role": "user", "content": "Hello, how are you?"}],+ mock_response="I'm fine, thank you!",+ priority=0,+ )+ assert response["choices"][0]["message"]["content"] == "I'm fine, thank you!"+ wait_for_turn.assert_awaited_once()+ assert wait_for_turn.call_args.kwargs["priority"] == 0+++@pytest.mark.asyncio+async def test_router_schedule_acompletion_with_default_priority_queues_once(model_list):+ """schedule_acompletion on a router with default_priority queues the request once, at the given priority."""+ router = Router(model_list=model_list, default_priority=10)+ with patch.object(router, "_wait_for_scheduler_turn", wraps=router._wait_for_scheduler_turn) as wait_for_turn:+ response = await router.schedule_acompletion(+ model="gpt-5-mini",+ messages=[{"role": "user", "content": "Hello, how are you?"}],+ mock_response="I'm fine, thank you!",+ priority=1,+ )+ assert response["choices"][0]["message"]["content"] == "I'm fine, thank you!"+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+ wait_for_turn.assert_awaited_once()+ assert wait_for_turn.call_args.kwargs["priority"] == 1+++@pytest.mark.asyncio+async def test_router_acompletion_prompt_management_model_is_scheduled_with_default_priority(model_list):+ """Prompt-management models are also queued under default_priority, and priority is not forwarded."""+ router = Router(model_list=model_list, default_priority=10)+ response = ModelResponse()+ with (+ patch.object(router, "_is_prompt_management_model", return_value=True),+ patch.object(router, "_prompt_management_factory", AsyncMock(return_value=response)) as mock_factory,+ patch.object(router, "_wait_for_scheduler_turn", wraps=router._wait_for_scheduler_turn) as wait_for_turn,+ ):+ result = await router.acompletion(+ model="gpt-5-mini",+ messages=[{"role": "user", "content": "Hello, how are you?"}],+ )+ assert result is response+ assert result._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+ mock_factory.assert_awaited_once()+ assert "priority" not in mock_factory.call_args.kwargs["kwargs"]+ wait_for_turn.assert_awaited_once()+ assert wait_for_turn.call_args.kwargs["priority"] == 10+++@pytest.mark.asyncio+async def test_router_atext_completion_without_priority_is_scheduled_with_default_priority(model_list):+ from litellm.types.utils import TextCompletionResponse++ router = Router(model_list=model_list, default_priority=10)+ with (+ patch.object(router, "_atext_completion", AsyncMock()) as mock_atext_completion,+ patch.object(router, "_wait_for_scheduler_turn", wraps=router._wait_for_scheduler_turn) as wait_for_turn,+ ):+ mock_atext_completion.return_value = TextCompletionResponse()+ response = await router.atext_completion(model="gpt-5-mini", prompt="Hello, how are you?")+ assert isinstance(response, TextCompletionResponse)+ mock_atext_completion.assert_awaited_once()+ assert "priority" not in mock_atext_completion.call_args.kwargs+ wait_for_turn.assert_awaited_once()+ assert wait_for_turn.call_args.kwargs["priority"] == 10++ @pytest.mark.asyncio async def test_router_schedule_atext_completion(model_list): """Test if the 'schedule_atext_completion' function is working correctly"""
litellm/litellm_core_utils/core_helpers.pyindex 1e5bc7ecba4..b7c6f805233 100644@@ -520,7 +520,7 @@ def reconstruct_model_name( # Helper functions used for OTEL logging def get_parent_otel_span_from_kwargs(- kwargs: dict[str, object] | None = None,+ kwargs: Mapping[str, object] | None = None, ) -> Span | None: try: if kwargs is None:litellm/proxy/proxy_server.pyindex 1840586a08e..ee8f20be243 100644@@ -18282,6 +18282,7 @@ async def async_queue_request( raise HTTPException(status_code=500, detail={"error": CommonProxyErrors.no_llm_router.value}) router: Final = llm_router + queued_priority, _ = router.split_request_priority(model=data["model"], kwargs=data) if "stream" in data and data["stream"] is True: # use generate_responses to stream responses async def produce_queue_stream() -> StreamingResponse:@@ -18293,6 +18294,7 @@ async def async_queue_request( request=request, ), media_type="text/event-stream",+ headers={"x-litellm-priority": str(queued_priority)}, ) async def audit_late_failure(exc: Exception) -> HTTPException | None:@@ -18307,7 +18309,7 @@ async def async_queue_request( ) response: Final = await router.schedule_acompletion(**data)- fastapi_response.headers.update({"x-litellm-priority": str(data["priority"])})+ fastapi_response.headers.update({"x-litellm-priority": str(queued_priority)}) return response except Exception as e: await proxy_logging_obj.post_call_failure_hook(@@ -18322,6 +18324,10 @@ async def async_queue_request( ) elif isinstance(e, ProxyException): raise e+ elif isinstance(e, litellm.BadRequestError):+ raise ProxyException(+ message=e.message, type="invalid_request_error", param=getattr(e, "param", None), code=e.status_code+ ) raise ProxyException( message="Authentication Error, " + str(e), type=ProxyErrorTypes.auth_error,litellm/router.pyindex e3c1e56b7e8..ba3b4d53725 100644@@ -327,6 +327,14 @@ from litellm.router_utils.reasoning_effort_capability import ( intersect_supported_reasoning_efforts, resolve_supported_reasoning_efforts, )+from litellm.router_utils.request_priority import (+ InvalidPriority,+ parse_default_priority,+ priority_error,+ request_drops_params,+ resolve_request_priority,+ unqueueable_priority_message,+) from litellm.router_utils.router_callbacks.track_deployment_metrics import ( find_deployment_metadata, get_counted_usage_tokens,@@ -1219,7 +1227,7 @@ class Router: caching_groups (Optional[List[tuple]]): List of model groups for caching across model groups. Defaults to None. client_ttl (int): Time-to-live for cached clients in seconds. Defaults to 3600. polling_interval: (Optional[float]): frequency of polling queue. Only for '.scheduler_acompletion()'. Default is 3ms.- default_priority: (Optional[int]): the default priority for a request. Only for '.scheduler_acompletion()'. Default is None.+ default_priority: (Optional[int]): the priority given to every chat or text completion request that sets no 'priority' of its own; '.acompletion()' and '.atext_completion()' queue it through the scheduler. Default is None. num_retries (Optional[int]): Number of retries for failed requests. Defaults to 2. timeout (Optional[float]): Timeout for requests. Defaults to None. default_litellm_params (dict): Default parameters for Router.chat.completion.create. Defaults to {}.@@ -1359,7 +1367,15 @@ class Router: ### SCHEDULER ### self.scheduler = Scheduler(polling_interval=polling_interval, redis_cache=redis_cache)- self.default_priority = default_priority+ parsed_default_priority: Final = parse_default_priority(default_priority)+ if isinstance(parsed_default_priority, InvalidPriority):+ verbose_router_logger.warning(+ "router_settings.default_priority is ignored, no request is queued by default: %s",+ parsed_default_priority.message,+ )+ self.default_priority = (+ None if isinstance(parsed_default_priority, InvalidPriority) else parsed_default_priority+ ) self.default_deployment = ( None # use this to track the users default deployment, when they want to use model = * )@@ -3188,22 +3204,27 @@ class Router: kwargs["original_function"] = self._acompletion self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)- request_priority: Final = kwargs.get("priority") or self.default_priority start_time: Final = time.time()- _is_prompt_management_model: Final = self._is_prompt_management_model(model)-- if _is_prompt_management_model:- return await self._prompt_management_factory(+ is_prompt_management_model: Final = self._is_prompt_management_model(model)+ if not is_prompt_management_model:+ controls: Final = per_request_fallback_controls(kwargs)+ kwargs[MID_STREAM_FALLBACK_CONTROLS_KEY] = controls # rebind-ok: forwarded to every hop+ request_priority, request_kwargs = self.split_request_priority(model=model, kwargs=kwargs)+ original_function: Final[Callable[..., Awaitable[object]]] = (+ self._prompt_management_acompletion(model=model, messages=messages)+ if is_prompt_management_model+ else self.async_function_with_fallbacks+ )+ if request_priority is None:+ response = await original_function(**request_kwargs)+ else:+ response = await self._schedule_factory( model=model,- messages=messages,- kwargs=kwargs,+ priority=request_priority,+ original_function=original_function,+ args=(),+ kwargs=request_kwargs, )- controls: Final = per_request_fallback_controls(kwargs)- kwargs[MID_STREAM_FALLBACK_CONTROLS_KEY] = controls # rebind-ok: forwarded to every hop- if request_priority is not None and isinstance(request_priority, int):- response = await self.schedule_acompletion(**kwargs)- else:- response = await self.async_function_with_fallbacks(**kwargs) end_time: Final = time.time() _duration: Final = end_time - start_time asyncio.create_task(@@ -5047,13 +5068,18 @@ class Router: @overload async def schedule_acompletion(- self, model: str, messages: list[AllMessageValues], priority: int, stream: Literal[False] = False, **kwargs+ self,+ model: str,+ messages: list[AllMessageValues],+ priority: int | None = None,+ stream: Literal[False] = False,+ **kwargs, ) -> ModelResponse: ... @overload async def schedule_acompletion(- self, model: str, messages: list[AllMessageValues], priority: int, stream: Literal[True], **kwargs+ self, model: str, messages: list[AllMessageValues], priority: int | None, stream: Literal[True], **kwargs ) -> CustomStreamWrapper: ... @@ -5063,36 +5089,36 @@ class Router: self, model: str, messages: list[AllMessageValues],- priority: int,+ priority: int | None = None, stream=False, **kwargs, ):- await self._wait_for_scheduler_turn(- model=model,- priority=priority,- parent_otel_span=get_parent_otel_span_from_kwargs(kwargs),- request_kwargs=cast(Mapping[str, object], kwargs), # cast-ok: kwargs keys are param names+ queued_priority, _ = self.split_request_priority(model=model, kwargs={**kwargs, "priority": priority})+ if queued_priority is None:+ raise priority_error(model=model, message=unqueueable_priority_message(priority))+ return await self.acompletion(model=model, messages=messages, stream=stream, priority=queued_priority, **kwargs)++ def split_request_priority(+ self, model: str, kwargs: Mapping[str, object]+ ) -> tuple[int | None, Mapping[str, object]]:+ request_priority: Final = resolve_request_priority(+ requested=kwargs.get("priority"),+ default_priority=self.default_priority,+ drops_params=lambda: request_drops_params(+ kwargs, self.default_litellm_params, self._request_deployment_params(model=model, kwargs=kwargs)+ ), )- try:- _response: Final = await self.acompletion(model=model, messages=messages, stream=stream, **kwargs)- response_hidden_params: Final = get_hidden_params(_response)- if response_hidden_params is not None:- additional_headers: Final = cast( # cast-ok: router headers are stored as a mutable mapping- dict[str, object], response_hidden_params.setdefault("additional_headers", {})- )- additional_headers.update({"x-litellm-request-prioritization-used": True})- return _response- except Exception as e:- setattr(e, "priority", priority)- raise e+ if isinstance(request_priority, InvalidPriority):+ raise priority_error(model=model, message=request_priority.message)+ return request_priority, {key: value for key, value in kwargs.items() if key != "priority"} async def _schedule_factory( self, model: str, priority: int,- original_function: Callable,+ original_function: Callable[..., Awaitable[object]], args: tuple[object, ...],- kwargs: dict[str, object],+ kwargs: Mapping[str, object], ): await self._wait_for_scheduler_turn( model=model,@@ -5116,7 +5142,11 @@ class Router: async def _wait_for_scheduler_turn( self, model: str, priority: int, parent_otel_span: Span | None, request_kwargs: Mapping[str, object] ) -> None:+ pinned: Final = self._pinned_deployment(model)+ async def healthy_deployments() -> Sequence[object]:+ if pinned is not None:+ return (pinned,) deployments, _ = await self._async_get_healthy_deployments( model=model, parent_otel_span=parent_otel_span,@@ -5142,6 +5172,30 @@ class Router: split_litellm_model: Final = litellm_model.split("/")[0] return split_litellm_model in litellm._known_custom_logger_compatible_callbacks + def _pinned_deployment(self, model: str) -> Deployment | None:+ return self.get_deployment(model_id=model) if model not in self.model_names else None++ def _request_deployment_params(self, model: str, kwargs: Mapping[str, object]) -> tuple[Mapping[str, object], ...]:+ deployment: Final = self._pinned_deployment(model)+ if deployment is not None:+ return (deployment.litellm_params.model_dump(exclude_none=True),)+ candidates: Final = (+ self._get_deployment_by_litellm_model(model=model)+ if kwargs.get("specific_deployment") is True+ else self.get_model_list(model_name=model, team_id=get_request_team_id(kwargs)) or ()+ )+ return tuple(candidate["litellm_params"] for candidate in candidates)++ def _prompt_management_acompletion(+ self,+ model: str,+ messages: list[AllMessageValues], # mutable-ok: _prompt_management_factory takes the caller's list+ ) -> Callable[..., Awaitable[object]]:+ async def acompletion(**kwargs: object) -> object: # kwargs-ok: called with the request's own keyword payload+ return await self._prompt_management_factory(model=model, messages=messages, kwargs=dict(kwargs))++ return acompletion+ async def _prompt_management_factory( self, model: str,@@ -5646,23 +5700,22 @@ class Router: is_async: bool | None = False, **kwargs, ):- if kwargs.get("priority", None) is not None:- return await self._schedule_factory(- model=model,- priority=kwargs.pop("priority"),- original_function=self.atext_completion,- args=(model, prompt),- kwargs=kwargs,- ) try: kwargs["model"] = model kwargs["prompt"] = prompt kwargs["original_function"] = self._atext_completion self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)- response: Final = await self.async_function_with_fallbacks(**kwargs)-- return response+ request_priority, request_kwargs = self.split_request_priority(model=model, kwargs=kwargs)+ if request_priority is None:+ return await self.async_function_with_fallbacks(**request_kwargs)+ return await self._schedule_factory(+ model=model,+ priority=request_priority,+ original_function=self.async_function_with_fallbacks,+ args=(),+ kwargs=request_kwargs,+ ) except Exception as e: asyncio.create_task( send_llm_exception_alert(@@ -14122,7 +14175,7 @@ class Router: _get_model_from_alias = get_model_from_alias - def _get_deployment_by_litellm_model(self, model: str) -> list:+ def _get_deployment_by_litellm_model(self, model: str) -> list[DeploymentTypedDict]: """ Get the deployment by litellm model. """litellm/router_utils/request_priority.pynew file mode 100644index 00000000000..14608cc1bcf@@ -0,0 +1,88 @@+from collections.abc import Callable, Iterable, Mapping+from dataclasses import dataclass+from typing import Final++from pydantic import TypeAdapter, ValidationError++import litellm+from litellm.litellm_core_utils.core_helpers import normalize_drop_params+++@dataclass(frozen=True, slots=True)+class InvalidPriority:+ value: object++ @property+ def message(self) -> str:+ return f"priority must be an integer, got {type(self.value).__name__} {self.value!r}"+++def priority_error(model: str, message: str) -> litellm.BadRequestError:+ return litellm.BadRequestError(+ message=message,+ model=model,+ llm_provider="",+ body={"message": message, "type": "invalid_request_error", "param": "priority", "code": "400"},+ )+++_PRIORITY: Final = TypeAdapter(int)+++def _as_priority(value: object) -> int | InvalidPriority:+ try:+ return _PRIORITY.validate_python(value)+ except ValidationError:+ return InvalidPriority(value=value)+++def resolve_request_priority(+ requested: object, default_priority: int | None, drops_params: Callable[[], bool]+) -> int | None | InvalidPriority:+ if requested is None:+ return default_priority+ priority: Final = _as_priority(requested)+ if isinstance(priority, InvalidPriority) and drops_params():+ return default_priority+ return priority+++def parse_default_priority(value: object) -> int | None | InvalidPriority:+ if value is None:+ return None+ if isinstance(value, bool):+ return InvalidPriority(value=value)+ return _as_priority(value)+++def unqueueable_priority_message(requested: object) -> str:+ priority: Final = None if requested is None else _as_priority(requested)+ if isinstance(priority, InvalidPriority):+ return f"{priority.message}, and there is no router_settings.default_priority to queue it at instead"+ return "priority is required to queue a request, send one or set router_settings.default_priority"+++def _explicit_drop_params(scope: Mapping[str, object]) -> bool | None:+ return normalize_drop_params(scope.get("drop_params"))+++def _drops_priority_by_name(scope: Mapping[str, object]) -> bool:+ names: Final = scope.get("additional_drop_params")+ return isinstance(names, list) and "priority" in names+++def request_drops_params(+ kwargs: Mapping[str, object],+ router_defaults: Mapping[str, object],+ deployment_params: Iterable[Mapping[str, object]],+) -> bool:+ deployments: Final = tuple(deployment_params)+ if litellm.drop_params is True or any(_drops_priority_by_name(s) for s in (kwargs, router_defaults, *deployments)):+ return True+ request_flag: Final = _explicit_drop_params(kwargs)+ if request_flag is not None:+ return request_flag+ router_flag: Final = _explicit_drop_params(router_defaults)+ if router_flag is not None:+ return router_flag+ return any(_explicit_drop_params(params) is True for params in deployments)tests/code_coverage_tests/router_code_coverage.pyindex 3f0ad07a876..7809a23e955 100644@@ -114,6 +114,10 @@ ignored_function_names = [ "_aanthropic_messages_yield_recovered", # Tested through every mid-stream retry and fallback test in test_router.py "_anthropic_messages_policy_retries", # Tested through the retry budget precedence test in test_router.py "_get_wildcard_deployments", # Tested through the get_model_list_of_routed_group wildcard test in test_router.py+ "_prompt_management_acompletion", # Tested through the scheduled prompt-management model test in test_router.py+ "_request_deployment_params", # Tested through the team-route and deployment-id drop_params priority tests in test_router.py+ "split_request_priority", # Tested through the acompletion, atext_completion and queue route priority tests+ "_pinned_deployment", # Tested through the deployment-id scheduler test in test_router.py "_is_fallback_hop", # Tested through the order fallback hop tests in test_router_order_fallback.py "_deployment_that_just_failed", # Tested through the same-boundary hop test in test_router_order_fallback.py "_merged_tools", # Tested through the _update_kwargs_with_deployment tools merge tests in test_router.pytests/integration/routing/test_priority_scheduler_queue_cleanup.pyindex 993da52fc67..9fc21843b04 100644@@ -11,6 +11,7 @@ from collections.abc import Callable, Iterator, Mapping, Sequence from concurrent.futures import ThreadPoolExecutor from contextlib import ExitStack from dataclasses import dataclass+from itertools import product from pathlib import Path from queue import SimpleQueue from typing import Final, Literal, TypeVar@@ -24,7 +25,7 @@ import yaml from integration._support.client import Gateway, eventually, gateway_from_environment from integration._support.database import read_rows from integration._support.openai_wire import answering_model_discovery, chat_reply, openai_error, responses_reply-from integration._support.process import graceful_stop_seconds, owned_proxy_process+from integration._support.process import OwnedState, graceful_stop_seconds, owned_proxy_process from integration._support.redis_process import OwnedRedis, owned_redis from integration._support.timing import assert_cut_short from integration._support.wire import Reply, Request, Wire, wire_server@@ -61,6 +62,18 @@ INMEM_GROUP: Final = "sched-inmem" OUTAGE_GROUP: Final = "sched-outage" KILL_GROUP: Final = "sched-kill" GHOST_ENTRY: Final[list[JsonValue]] = [1, "ghost"]+DEFAULT_PRIORITY: Final = 3+DEFAULT_GROUPS: Final = tuple(f"sched-default-{row}" for row in ("d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9"))+PROMPT_GROUP: Final = "sched-default-prompt"+PROMPT_ID: Final = "sched_default"+PROMPT_PREFIX: Final = "Scheduled by default for"+DROP_GROUP: Final = "sched-default-drop"+TEAM_GROUP: Final = "sched-default-team"+ZERO_GROUPS: Final = ("sched-zero-plain", "sched-zero-dropped")+DIGIT_GROUPS: Final = ("sched-digit-plain", "sched-digit-dropped")+IGNORED_GROUP: Final = "sched-ignored"+PRIORITIZED: Final = "x-litellm-request-prioritization-used"+IGNORED_WARNING: Final = "router_settings.default_priority is ignored" JSON_OBJECT: Final = TypeAdapter(dict[str, JsonValue]) JSON_LIST: Final = TypeAdapter(list[JsonValue]) @@ -160,15 +173,24 @@ def assert_never(received: Sequence[str], markers: Sequence[str]) -> None: assert not set(markers) & set(received), (markers, received) -def deployment_entry(group: str, wire: Wire) -> dict[str, JsonValue]:+def deployment_entry(+ group: str,+ wire: Wire,+ *,+ model_name: str | None = None,+ model: str | None = None,+ model_info: Mapping[str, JsonValue] | None = None,+ **litellm_params: JsonValue,+) -> dict[str, JsonValue]: return {- "model_name": group,+ "model_name": model_name or group, "litellm_params": {- "model": f"openai/{upstream_name(group)}",+ "model": model or f"openai/{upstream_name(group)}", "api_base": wire.url + "/v1", "api_key": "synthetic-openai-key",+ **litellm_params, },- "model_info": {"id": f"{group}-deployment"},+ "model_info": {"id": f"{group}-deployment", **(model_info or {})}, } @@ -179,12 +201,16 @@ def owned_config( settings: Mapping[str, JsonValue], *, cancel_on_disconnect: bool = False,+ extra_models: Sequence[Mapping[str, JsonValue]] = (),+ litellm_settings: Mapping[str, JsonValue] | None = None, ) -> Path: base: Final = JSON_OBJECT.validate_python( yaml.safe_load((Path(__file__).parents[1] / "proxy_config.yaml").read_text()) ) general_settings: Final = base.get("general_settings", {}) assert isinstance(general_settings, dict)+ base_litellm_settings: Final = base.get("litellm_settings", {})+ assert isinstance(base_litellm_settings, dict) general: Final[dict[str, JsonValue]] = { **general_settings, **({"cancel_on_disconnect": True} if cancel_on_disconnect else {}),@@ -192,8 +218,9 @@ def owned_config( config: Final[dict[str, JsonValue]] = { **base, "general_settings": general,+ "litellm_settings": {**base_litellm_settings, **(litellm_settings or {})}, "router_settings": dict(settings),- "model_list": [deployment_entry(group, wire) for group in groups],+ "model_list": [*(deployment_entry(group, wire) for group in groups), *(dict(entry) for entry in extra_models)], } path: Final = directory / f"scheduler-{uuid.uuid4().hex[:8]}.yaml" path.write_text(yaml.safe_dump(config))@@ -209,6 +236,10 @@ def redis_settings(cache: OwnedRedis) -> dict[str, JsonValue]: return {"num_retries": 0, "redis_host": cache.host, "redis_port": cache.port} +def priority_field(priority: int | None) -> dict[str, JsonValue]:+ return {} if priority is None else {"priority": priority}++ def chat_body(group: str, marker: str, **extra: JsonValue) -> dict[str, JsonValue]: return {"model": group, "messages": [{"role": "user", "content": marker}], **extra} @@ -279,26 +310,21 @@ def stream_identities(text: str) -> frozenset[str]: return frozenset(str(frame["id"]) for frame in frames) -def call(gateway: Gateway, endpoint: Endpoint, group: str, marker: str, priority: int = 1) -> Answer:+def call(gateway: Gateway, endpoint: Endpoint, group: str, marker: str, priority: int | None = 1) -> Answer:+ field: Final = priority_field(priority) match endpoint: case "chat":- return post(gateway, "/v1/chat/completions", chat_body(group, marker, priority=priority))+ return post(gateway, "/v1/chat/completions", chat_body(group, marker, **field)) case "chat-stream":- return stream_text(- gateway, "/v1/chat/completions", chat_body(group, marker, priority=priority, stream=True)- )+ return stream_text(gateway, "/v1/chat/completions", chat_body(group, marker, stream=True, **field)) case "completions":- return post(gateway, "/v1/completions", completion_body(group, marker, priority=priority))+ return post(gateway, "/v1/completions", completion_body(group, marker, **field)) case "completions-stream":- return stream_text(- gateway, "/v1/completions", completion_body(group, marker, priority=priority, stream=True)- )+ return stream_text(gateway, "/v1/completions", completion_body(group, marker, stream=True, **field)) case "queue":- return post(gateway, "/queue/chat/completions", chat_body(group, marker, priority=priority))+ return post(gateway, "/queue/chat/completions", chat_body(group, marker, **field)) case "queue-stream":- return stream_text(- gateway, "/queue/chat/completions", chat_body(group, marker, priority=priority, stream=True)- )+ return stream_text(gateway, "/queue/chat/completions", chat_body(group, marker, stream=True, **field)) def served_identity(status: int, text: str, marker: str) -> str:@@ -363,15 +389,19 @@ def waiting_entries(cache: OwnedRedis, group: str, priority: int) -> list[JsonVa ) -def cooled(gateway: Gateway, upstream: Upstream, group: str) -> None:+def cooled_with(upstream: Upstream, group: str, send: Callable[[], Answer]) -> None: upstream.refusing[upstream_name(group)].set()- trip: Final = post(gateway, "/v1/chat/completions", chat_body(group, new_marker()))+ trip: Final = send() assert trip.status == 401, (trip.status, trip.text)- eventually(- lambda: post(gateway, "/v1/chat/completions", chat_body(group, new_marker())),- lambda answer: answer.status == 429 and NO_DEPLOYMENTS in answer.text,- seconds=10,- )+ eventually(send, lambda answer: answer.status == 429 and NO_DEPLOYMENTS in answer.text, seconds=10)+++def cooled(gateway: Gateway, upstream: Upstream, group: str) -> None:+ cooled_with(upstream, group, lambda: post(gateway, "/v1/chat/completions", chat_body(group, new_marker())))+++def cooled_outside_the_scheduler(gateway: Gateway, upstream: Upstream, group: str) -> None:+ cooled_with(upstream, group, lambda: post(gateway, "/v1/responses", {"model": group, "input": new_marker()})) def assert_refused_at_once(answer: Answer) -> None:@@ -488,16 +518,42 @@ def test_other_prioritized_endpoints_are_served_once_and_billed( spend_row_lands(identity) -@pytest.mark.parametrize("priority", ("1", [1], "", "p" * 5120, 0), ids=("string", "list", "empty", "5kb", "zero"))-def test_non_integer_priority_bypasses_the_scheduler_and_is_forwarded(+def upstream_body_for(wire: Wire, marker: str) -> dict[str, JsonValue]:+ (upstream_request,) = tuple(request for request in wire.drain() if marker.encode() in request.body)+ return JSON_OBJECT.validate_json(upstream_request.body)+++@pytest.mark.parametrize("priority", ("high", [1], "", "p" * 5120, 1.5), ids=("word", "list", "empty", "5kb", "float"))+def test_non_integer_priority_is_refused_before_the_upstream( gateway: Gateway, rig_model: tuple[str, Wire], priority: JsonValue ) -> None: model, wire = rig_model marker: Final = new_marker() answer: Final = post(gateway, "/v1/chat/completions", chat_body(model, marker, priority=priority))+ assert answer.status == 400, (answer.status, answer.text)+ error: Final = JSON_OBJECT.validate_json(answer.text)["error"]+ assert isinstance(error, dict), answer.text+ assert error["param"] == "priority", answer.text+ assert "priority must be an integer" in str(error["message"]), answer.text+ assert_never(received_markers(wire), (marker,))+++def test_non_integer_priority_is_dropped_under_drop_params(gateway: Gateway, rig_model: tuple[str, Wire]) -> None:+ model, wire = rig_model+ marker: Final = new_marker()+ answer: Final = post(gateway, "/v1/chat/completions", chat_body(model, marker, priority="high", drop_params=True)) assert_served(answer, marker)- (upstream_request,) = tuple(request for request in wire.drain() if marker.encode() in request.body)- assert JSON_OBJECT.validate_json(upstream_request.body).get("priority") == priority+ assert answer.headers.get("x-litellm-request-prioritization-used") is None, answer.headers+ assert "priority" not in upstream_body_for(wire, marker)+++def test_zero_priority_is_scheduled_and_not_forwarded(gateway: Gateway, rig_model: tuple[str, Wire]) -> None:+ model, wire = rig_model+ marker: Final = new_marker()+ answer: Final = post(gateway, "/v1/chat/completions", chat_body(model, marker, priority=0))+ assert_served(answer, marker)+ assert answer.headers.get("x-litellm-request-prioritization-used") == "True", answer.headers+ assert "priority" not in upstream_body_for(wire, marker) def test_unauthenticated_prioritized_request_never_reaches_the_upstream(@@ -747,7 +803,9 @@ def test_disconnected_waiter_is_removed_from_the_queue(pair: Pair) -> None: Outcome = Answer | httpx.TransportError -def burst(gateway: Gateway, group: str, count: int) -> tuple[tuple[Endpoint, str, Outcome], ...]:+def burst(+ gateway: Gateway, group: str, count: int, priority: int | None = 1+) -> tuple[tuple[Endpoint, str, Outcome], ...]: plan: Final[tuple[tuple[Endpoint, str], ...]] = tuple( (ENDPOINTS[index % len(ENDPOINTS)], new_marker()) for index in range(count) )@@ -756,7 +814,7 @@ def burst(gateway: Gateway, group: str, count: int) -> tuple[tuple[Endpoint, str endpoint, marker = item with httpx.Client(base_url=gateway.client.base_url, timeout=60, trust_env=False) as client: try:- return call(Gateway(client, gateway.key, gateway.upstream_url), endpoint, group, marker)+ return call(Gateway(client, gateway.key, gateway.upstream_url), endpoint, group, marker, priority) except httpx.TransportError as error: return error @@ -772,10 +830,14 @@ def assert_all_served(outcomes: Sequence[tuple[Endpoint, str, Outcome]]) -> tupl return tuple(marker for _, marker, _ in outcomes) -def served_eventually(gateway: Gateway, group: str, marker: str) -> None:+def default_for(priority: int | None) -> dict[str, JsonValue]:+ return {} if priority is not None else {"default_priority": DEFAULT_PRIORITY}+++def served_eventually(gateway: Gateway, group: str, marker: str, priority: int | None = 1) -> None: def send() -> Outcome: try:- return post(gateway, "/v1/chat/completions", chat_body(group, marker, priority=1))+ return post(gateway, "/v1/chat/completions", chat_body(group, marker, **priority_field(priority))) except httpx.TransportError as error: return error @@ -784,22 +846,25 @@ def served_eventually(gateway: Gateway, group: str, marker: str) -> None: assert_served(answer, marker) +@pytest.mark.parametrize("priority", (1, None), ids=("explicit", "default")) @pytest.mark.timeout(OWNED_CELL_TIMEOUT)-def test_prioritized_requests_survive_a_redis_outage(gateway: Gateway, tmp_path: Path) -> None:+def test_prioritized_requests_survive_a_redis_outage(gateway: Gateway, tmp_path: Path, priority: int | None) -> None: upstream: Final = upstream_for((OUTAGE_GROUP,)) with owned_redis(tmp_path) as cache, wire_server(answering_model_discovery(upstream.respond)) as wire:- config: Final = owned_config(tmp_path, wire, (OUTAGE_GROUP,), redis_settings(cache))+ config: Final = owned_config(+ tmp_path, wire, (OUTAGE_GROUP,), {**redis_settings(cache), **default_for(priority)}+ ) overrides: Final = { "REDIS_HOST": cache.host, "REDIS_PORT": str(cache.port), "REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT": "1", } with owned_proxy_process(gateway, tmp_path, overrides, config=config, workers=2) as owned:- before: Final = assert_all_served(burst(owned.gateway, OUTAGE_GROUP, 12))+ before: Final = assert_all_served(burst(owned.gateway, OUTAGE_GROUP, 12, priority)) cache.stop()- during: Final = assert_all_served(burst(owned.gateway, OUTAGE_GROUP, 12))+ during: Final = assert_all_served(burst(owned.gateway, OUTAGE_GROUP, 12, priority)) cache.start()- after: Final = assert_all_served(burst(owned.gateway, OUTAGE_GROUP, 12))+ after: Final = assert_all_served(burst(owned.gateway, OUTAGE_GROUP, 12, priority)) assert_once(received_markers(wire), (*before, *during, *after)) eventually(lambda: queue_entries(cache, OUTAGE_GROUP), lambda entries: entries in (None, []), seconds=10) @@ -813,13 +878,14 @@ def established_upstream_connections(pid: int, wire: Wire) -> int: ) +@pytest.mark.parametrize("priority", (1, None), ids=("explicit", "default")) @pytest.mark.timeout(OWNED_CELL_TIMEOUT) def test_sibling_worker_keeps_serving_prioritized_requests_after_a_worker_is_killed(- gateway: Gateway, tmp_path: Path+ gateway: Gateway, tmp_path: Path, priority: int | None ) -> None: upstream: Final = upstream_for((KILL_GROUP,), hold=(KILL_GROUP,)) with owned_redis(tmp_path) as cache, wire_server(answering_model_discovery(upstream.respond)) as wire:- config: Final = owned_config(tmp_path, wire, (KILL_GROUP,), redis_settings(cache))+ config: Final = owned_config(tmp_path, wire, (KILL_GROUP,), {**redis_settings(cache), **default_for(priority)}) overrides: Final = {"REDIS_HOST": cache.host, "REDIS_PORT": str(cache.port)} with owned_proxy_process(gateway, tmp_path, overrides, config=config, workers=2) as owned: workers: Final = eventually(@@ -828,7 +894,7 @@ def test_sibling_worker_keeps_serving_prioritized_requests_after_a_worker_is_kil seconds=graceful_stop_seconds(), ) with ThreadPoolExecutor(max_workers=1) as pool:- pending: Final = pool.submit(burst, owned.gateway, KILL_GROUP, 20)+ pending: Final = pool.submit(burst, owned.gateway, KILL_GROUP, 20, priority) eventually(upstream.held.qsize, lambda size: size == 20, seconds=60) held_by: Final = {pid: established_upstream_connections(pid, wire) for pid in workers} assert sum(held_by.values()) == 20, held_by@@ -844,10 +910,502 @@ def test_sibling_worker_keeps_serving_prioritized_requests_after_a_worker_is_kil assert len(failed) == held_by[victim_pid], (held_by, len(failed)) assert_all_served(answered) follow_up: Final = new_marker()- served_eventually(owned.gateway, KILL_GROUP, follow_up)+ served_eventually(owned.gateway, KILL_GROUP, follow_up, priority) eventually( lambda: len(STARTED_WORKER.findall(owned.log.read_text())), lambda count: count == 3, seconds=graceful_stop_seconds(), ) assert_once(received_markers(wire), (*(marker for _, marker, _ in outcomes), follow_up))+++@dataclass(frozen=True, slots=True)+class Defaulted:+ pair: Pair+ team_key: str++ def as_team(self) -> Gateway:+ return Gateway(self.pair.first.client, self.team_key, self.pair.first.upstream_url)+++def prompt_directory(directory: Path) -> Path:+ prompts: Final = directory / "prompts"+ prompts.mkdir()+ (prompts / f"{PROMPT_ID}.prompt").write_text(+ "---\n"+ f"model: openai/{upstream_name(PROMPT_GROUP)}\n"+ "input:\n"+ " schema:\n"+ " user_message: string\n"+ "---\n"+ "\n"+ f"{PROMPT_PREFIX} {{{{user_message}}}}\n"+ )+ return prompts+++@pytest.fixture(scope="module")+def defaulted(tmp_path_factory: pytest.TempPathFactory) -> Iterator[Defaulted]:+ directory: Final = tmp_path_factory.mktemp("scheduler-default")+ upstream: Final = upstream_for((*DEFAULT_GROUPS, PROMPT_GROUP, DROP_GROUP, TEAM_GROUP))+ with ExitStack() as stack:+ gateway: Final = stack.enter_context(gateway_from_environment())+ scenario: Final = stack.enter_context(gateway.scenario())+ team: Final = scenario.team(models=[])+ team_key: Final = scenario.key(team_id=team)+ cache: Final = stack.enter_context(owned_redis(directory))+ wire: Final = stack.enter_context(wire_server(answering_model_discovery(upstream.respond)))+ extra_models: Final = (+ deployment_entry(+ PROMPT_GROUP, wire, model=f"dotprompt/openai/{upstream_name(PROMPT_GROUP)}", prompt_id=PROMPT_ID+ ),+ deployment_entry(DROP_GROUP, wire, drop_params=True),+ deployment_entry(+ TEAM_GROUP,+ wire,+ model_name=f"{TEAM_GROUP}-internal",+ model_info={"team_id": team, "team_public_model_name": TEAM_GROUP},+ drop_params=True,+ ),+ )+ settings: Final[dict[str, JsonValue]] = {+ **cooldown_settings(cache),+ "default_priority": DEFAULT_PRIORITY,+ "fallbacks": [{DEFAULT_GROUPS[7]: [DEFAULT_GROUPS[8]]}],+ }+ config: Final = owned_config(+ directory,+ wire,+ DEFAULT_GROUPS,+ settings,+ cancel_on_disconnect=True,+ extra_models=extra_models,+ litellm_settings={"global_prompt_directory": str(prompt_directory(directory))},+ )+ overrides: Final = {"REDIS_HOST": cache.host, "REDIS_PORT": str(cache.port)}+ shared: Final = OwnedState.SHARED_WITH_WORKER+ first: Final = stack.enter_context(+ owned_proxy_process(gateway, directory, overrides, config=config, workers=2, state=shared)+ )+ second: Final = stack.enter_context(+ owned_proxy_process(gateway, directory, overrides, config=config, workers=2, state=shared)+ )+ yield Defaulted(Pair(first.gateway, second.gateway, cache, wire, upstream), team_key)+++def assert_prioritized(answer: Answer) -> None:+ assert answer.headers.get(PRIORITIZED) == "True", answer.headers+++def assert_unprioritized(answer: Answer) -> None:+ assert answer.headers.get(PRIORITIZED) is None, answer.headers+++def assert_refused_as_non_integer(answer: Answer) -> None:+ assert answer.status == 400, (answer.status, answer.text)+ error: Final = JSON_OBJECT.validate_json(answer.text)["error"]+ assert isinstance(error, dict), answer.text+ assert error["param"] == "priority", answer.text+ assert "priority must be an integer" in str(error["message"]), answer.text+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_raw_chat_without_priority_is_scheduled_at_the_default_and_billed(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[0]+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.first, "/v1/chat/completions", chat_body(group, marker))+ identity: Final = assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+ spend_row_lands(identity)+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_raw_chat_stream_without_priority_is_served_once(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[0]+ marker: Final = new_marker()+ answer: Final = stream_text(defaulted.pair.second, "/v1/chat/completions", chat_body(group, marker, stream=True))+ assert_served(answer, marker)+ assert_prioritized(answer)+ assert_once(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_sdk_chat_without_priority_is_scheduled_at_the_default_and_billed(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[0]+ marker: Final = new_marker()+ client: Final = sdk(defaulted.pair.first)+ raw: Final = sdk_settled(+ lambda: client.chat.completions.with_raw_response.create(+ model=group, messages=[{"role": "user", "content": marker}]+ )+ )+ assert served_identity(raw.status_code, raw.text, marker) == f"chatcmpl-{marker}"+ assert raw.headers.get(PRIORITIZED) == "True", dict(raw.headers)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+ spend_row_lands(f"chatcmpl-{marker}")+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_sdk_chat_stream_without_priority_is_served_once(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[0]+ marker: Final = new_marker()+ client: Final = sdk(defaulted.pair.second)+ chunks: Final = sdk_settled(+ lambda: tuple(+ client.chat.completions.create(model=group, messages=[{"role": "user", "content": marker}], stream=True)+ )+ )+ assert {chunk.id for chunk in chunks} == {f"chatcmpl-{marker}"}, chunks+ content: Final = "".join(chunk.choices[0].delta.content or "" for chunk in chunks if chunk.choices)+ assert content == f"served {marker}", chunks+ assert_once(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_async_sdk_chat_without_priority_is_scheduled_at_the_default(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[0]+ marker: Final = new_marker()+ gateway: Final = defaulted.pair.first++ async def send() -> tuple[int, str, str | None]:+ async with openai.AsyncOpenAI(+ base_url=str(gateway.client.base_url), api_key=gateway.key, max_retries=0, timeout=30+ ) as client:+ raw: Final = await client.chat.completions.with_raw_response.create(+ model=group, messages=[{"role": "user", "content": marker}]+ )+ return raw.status_code, raw.text, raw.headers.get(PRIORITIZED)++ status, text, prioritized = sdk_settled(lambda: asyncio.run(send()))+ assert served_identity(status, text, marker) == f"chatcmpl-{marker}"+ assert prioritized == "True"+ assert_once(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.parametrize("priority", (1, 0, -1, 2**40), ids=("one", "zero", "negative", "huge"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_explicit_integer_priority_under_a_default_is_scheduled_once(defaulted: Defaulted, priority: int) -> None:+ group: Final = DEFAULT_GROUPS[1]+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.first, "/v1/chat/completions", chat_body(group, marker, priority=priority))+ assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_null_priority_under_a_default_takes_the_default(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[1]+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.second, "/v1/chat/completions", chat_body(group, marker, priority=None))+ assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++@pytest.mark.parametrize(+ "priority",+ ("high", [1], 1.5, {"level": 3}, "", "p" * 5120),+ ids=("word", "list", "float", "object", "empty", "5kb"),+)+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_non_integer_priority_under_a_default_is_refused_before_the_upstream(+ defaulted: Defaulted, priority: JsonValue+) -> None:+ group: Final = DEFAULT_GROUPS[1]+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.first, "/v1/chat/completions", chat_body(group, marker, priority=priority))+ assert_refused_as_non_integer(answer)+ assert_never(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_request_drop_params_replaces_a_non_integer_priority_with_the_default(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[1]+ marker: Final = new_marker()+ body: Final = chat_body(group, marker, priority="high", drop_params=True)+ answer: Final = post(defaulted.pair.second, "/v1/chat/completions", body)+ assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++@pytest.mark.parametrize("model", (DROP_GROUP, f"{DROP_GROUP}-deployment"), ids=("group", "deployment-id"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_deployment_drop_params_replaces_a_non_integer_priority_with_the_default(+ defaulted: Defaulted, model: str+) -> None:+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.first, "/v1/chat/completions", chat_body(model, marker, priority="high"))+ assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_team_deployment_drop_params_replaces_a_non_integer_priority_with_the_default(defaulted: Defaulted) -> None:+ marker: Final = new_marker()+ answer: Final = post(defaulted.as_team(), "/v1/chat/completions", chat_body(TEAM_GROUP, marker, priority="high"))+ assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++def assert_waits_at(+ gateway: Gateway,+ cache: OwnedRedis,+ upstream: Upstream,+ group: str,+ priority: int,+ body: Mapping[str, JsonValue],+) -> None:+ cooled_outside_the_scheduler(gateway, upstream, group)+ write_queue(cache, group, [[priority - 1, "ghost"]])+ with ThreadPoolExecutor(max_workers=1) as pool:+ waiter: Final = pool.submit(post, gateway, "/v1/chat/completions", body)+ entries: Final = waiting_entries(cache, group, priority)+ write_queue(cache, group, [entry for entry in entries if entry_priority(entry) == priority])+ assert_refused_before_the_router_timeout(waiter.result(timeout=30))+ assert_drained(cache, group)+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_requests_without_priority_wait_at_the_default_priority(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[2]+ pair: Final = defaulted.pair+ assert_waits_at(pair.second, pair.cache, pair.upstream, group, DEFAULT_PRIORITY, chat_body(group, new_marker()))+++@pytest.mark.timeout(OWNED_CELL_TIMEOUT)+def test_zero_default_priority_queues_requests_without_one_and_dropped_ones(gateway: Gateway, tmp_path: Path) -> None:+ upstream: Final = upstream_for(ZERO_GROUPS)+ with owned_redis(tmp_path) as cache, wire_server(answering_model_discovery(upstream.respond)) as wire:+ settings: Final[dict[str, JsonValue]] = {+ **cooldown_settings(cache),+ "default_priority": 0,+ "default_litellm_params": {"drop_params": True},+ }+ config: Final = owned_config(tmp_path, wire, ZERO_GROUPS, settings)+ overrides: Final = {"REDIS_HOST": cache.host, "REDIS_PORT": str(cache.port)}+ with owned_proxy_process(gateway, tmp_path, overrides, config=config, workers=2) as owned:+ plain, dropped = ZERO_GROUPS+ assert_waits_at(owned.gateway, cache, upstream, plain, 0, chat_body(plain, new_marker()))+ assert_waits_at(+ owned.gateway, cache, upstream, dropped, 0, chat_body(dropped, new_marker(), priority="high")+ )+++@pytest.mark.timeout(OWNED_CELL_TIMEOUT)+def test_digit_string_default_priority_queues_requests_without_one_and_dropped_ones(+ gateway: Gateway, tmp_path: Path+) -> None:+ upstream: Final = upstream_for(DIGIT_GROUPS)+ with owned_redis(tmp_path) as cache, wire_server(answering_model_discovery(upstream.respond)) as wire:+ settings: Final[dict[str, JsonValue]] = {**cooldown_settings(cache), "default_priority": "2"}+ config: Final = owned_config(tmp_path, wire, DIGIT_GROUPS, settings, litellm_settings={"drop_params": True})+ overrides: Final = {"REDIS_HOST": cache.host, "REDIS_PORT": str(cache.port)}+ with owned_proxy_process(gateway, tmp_path, overrides, config=config, workers=2) as owned:+ plain, dropped = DIGIT_GROUPS+ assert_waits_at(owned.gateway, cache, upstream, plain, 2, chat_body(plain, new_marker()))+ assert_waits_at(+ owned.gateway, cache, upstream, dropped, 2, chat_body(dropped, new_marker(), priority="high")+ )+++@pytest.mark.parametrize("configured", ("high", True), ids=("word", "bool"))+@pytest.mark.timeout(OWNED_CELL_TIMEOUT)+def test_an_invalid_default_priority_is_ignored_with_a_warning(+ gateway: Gateway, tmp_path: Path, configured: JsonValue+) -> None:+ upstream: Final = upstream_for((IGNORED_GROUP,))+ with wire_server(answering_model_discovery(upstream.respond)) as wire:+ config: Final = owned_config(+ tmp_path, wire, (IGNORED_GROUP,), {"num_retries": 0, "default_priority": configured}+ )+ with owned_proxy_process(gateway, tmp_path, {}, config=config, workers=2) as owned:+ marker: Final = new_marker()+ answer: Final = post(owned.gateway, "/v1/chat/completions", chat_body(IGNORED_GROUP, marker))+ assert_served(answer, marker)+ assert_unprioritized(answer)+ assert IGNORED_WARNING in owned.log.read_text()+ assert_once(received_markers(wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_requests_without_priority_fall_back_to_the_fallback_group(defaulted: Defaulted) -> None:+ primary, fallback = DEFAULT_GROUPS[7], DEFAULT_GROUPS[8]+ defaulted.pair.upstream.refusing[upstream_name(primary)].set()+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.first, "/v1/chat/completions", chat_body(primary, marker))+ assert_served(answer, marker)+ assert_prioritized(answer)+ attempts: Final = tuple(+ str(JSON_OBJECT.validate_json(request.body)["model"])+ for request in defaulted.pair.wire.drain()+ if marker.encode() in request.body+ )+ assert attempts == (upstream_name(primary), upstream_name(fallback)), attempts+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_cooled_group_refuses_requests_without_priority_at_once(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[3]+ cooled(defaulted.pair.second, defaulted.pair.upstream, group)+ assert_refused_at_once(post(defaulted.pair.second, "/v1/chat/completions", chat_body(group, new_marker())))+ assert_drained(defaulted.pair.cache, group)+++@pytest.mark.parametrize("priority", (None, 1), ids=("default", "explicit"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_prompt_management_model_is_scheduled(defaulted: Defaulted, priority: int | None) -> None:+ marker: Final = new_marker()+ body: Final = chat_body(PROMPT_GROUP, marker, prompt_variables={"user_message": marker}, **priority_field(priority))+ answer: Final = post(defaulted.pair.first, "/v1/chat/completions", body)+ assert_served(answer, marker)+ assert_prioritized(answer)+ upstream_body: Final = upstream_body_for(defaulted.pair.wire, marker)+ assert f"{PROMPT_PREFIX} {marker}" in json.dumps(upstream_body), upstream_body+ assert "priority" not in upstream_body+++@pytest.mark.parametrize("endpoint", ("completions", "completions-stream"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_text_completions_without_priority_are_scheduled_at_the_default(+ defaulted: Defaulted, endpoint: Endpoint+) -> None:+ group: Final = DEFAULT_GROUPS[4]+ marker: Final = new_marker()+ answer: Final = call(defaulted.pair.second, endpoint, group, marker, priority=None)+ identity: Final = assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+ spend_row_lands(identity)+++@pytest.mark.parametrize("priority", ("high", [1]), ids=("word", "list"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_text_completion_with_a_non_integer_priority_is_refused_before_the_upstream(+ defaulted: Defaulted, priority: JsonValue+) -> None:+ group: Final = DEFAULT_GROUPS[4]+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.first, "/v1/completions", completion_body(group, marker, priority=priority))+ assert_refused_as_non_integer(answer)+ assert_never(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_text_completion_drop_params_replaces_a_non_integer_priority_with_the_default(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[4]+ marker: Final = new_marker()+ body: Final = completion_body(group, marker, priority="high", drop_params=True)+ answer: Final = post(defaulted.pair.second, "/v1/completions", body)+ assert_served(answer, marker)+ assert_prioritized(answer)+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++@pytest.mark.parametrize("endpoint", ("queue", "queue-stream"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_queue_route_without_priority_takes_the_default(defaulted: Defaulted, endpoint: Endpoint) -> None:+ group: Final = DEFAULT_GROUPS[5]+ marker: Final = new_marker()+ answer: Final = call(defaulted.pair.first, endpoint, group, marker, priority=None)+ assert_served(answer, marker)+ if endpoint == "queue":+ assert answer.headers.get("x-litellm-priority") == str(DEFAULT_PRIORITY), answer.headers+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++@pytest.mark.parametrize("priority", (1, "1", True, 1.0), ids=("integer", "digit-string", "bool", "whole-float"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_queue_route_with_an_explicit_priority_under_a_default_reports_it(+ defaulted: Defaulted, priority: JsonValue+) -> None:+ group: Final = DEFAULT_GROUPS[5]+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.second, "/queue/chat/completions", chat_body(group, marker, priority=priority))+ assert_served(answer, marker)+ assert answer.headers.get("x-litellm-priority") == "1", answer.headers+ assert "priority" not in upstream_body_for(defaulted.pair.wire, marker)+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_queue_route_refuses_a_word_priority(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[5]+ marker: Final = new_marker()+ answer: Final = post(defaulted.pair.first, "/queue/chat/completions", chat_body(group, marker, priority="high"))+ assert_refused_as_non_integer(answer)+ assert_never(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.parametrize("path", ("/v1/messages", "/v1/responses"), ids=("messages", "responses"))+@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_unscheduled_routes_without_priority_are_served_once_under_a_default(defaulted: Defaulted, path: str) -> None:+ group: Final = DEFAULT_GROUPS[6]+ marker: Final = new_marker()+ body: Final[dict[str, JsonValue]] = (+ {"model": group, "max_tokens": 32, "messages": [{"role": "user", "content": marker}]}+ if path == "/v1/messages"+ else {"model": group, "input": marker}+ )+ answer: Final = post(defaulted.pair.second, path, body)+ assert answer.status == 200, (answer.status, answer.text)+ assert markers_in(answer.text) == {marker}, answer.text+ assert_once(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_unauthenticated_request_without_priority_never_reaches_the_upstream(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[6]+ marker: Final = new_marker()+ response: Final = defaulted.pair.first.client.post("/v1/chat/completions", json=chat_body(group, marker))+ assert response.status_code == 401, response.text+ assert_never(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_cached_replay_under_a_default_reaches_the_upstream_once(defaulted: Defaulted) -> None:+ group: Final = DEFAULT_GROUPS[6]+ marker: Final = new_marker()+ body: Final = chat_body(group, marker)+ first: Final = post(defaulted.pair.first, "/v1/chat/completions", body)+ assert_served(first, marker)+ assert_prioritized(first)+ replay: Final = post(defaulted.pair.second, "/v1/chat/completions", body)+ assert_served(replay, marker)+ assert_prioritized(replay)+ assert_once(received_markers(defaulted.pair.wire), (marker,))+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_concurrent_burst_without_priority_across_instances_and_endpoints_is_served(defaulted: Defaulted) -> None:+ serve_burst_without_priority(defaulted.pair, DEFAULT_GROUPS[0])+++@pytest.mark.timeout(PAIR_CELL_TIMEOUT)+def test_concurrent_burst_without_priority_across_instances_leaves_no_queue_entry(defaulted: Defaulted) -> None:+ pytest.skip("BUG: concurrent queue writes across workers can leave a served request's entry in the queue")+ group: Final = DEFAULT_GROUPS[0]+ serve_burst_without_priority(defaulted.pair, group)+ assert_drained(defaulted.pair.cache, group)+++def serve_burst_without_priority(pair: Pair, group: str) -> None:+ primer: Final = new_marker()+ assert_served(post(pair.first, "/v1/chat/completions", chat_body(group, primer)), primer)+ instances: Final[tuple[Instance, ...]] = ("first", "second")+ plan: Final = tuple((instance, endpoint, new_marker()) for instance, endpoint in product(instances, ENDPOINTS * 2))++ def one(item: tuple[Instance, Endpoint, str]) -> Answer:+ instance, endpoint, marker = item+ return call(pair.at(instance), endpoint, group, marker, priority=None)++ with ThreadPoolExecutor(max_workers=len(plan)) as pool:+ answers: Final = tuple(pool.map(one, plan))+ for (_, _, marker), answer in zip(plan, answers, strict=True):+ assert_served(answer, marker)+ closing: Final = new_marker()+ assert_served(post(pair.second, "/v1/chat/completions", chat_body(group, closing)), closing)+ assert_once(received_markers(pair.wire), (primer, *(marker for _, _, marker in plan), closing))tests/unit/proxy/proxy_server/test_routes_queue.pyindex 27cc6300711..07a61c8587b 100644@@ -6,10 +6,12 @@ Pins (PR2): from __future__ import annotations -from unittest.mock import AsyncMock, MagicMock+from typing import Final+from unittest.mock import AsyncMock, MagicMock, patch import pytest +from litellm import Router from litellm.proxy import proxy_server from .conftest import normalize # type: ignore[import-not-found]@@ -34,6 +36,7 @@ HAPPY_RESPONSE = { @pytest.fixture def patched_queue(monkeypatch): router = MagicMock()+ router.split_request_priority = MagicMock(return_value=(0, {})) router.schedule_acompletion = AsyncMock(return_value=dict(HAPPY_RESPONSE)) monkeypatch.setattr(proxy_server, "llm_router", router) monkeypatch.setattr(@@ -89,3 +92,103 @@ def test_queue_chat_completions_no_router_error(client, auth_as, queue_no_router response = client.post("/queue/chat/completions", json=payload) assert response.status_code == 500 assert len(response.content) > 0+++def _queue_router(+ monkeypatch: pytest.MonkeyPatch, default_priority: int | None, mock_response: str = "queued"+) -> Router:+ router: Final = Router(+ model_list=[+ {+ "model_name": "gpt-4",+ "litellm_params": {"model": "openai/gpt-4", "api_key": "sk-queue-test", "mock_response": mock_response},+ }+ ],+ default_priority=default_priority,+ )+ monkeypatch.setattr(proxy_server, "llm_router", router)+ monkeypatch.setattr(proxy_server, "proxy_logging_obj", MagicMock(post_call_failure_hook=AsyncMock()))+ return router+++@pytest.mark.parametrize(+ ("default_priority", "priority_fields", "expected_priority"),+ [+ (3, {"priority": "high", "drop_params": True}, 3),+ (3, {"priority": "1"}, 1),+ (None, {"priority": 1.0}, 1),+ (3, {"priority": 1}, 1),+ (3, {}, 3),+ ],+ ids=("dropped-word-takes-default", "digit-string", "integral-float", "explicit", "absent-takes-default"),+)+def test_queue_chat_completions_reports_the_priority_the_router_queued_at(+ client,+ auth_as,+ monkeypatch: pytest.MonkeyPatch,+ default_priority: int | None,+ priority_fields: dict[str, object],+ expected_priority: int,+) -> None:+ router: Final = _queue_router(monkeypatch, default_priority)+ payload: Final = {"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}], **priority_fields}+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ with auth_as():+ response: Final = client.post("/queue/chat/completions", json=payload)+ assert response.status_code == 200, response.text+ assert [call.kwargs["request"].priority for call in add_request.await_args_list] == [expected_priority]+ assert response.headers.get("x-litellm-priority") == str(expected_priority)+++@pytest.mark.parametrize(+ ("default_priority", "priority_fields", "message"),+ [+ (None, {}, "priority is required"),+ (3, {"priority": "high"}, "priority must be an integer"),+ (None, {"priority": "high", "drop_params": True}, "priority must be an integer, got str 'high'"),+ ],+ ids=("no-priority-and-no-default", "malformed", "dropped-word-and-no-default"),+)+def test_queue_chat_completions_rejects_a_request_it_cannot_queue_as_an_invalid_request(+ client,+ auth_as,+ monkeypatch: pytest.MonkeyPatch,+ default_priority: int | None,+ priority_fields: dict[str, object],+ message: str,+) -> None:+ router: Final = _queue_router(monkeypatch, default_priority)+ payload: Final = {"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}], **priority_fields}+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ with auth_as():+ response: Final = client.post("/queue/chat/completions", json=payload)+ assert response.status_code == 400, response.text+ error: Final = response.json()["error"]+ assert (error["type"], error["param"]) == ("invalid_request_error", "priority")+ assert message in error["message"]+ assert add_request.await_count == 0+++def test_streamed_queue_responses_report_the_priority_the_router_queued_at(+ client, auth_as, monkeypatch: pytest.MonkeyPatch+) -> None:+ _queue_router(monkeypatch, default_priority=3)+ payload: Final = {"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}], "priority": "1", "stream": True}+ with auth_as():+ response: Final = client.post("/queue/chat/completions", json=payload)+ assert response.status_code == 200, response.text+ assert response.headers.get("x-litellm-priority") == "1"+++def test_a_provider_bad_request_on_the_queue_route_keeps_its_own_message(+ client, auth_as, monkeypatch: pytest.MonkeyPatch+) -> None:+ _queue_router(monkeypatch, default_priority=3, mock_response="litellm.ContextWindowExceededError")+ payload: Final = {"model": "gpt-4", "messages": [{"role": "user", "content": "hi"}]}+ with auth_as():+ response: Final = client.post("/queue/chat/completions", json=payload)+ assert response.status_code == 400, response.text+ error: Final = response.json()["error"]+ assert error["type"] == "invalid_request_error"+ assert "context window" in error["message"].lower()+ assert "Authentication Error" not in error["message"]tests/unit/proxy/proxy_server/test_streaming_helpers.pyindex 227a099889d..9a3516d20d6 100644@@ -2322,6 +2322,7 @@ async def _queue_streaming(monkeypatch, interval, delay=0.3, fails_with=None): router = MagicMock() router.get_model_list.return_value = []+ router.split_request_priority.return_value = (0, {}) async def _schedule_after_the_scheduler_queue_drains(**kwargs): await asyncio.sleep(delay)tests/unit/router_utils/test_request_priority.pynew file mode 100644index 00000000000..71ab441d199@@ -0,0 +1,112 @@+from typing import Final++import pytest++import litellm+from litellm.router_utils.request_priority import (+ InvalidPriority,+ parse_default_priority,+ request_drops_params,+ resolve_request_priority,+ unqueueable_priority_message,+)+++@pytest.mark.parametrize(+ ("requested", "default_priority", "expected"),+ [+ (None, None, None),+ (None, 3, 3),+ (2, 3, 2),+ (0, 3, 0),+ (0, None, 0),+ (255, None, 255),+ ("1", 3, 1),+ (" 2 ", None, 2),+ ("-1", 3, -1),+ (1.0, 3, 1),+ (True, None, 1),+ ],+)+def test_an_absent_or_null_priority_takes_the_default_and_an_integer_like_value_wins(+ requested: object, default_priority: int | None, expected: int | None+) -> None:+ assert resolve_request_priority(requested, default_priority, drops_params=lambda: False) == expected+ assert resolve_request_priority(requested, default_priority, drops_params=lambda: True) == expected+++@pytest.mark.parametrize("requested", ["high", "", [1], 1.5, {"level": 1}])+def test_a_non_integer_priority_is_rejected_unless_params_are_dropped(requested: object) -> None:+ rejected: Final = resolve_request_priority(requested, 3, drops_params=lambda: False)+ assert isinstance(rejected, InvalidPriority)+ assert rejected.value == requested+ assert rejected.message.startswith("priority must be an integer, got ")+ assert resolve_request_priority(requested, 3, drops_params=lambda: True) == 3+ assert resolve_request_priority(requested, None, drops_params=lambda: True) is None+++def test_the_drop_params_decision_is_consulted_only_for_a_non_integer_priority() -> None:+ def _never() -> bool:+ raise AssertionError("drop_params was consulted")++ assert resolve_request_priority(None, 3, drops_params=_never) == 3+ assert resolve_request_priority(2, 3, drops_params=_never) == 2+ assert resolve_request_priority(0, None, drops_params=_never) == 0+ assert resolve_request_priority("1", 3, drops_params=_never) == 1+++def test_the_global_drop_params_wins_then_the_request_then_the_router_defaults_then_the_deployments(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setattr(litellm, "drop_params", False)+ assert request_drops_params({}, {}, ()) is False+ assert request_drops_params({"drop_params": True}, {}, ()) is True+ assert request_drops_params({"drop_params": "true"}, {}, ()) is True+ assert request_drops_params({}, {"drop_params": True}, ()) is True+ assert request_drops_params({"drop_params": False}, {"drop_params": True}, ()) is False+ assert request_drops_params({}, {}, ({"drop_params": True},)) is True+ assert request_drops_params({}, {}, ({"drop_params": True}, {"model": "openai/gpt-5.4-nano"})) is True+ assert request_drops_params({}, {}, ({"drop_params": False}, {"model": "openai/gpt-5.4-nano"})) is False+ assert request_drops_params({}, {"drop_params": False}, ({"drop_params": True},)) is False+ monkeypatch.setattr(litellm, "drop_params", True)+ assert request_drops_params({}, {}, ()) is True+ assert request_drops_params({}, {}, ({"model": "openai/gpt-5.4-nano"},)) is True+ assert request_drops_params({"drop_params": False}, {}, ()) is True+ assert request_drops_params({}, {"drop_params": False}, ()) is True+ assert request_drops_params({}, {}, ({"drop_params": False},)) is True+ assert request_drops_params({"drop_params": "not-a-flag"}, {}, ()) is True+++def test_a_priority_named_in_additional_drop_params_at_any_level_is_dropped(monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setattr(litellm, "drop_params", False)+ by_name: Final = {"additional_drop_params": ["priority"]}+ assert request_drops_params(by_name, {}, ()) is True+ assert request_drops_params({**by_name, "drop_params": False}, {}, ()) is True+ assert request_drops_params({}, by_name, ()) is True+ assert request_drops_params({}, {"drop_params": False}, (by_name,)) is True+ assert request_drops_params({"additional_drop_params": ["seed"]}, {}, ()) is False+ assert request_drops_params({}, {}, (params for params in ({"drop_params": True},))) is True+++def test_a_request_it_cannot_queue_is_told_why() -> None:+ assert "priority is required" in unqueueable_priority_message(None)+ assert unqueueable_priority_message("high").startswith("priority must be an integer, got str 'high'")+++@pytest.mark.parametrize(+ ("configured", "expected"),+ [(None, None), (3, 3), (0, 0), ("3", 3), (" 12 ", 12), ("0", 0), ("-1", -1), (2.0, 2)],+ ids=["unset", "int", "zero", "digit-string", "padded-digit-string", "zero-string", "negative-string", "whole-float"],+)+def test_the_default_priority_is_taken_as_an_integer_or_an_integer_like_value(+ configured: object, expected: int | None+) -> None:+ assert parse_default_priority(configured) == expected+++@pytest.mark.parametrize("configured", ["high", "", "1.5", 1.5, [3], {"level": 3}, True, False])+def test_any_other_default_priority_is_invalid(configured: object) -> None:+ parsed: Final = parse_default_priority(configured)+ assert isinstance(parsed, InvalidPriority)+ assert parsed.value == configured+ assert parsed.message.startswith("priority must be an integer, got ")tests/unit/test_router/test_router.pyindex 90674dcb3f9..cc25d73b83a 100644@@ -26055,6 +26055,359 @@ class TestRouterIndexManagement: ) +_PRIORITY_CHAT_COMPLETION: Final = {+ "id": "chatcmpl-priority",+ "object": "chat.completion",+ "created": 1,+ "model": "gpt-5.4-nano",+ "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": "pong"}}],+ "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},+}+_PRIORITY_MESSAGES: Final = [{"role": "user", "content": "Reply with the single word pong"}]+++def _priority_router(default_priority: int | None) -> Router:+ return Router(+ model_list=[+ {+ "model_name": "qa-chat",+ "litellm_params": {"model": "openai/gpt-5.4-nano", "api_key": "sk-priority-wire"},+ }+ ],+ default_priority=default_priority,+ )+++def _priority_provider(respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch) -> respx.Route:+ monkeypatch.setattr(litellm, "disable_aiohttp_transport", True)+ return respx_mock.post("https://api.openai.com/v1/chat/completions").mock(+ return_value=httpx.Response(200, json=_PRIORITY_CHAT_COMPLETION)+ )+++@pytest.mark.parametrize(+ ("default_priority", "request_kwargs", "queued_priority"),+ [+ (3, {}, 3),+ (3, {"priority": None}, 3),+ (3, {"priority": 2}, 2),+ (None, {"priority": 2}, 2),+ (None, {"priority": 0}, 0),+ (None, {"priority": "1"}, 1),+ (None, {"priority": True}, 1),+ (3, {"priority": 1.0}, 1),+ (3, {"priority": "high", "drop_params": True}, 3),+ ],+)+async def test_acompletion_queues_the_resolved_priority_exactly_once(+ default_priority: int | None, request_kwargs: dict[str, object], queued_priority: int+) -> None:+ router: Final = _priority_router(default_priority)+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.acompletion(+ model="qa-chat", messages=_PRIORITY_MESSAGES, mock_response="pong", **request_kwargs+ )+ assert add_request.await_count == 1+ assert add_request.await_args.kwargs["request"].priority == queued_priority+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+++async def test_acompletion_without_any_priority_never_enters_the_scheduler() -> None:+ router: Final = _priority_router(None)+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.acompletion(model="qa-chat", messages=_PRIORITY_MESSAGES, mock_response="pong")+ assert add_request.await_count == 0+ assert "x-litellm-request-prioritization-used" not in response._hidden_params.get("additional_headers", {})+++@pytest.mark.parametrize(+ ("default_priority", "request_kwargs"), [(3, {}), (None, {"priority": 3})], ids=["default", "explicit"]+)+async def test_a_request_pinned_to_a_deployment_id_is_served_while_an_earlier_one_waits_in_the_queue(+ default_priority: int | None, request_kwargs: dict[str, object]+) -> None:+ router: Final = Router(+ model_list=[+ {+ "model_name": "qa-chat",+ "litellm_params": {"model": "openai/gpt-5.4-nano", "api_key": "sk-priority-wire"},+ "model_info": {"id": "qa-chat-deployment"},+ }+ ],+ default_priority=default_priority,+ )+ await router.scheduler.add_request(FlowItem(priority=0, request_id="ahead", model_name="qa-chat-deployment"))+ response: Final = await asyncio.wait_for(+ router.acompletion(+ model="qa-chat-deployment", messages=_PRIORITY_MESSAGES, mock_response="pong", **request_kwargs+ ),+ timeout=5,+ )+ assert response.choices[0].message.content == "pong"+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+++@pytest.mark.parametrize("priority", ["high", [1], 1.5])+async def test_acompletion_rejects_a_non_integer_priority_before_the_provider_sees_it(+ priority: object, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch+) -> None:+ provider: Final = _priority_provider(respx_mock, monkeypatch)+ router: Final = _priority_router(3)+ with pytest.raises(litellm.BadRequestError, match="priority must be an integer") as raised:+ await router.acompletion(model="qa-chat", messages=_PRIORITY_MESSAGES, priority=priority)+ assert raised.value.param == "priority"+ assert not provider.called+++async def test_acompletion_drops_a_non_integer_priority_from_the_provider_call_under_drop_params(+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch+) -> None:+ provider: Final = _priority_provider(respx_mock, monkeypatch)+ router: Final = _priority_router(None)+ await router.acompletion(model="qa-chat", messages=_PRIORITY_MESSAGES, priority="high", drop_params=True)+ assert "priority" not in json.loads(provider.calls.last.request.content)+++@pytest.mark.parametrize("priority", [0, "1"])+async def test_acompletion_with_an_integer_priority_sends_no_priority_to_the_provider(+ priority: object, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch+) -> None:+ provider: Final = _priority_provider(respx_mock, monkeypatch)+ router: Final = _priority_router(None)+ await router.acompletion(model="qa-chat", messages=_PRIORITY_MESSAGES, priority=priority)+ assert "priority" not in json.loads(provider.calls.last.request.content)+++async def test_schedule_acompletion_under_a_default_priority_queues_the_explicit_priority_once() -> None:+ router: Final = _priority_router(3)+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.schedule_acompletion(+ model="qa-chat", messages=_PRIORITY_MESSAGES, priority=1, mock_response="pong"+ )+ assert add_request.await_count == 1+ assert add_request.await_args.kwargs["request"].priority == 1+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+++@pytest.mark.parametrize(+ ("deployment_params", "router_defaults"),+ [+ ({}, {"drop_params": True}),+ ({"drop_params": True}, {}),+ ({"drop_params": "true"}, {}),+ ({"additional_drop_params": ["priority"]}, {}),+ ({}, {"additional_drop_params": ["priority"]}),+ ],+)+async def test_acompletion_drops_a_non_integer_priority_under_the_router_or_deployment_drop_params(+ deployment_params: dict[str, object], router_defaults: dict[str, object]+) -> None:+ router: Final = Router(+ model_list=[+ {+ "model_name": "qa-chat",+ "litellm_params": {"model": "openai/gpt-5.4-nano", "api_key": "sk-priority-wire", **deployment_params},+ }+ ],+ default_priority=3,+ default_litellm_params=router_defaults,+ )+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.acompletion(+ model="qa-chat", messages=_PRIORITY_MESSAGES, mock_response="pong", priority="high"+ )+ assert add_request.await_args.kwargs["request"].priority == 3+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+++@pytest.mark.parametrize(+ ("deployment", "requested_model", "routing_fields"),+ [+ (+ {+ "model_name": "model_name_team-a_team-a-chat",+ "litellm_params": {"model": "openai/gpt-5.4-nano", "api_key": "sk-priority-wire", "drop_params": True},+ "model_info": {"id": "team-a-chat", "team_id": "team-a", "team_public_model_name": "team-chat"},+ },+ "team-chat",+ {"metadata": {"user_api_key_team_id": "team-a"}},+ ),+ (+ {+ "model_name": "qa-chat",+ "litellm_params": {"model": "openai/gpt-5.4-nano", "api_key": "sk-priority-wire", "drop_params": True},+ "model_info": {"id": "qa-chat-deployment"},+ },+ "qa-chat-deployment",+ {},+ ),+ (+ {+ "model_name": "qa-chat",+ "litellm_params": {"model": "openai/gpt-5.4-nano", "api_key": "sk-priority-wire", "drop_params": True},+ },+ "openai/gpt-5.4-nano",+ {"specific_deployment": True},+ ),+ ],+ ids=["team-route", "deployment-id", "provider-model-name"],+)+async def test_deployment_drop_params_drop_a_non_integer_priority_on_every_deployment_route(+ deployment: dict[str, object],+ requested_model: str,+ routing_fields: dict[str, object],+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ monkeypatch.setattr(litellm, "drop_params", False)+ router: Final = Router(model_list=[deployment], default_priority=3)+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.acompletion(+ model=requested_model,+ messages=_PRIORITY_MESSAGES,+ mock_response="pong",+ priority="high",+ **routing_fields,+ )+ assert add_request.await_args.kwargs["request"].priority == 3+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+++async def test_schedule_acompletion_queues_a_prompt_management_model(monkeypatch: pytest.MonkeyPatch) -> None:+ router: Final = Router(+ model_list=[+ {+ "model_name": "cached-chat",+ "litellm_params": {+ "model": "anthropic_cache_control_hook/openai/gpt-5.4-nano",+ "api_key": "sk-priority-wire",+ "prompt_id": "cache-points",+ },+ }+ ]+ )+ provider: Final = AsyncMock(return_value=litellm.ModelResponse())+ monkeypatch.setattr(litellm, "acompletion", provider)+ logging_obj: Final = LiteLLMLogging(+ model="cached-chat",+ messages=_PRIORITY_MESSAGES,+ stream=False,+ call_type="acompletion",+ start_time=time.time(),+ litellm_call_id="prompt-management-priority",+ function_id="f",+ )+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.schedule_acompletion(+ model="cached-chat", messages=_PRIORITY_MESSAGES, priority=1, litellm_logging_obj=logging_obj+ )+ assert add_request.await_count == 1+ assert add_request.await_args.kwargs["request"].priority == 1+ assert provider.await_count == 1+ assert "priority" not in provider.await_args.kwargs+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+++@pytest.mark.parametrize(+ ("configured", "expected"),+ [(3, 3), ("3", 3), ("-1", -1), (0, 0), (None, None)],+ ids=["int", "digit-string", "negative-string", "zero", "unset"],+)+def test_the_router_parses_default_priority_once_at_init(configured: object, expected: int | None) -> None:+ assert _priority_router(configured).default_priority == expected+++@pytest.mark.parametrize("configured", ["high", 1.5, [3], False], ids=["word", "float", "list", "yaml-off"])+def test_an_invalid_default_priority_is_ignored_with_a_warning(+ configured: object, caplog: pytest.LogCaptureFixture+) -> None:+ with caplog.at_level(logging.WARNING, logger="LiteLLM Router"):+ router: Final = _priority_router(configured)+ assert router.default_priority is None+ assert any("default_priority is ignored" in record.getMessage() for record in caplog.records), caplog.records+++async def test_an_invalid_default_priority_leaves_every_request_unscheduled() -> None:+ router: Final = _priority_router("high")+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.acompletion(model="qa-chat", messages=_PRIORITY_MESSAGES, mock_response="pong")+ assert add_request.await_count == 0+ assert "x-litellm-request-prioritization-used" not in response._hidden_params.get("additional_headers", {})+++async def test_schedule_acompletion_without_a_priority_or_a_default_is_rejected_before_the_provider(+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch+) -> None:+ provider: Final = _priority_provider(respx_mock, monkeypatch)+ router: Final = _priority_router(None)+ with pytest.raises(litellm.BadRequestError, match="priority is required") as raised:+ await router.schedule_acompletion(model="qa-chat", messages=_PRIORITY_MESSAGES)+ assert raised.value.param == "priority"+ assert not provider.called+++async def test_schedule_acompletion_without_a_priority_queues_the_default_once() -> None:+ router: Final = _priority_router(3)+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.schedule_acompletion(+ model="qa-chat", messages=_PRIORITY_MESSAGES, mock_response="pong"+ )+ assert add_request.await_count == 1+ assert add_request.await_args.kwargs["request"].priority == 3+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+++@pytest.mark.parametrize(+ ("default_priority", "request_kwargs", "queued_priority"),+ [+ (3, {}, 3),+ (3, {"priority": None}, 3),+ (3, {"priority": 2}, 2),+ (None, {"priority": 0}, 0),+ (None, {"priority": "1"}, 1),+ (3, {"priority": "high", "drop_params": True}, 3),+ ],+)+async def test_atext_completion_queues_the_resolved_priority_exactly_once(+ default_priority: int | None,+ request_kwargs: dict[str, object],+ queued_priority: int,+ respx_mock: respx.MockRouter,+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ provider: Final = _priority_provider(respx_mock, monkeypatch)+ router: Final = _priority_router(default_priority)+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.atext_completion(model="qa-chat", prompt="pong?", **request_kwargs)+ assert add_request.await_count == 1+ assert add_request.await_args.kwargs["request"].priority == queued_priority+ assert response._hidden_params["additional_headers"]["x-litellm-request-prioritization-used"] is True+ assert "priority" not in json.loads(provider.calls.last.request.content)+++async def test_atext_completion_without_any_priority_never_enters_the_scheduler(+ respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch+) -> None:+ provider: Final = _priority_provider(respx_mock, monkeypatch)+ router: Final = _priority_router(None)+ with patch.object(router.scheduler, "add_request", wraps=router.scheduler.add_request) as add_request:+ response: Final = await router.atext_completion(model="qa-chat", prompt="pong?")+ assert add_request.await_count == 0+ assert "x-litellm-request-prioritization-used" not in response._hidden_params.get("additional_headers", {})+ assert provider.called+++@pytest.mark.parametrize("priority", ["high", [1], 1.5])+async def test_atext_completion_rejects_a_non_integer_priority_before_the_provider_sees_it(+ priority: object, respx_mock: respx.MockRouter, monkeypatch: pytest.MonkeyPatch+) -> None:+ provider: Final = _priority_provider(respx_mock, monkeypatch)+ router: Final = _priority_router(3)+ with pytest.raises(litellm.BadRequestError, match="priority must be an integer") as raised:+ await router.atext_completion(model="qa-chat", prompt="pong?", priority=priority)+ assert raised.value.param == "priority"+ assert not provider.called++ def test_router_multi_org_list() -> None: """ Pass list of orgs in 1 model definition,