Backtestslitellmrun-20261019-033420#38547
[Bug]: /v1/models never expands litellm_proxy/* wildcards — provider-endpoint discovery is gated on a static dict that excludes litellm_proxy
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches hit the root cause: the static `models_by_provider` membership test in `get_provider_models` that rejected `litellm_proxy`/`hosted_vllm` before discovery was attempted. Both admit providers whose `ProviderConfigManager.get_provider_model_info` is registered, and both pass all four independent tests, so correctness on the reported scenario is equivalent. The differences are in guarding and tests. Patch B gates the new admission on `litellm.check_provider_endpoint`, matching the issue's suggested fix, so with discovery disabled the function still returns `None` and never reaches `get_valid_models` for catalog-less providers. Patch A admits such providers unconditionally and relies on `get_valid_models` falling back to an empty static list; its own test documents the resulting `[]`-instead-of-`None` behaviour change with discovery off. That is likely harmless downstream, but it is an unnecessary semantic shift and a small added risk that B avoids. B also factors the check into a documented helper, which reads more clearly than A's inline early-returns. On tests, B is stronger: its main regression test mocks only the HTTP client and asserts the real `LiteLLMProxyChatConfig.get_models` path builds `<api_base>/v1/models` with the deployment's bearer key and that the final list is exactly the prefixed upstream models. It also covers `hosted_vllm` wildcard expansion with deployment params, the disabled-flag no-network case, and unknown providers. Patch A's integration test mocks `OpenAIGPTConfig.get_models` (relying on inheritance rather than exercising the proxy config's request building), and its first unit test mostly asserts the `[]` side effect rather than the fixed behaviour. Scope is comparable and minimal for both. B wins on tighter guarding consistent with the issue's intent and on more meaningful tests.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
45m 17s
Wall-clock time in the container
Full test suite
No new failures
29 passed · 0 skipped · 29 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
127vs 86 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
In a chained setup (LiteLLM in front of LiteLLM) using the documented wildcard config for the litellm_proxy provider with check_provider_endpoint: true, the downstream proxy's GET /v1/models returns the literal string litellm_proxy/* as its only model entry instead of the upstream's expanded model list. It fails silently: no HTTP request to the upstream is made and nothing is logged. Routing is unaffected — model: litellm_proxy/<name> requests forward to the upstream correctly — so only discovery is broken.
Expected: one litellm_proxy/<name> entry per model listed by the upstream, discovered via the wildcard deployment's api_base/api_key, as described in the model discovery docs, which list LiteLLM Proxy as a supported provider.
Root cause (from the source, main @ 6e569ee0): get_provider_models in litellm/proxy/auth/model_checks.py gates provider-endpoint discovery on membership in the static litellm.models_by_provider dict:
if provider in litellm.models_by_provider:
provider_models = get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params)
return provider_models
return None
litellm_proxy is not a key in that dict (litellm/__init__.py, _build_models_by_provider()), so the function returns None before discovery is attempted, and get_known_models_from_wildcard silently returns []. The wildcard-expansion path upstream of this point works correctly — _get_wildcard_models finds the deployment and extracts its litellm_params — and the provider's model-list fetcher exists and works (LiteLLMProxyChatConfig.get_models(), registered in ProviderConfigManager.get_provider_model_info), but it is unreachable through this gate. The effect is ironic: the providers that most need endpoint discovery (litellm_proxy, hosted_vllm — dynamic model lists with no static catalog) are exactly the ones a static-catalog membership test excludes. Setting LITELLM_PROXY_API_BASE/LITELLM_PROXY_API_KEY does not help this path; the code returns before credentials are consulted.
Suggested fix: admit providers that have a registered model-info config when check_provider_endpoint is enabled, rather than requiring membership in the static catalog:
if provider in litellm.models_by_provider or (
litellm.check_provider_endpoint
and ProviderConfigManager.get_provider_model_info(model=None, provider=LlmProviders(provider)) is not None
):
return get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params)
This fixes the same class of failure for any dynamically-listed provider, not just litellm_proxy.
Closest existing issues, none of which cover this: #13643 — same symptom family, stale-closed without a fix; #20064 — open, one comment describes this exact chained-proxy case; #13078 — closed "unable to repro", but the repro attempt used an openai config while the reporter hit it with litellm_proxy/hosted_vllm; #36945 — same symptom, different cause (STORE_MODEL_IN_DB drops the flag), for vertex_ai; #28264 — discovery failing to resolve stored litellm_credential_name credentials, fixed by #28419 (that fix is present and doesn't help here — this bug bails out before credentials are consulted at all); #31856 — open PR fixing the sibling gate for custom_openai only; #11784 — fixed a different variant.
User Flow
Before a (hypothetical) fix: a developer whose app lists models from a chained LLM gateway (a downstream LiteLLM proxy forwarding to a central upstream LiteLLM proxy) sees only a wildcard string instead of usable model names, so their model picker is broken.
- The developer sends GET http://litellm-downstream:4000/v1/models with their gateway key
- The response's
dataarray contains a single entry whose id is the literal stringlitellm_proxy/*— not a callable model name — no matter how many models the upstream serves - Their client app's model picker, which populates from that endpoint, shows only
litellm_proxy/*; picking it fails, and the developer has to learn real model names out-of-band - If the developer already knows a name, POST http://litellm-downstream:4000/v1/chat/completions with
"model": "litellm_proxy/gpt-4o"works — the request forwards through to the upstream and on to the provider
After a (hypothetical) fix: the same gateway's model list shows every model the upstream serves, so the picker just works.
- The developer sends GET http://litellm-downstream:4000/v1/models with their gateway key
- The response's
dataarray contains one entry per upstream model — e.g.litellm_proxy/gpt-4o,litellm_proxy/claude-sonnet - Their client app's model picker shows those names; the developer picks one
- POST http://litellm-downstream:4000/v1/chat/completions with the picked name works exactly as before
Proof the bug occurs
Version: v1.98.0 (pip litellm[proxy]==1.98.0), locally on Linux, no database.
Config the upstream proxy ran with (upstream.yaml, port 4010):
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-fake-upstream-provider-key
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-5
api_key: sk-fake-upstream-provider-key
general_settings:
master_key: sk-upstream-1234
Config the downstream proxy ran with (downstream.yaml, port 4020), no extra env vars:
model_list:
- model_name: "litellm_proxy/*"
litellm_params:
model: "litellm_proxy/*"
api_base: http://127.0.0.1:4010
api_key: sk-upstream-1234
litellm_settings:
check_provider_endpoint: true
general_settings:
master_key: sk-downstream-1234
Started with litellm --config upstream.yaml --port 4010 and litellm --config downstream.yaml --port 4020.
The upstream lists its models fine:
$ curl -s http://127.0.0.1:4010/v1/models -H "Authorization: Bearer sk-upstream-1234"
{"data":[{"id":"gpt-4o","object":"model","created":1677610602,"owned_by":"openai","mode":"chat","max_input_tokens":128000,"max_output_tokens":16384},{"id":"claude-sonnet","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}
The downstream returns only the literal wildcard — this is the bug:
$ curl -s http://127.0.0.1:4020/v1/models -H "Authorization: Bearer sk-downstream-1234"
{"data":[{"id":"litellm_proxy/*","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}
Routing through the same chain works (the request traverses downstream → upstream → OpenAI's real API, which rejects the deliberately fake provider key — proving the forwarding path is fully functional):
$ curl -s http://127.0.0.1:4020/v1/chat/completions -H "Authorization: Bearer sk-downstream-1234" \
-H "Content-Type: application/json" \
-d '{"model":"litellm_proxy/gpt-4o","messages":[{"role":"user","content":"hi"}]}'
{"error":{"message":"litellm.AuthenticationError: Error code: 401 - {'error': {'message': 'litellm.AuthenticationError: AuthenticationError: OpenAIException - Incorrect API key provided: sk-fake-*****************-key. ...'}}. Received Model Group=litellm_proxy/gpt-4o\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"401"}}
During the /v1/models request the upstream's access log shows no incoming request — the discovery fetch is never attempted.
Workaround confirming the diagnosis: the only change below is adding infer_model_from_keys: true and the two env vars — this reaches the same provider fetcher through the env-inference path, which does not go through the models_by_provider gate:
general_settings:
master_key: sk-downstream-1234
infer_model_from_keys: true
$ LITELLM_PROXY_API_BASE=http://127.0.0.1:4010 LITELLM_PROXY_API_KEY=sk-upstream-1234 \
litellm --config downstream-workaround.yaml --port 4020
$ curl -s http://127.0.0.1:4020/v1/models -H "Authorization: Bearer sk-downstream-1234"
{"data":[{"id":"litellm_proxy/*","object":"model","created":1677610602,"owned_by":"openai"},{"id":"litellm_proxy/gpt-4o","object":"model","created":1677610602,"owned_by":"openai","mode":"chat","max_input_tokens":128000,"max_output_tokens":16384},{"id":"litellm_proxy/claude-sonnet","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}
(The literal litellm_proxy/* entry still leaking into the successful listing is the separate known issue being addressed in #29165/#36398.)
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
v1.98.0
Twitter / LinkedIn details
_No response_
The two patches
litellm/proxy/auth/model_checks.pyindex f48f5dd716..79cc616f00 100644@@ -13,7 +13,7 @@ from litellm.router import Router from litellm.router_utils.fallback_event_handlers import get_fallback_model_group from litellm.types.router import CredentialLiteLLMParams, LiteLLM_Params from litellm.types.utils import LlmProviders-from litellm.utils import get_valid_models+from litellm.utils import ProviderConfigManager, get_valid_models _CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields) @@ -42,12 +42,29 @@ def get_provider_models(provider: str, litellm_params: LiteLLM_Params | None = N if provider == "*": return get_valid_models(litellm_params=litellm_params) - if provider in litellm.models_by_provider:+ if provider in litellm.models_by_provider or (+ litellm.check_provider_endpoint and _has_provider_model_info(provider)+ ): provider_models: Final = get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params) return provider_models return None +def _has_provider_model_info(provider: str) -> bool:+ """+ Returns True if the provider can list its models from its own endpoint.++ Dynamically-listed providers (e.g. `litellm_proxy`, `hosted_vllm`) have no static+ catalog in `litellm.models_by_provider`, so discovery must be admitted via their+ registered model-info config instead.+ """+ try:+ llm_provider: Final = LlmProviders(provider)+ except ValueError:+ return False+ return ProviderConfigManager.get_provider_model_info(model=None, provider=llm_provider) is not None++ def _get_models_from_access_groups( model_access_groups: dict[str, list[str]], all_models: list[str],tests/unit/proxy/auth/test_model_checks.pyindex 13171a42cd..156804c43a 100644@@ -980,3 +980,109 @@ def test_transcribe_is_a_known_provider_for_wildcard_expansion(): assert get_known_models_from_wildcard("transcribe/*") == [ "transcribe/StartTranscriptionJob" ]+++def _mock_models_endpoint_response(model_ids):+ from unittest.mock import MagicMock++ response = MagicMock()+ response.status_code = 200+ response.json.return_value = {"data": [{"id": model_id} for model_id in model_ids]}+ return response+++def test_get_complete_model_list_expands_litellm_proxy_wildcard_via_provider_endpoint(+ monkeypatch,+):+ """litellm_proxy has no static catalog in models_by_provider, so a chained-proxy+ `litellm_proxy/*` deployment with check_provider_endpoint must still be expanded by+ querying the upstream proxy's /v1/models."""+ import litellm+ from litellm import Router+ from litellm.proxy.auth.model_checks import get_complete_model_list++ assert "litellm_proxy" not in litellm.models_by_provider+ monkeypatch.setattr(litellm, "check_provider_endpoint", True)+ router = Router(+ model_list=[+ {+ "model_name": "litellm_proxy/*",+ "litellm_params": {+ "model": "litellm_proxy/*",+ "api_base": "http://upstream-litellm-proxy-regression:4010",+ "api_key": "sk-upstream-1234",+ },+ }+ ]+ )++ with patch.object(+ litellm.module_level_client,+ "get",+ return_value=_mock_models_endpoint_response(["gpt-4o", "claude-sonnet"]),+ ) as mock_get:+ result = get_complete_model_list(+ key_models=[],+ team_models=[],+ proxy_model_list=["litellm_proxy/*"],+ user_model=None,+ infer_model_from_keys=False,+ llm_router=router,+ )++ mock_get.assert_called_once()+ assert mock_get.call_args.kwargs["url"] == "http://upstream-litellm-proxy-regression:4010/v1/models"+ assert mock_get.call_args.kwargs["headers"] == {"Authorization": "Bearer sk-upstream-1234"}+ assert result == ["litellm_proxy/gpt-4o", "litellm_proxy/claude-sonnet"]+++def test_get_known_models_from_wildcard_hosted_vllm_reaches_provider_discovery(monkeypatch):+ """hosted_vllm is another dynamically-listed provider without a static catalog; its+ wildcard must be handed to provider discovery with the deployment's params."""+ import litellm+ from litellm.proxy.auth import model_checks+ from litellm.types.router import LiteLLM_Params++ assert "hosted_vllm" not in litellm.models_by_provider+ monkeypatch.setattr(litellm, "check_provider_endpoint", True)+ deployment_params = LiteLLM_Params(+ model="hosted_vllm/*",+ api_base="http://hosted-vllm-regression:8000/v1",+ )++ with patch.object(+ model_checks, "get_valid_models", return_value=["meta-llama/Llama-3-8B"]+ ) as mock_get_valid_models:+ result = model_checks.get_known_models_from_wildcard(+ "hosted_vllm/*", litellm_params=deployment_params+ )++ mock_get_valid_models.assert_called_once()+ assert mock_get_valid_models.call_args.kwargs["custom_llm_provider"] == "hosted_vllm"+ assert (+ mock_get_valid_models.call_args.kwargs["litellm_params"].api_base+ == "http://hosted-vllm-regression:8000/v1"+ )+ assert result == ["hosted_vllm/meta-llama/Llama-3-8B"]+++def test_get_provider_models_without_check_provider_endpoint_skips_dynamic_provider(+ monkeypatch,+):+ """Without check_provider_endpoint, providers lacking a static catalog are not+ queried over the network."""+ import litellm+ from litellm.proxy.auth.model_checks import get_provider_models++ monkeypatch.setattr(litellm, "check_provider_endpoint", False)+ with patch.object(litellm.module_level_client, "get") as mock_get:+ assert get_provider_models("litellm_proxy") is None+ mock_get.assert_not_called()+++def test_get_provider_models_unknown_provider_returns_none(monkeypatch):+ import litellm+ from litellm.proxy.auth.model_checks import get_provider_models++ monkeypatch.setattr(litellm, "check_provider_endpoint", True)+ assert get_provider_models("not-a-real-provider") is None
litellm/proxy/auth/model_checks.pyindex f48f5dd7162..133e95eadba 100644@@ -13,7 +13,7 @@ from litellm.router import Router from litellm.router_utils.fallback_event_handlers import get_fallback_model_group from litellm.types.router import CredentialLiteLLMParams, LiteLLM_Params from litellm.types.utils import LlmProviders-from litellm.utils import get_valid_models+from litellm.utils import ProviderConfigManager, get_valid_models _CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields) @@ -45,7 +45,14 @@ def get_provider_models(provider: str, litellm_params: LiteLLM_Params | None = N if provider in litellm.models_by_provider: provider_models: Final = get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params) return provider_models- return None++ try:+ llm_provider: Final = LlmProviders(provider)+ except ValueError:+ return None+ if ProviderConfigManager.get_provider_model_info(model=None, provider=llm_provider) is None:+ return None+ return get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params) def _get_models_from_access_groups(tests/unit/proxy/auth/test_model_checks.pyindex 13171a42cda..88c2d9c684a 100644@@ -967,6 +967,81 @@ def test_get_complete_model_list_sentinel_only_grants_nothing(): assert result == [] +def test_get_provider_models_admits_providers_without_a_static_catalog():+ """Providers without a static model list are no longer rejected up front.++ With endpoint discovery off, get_valid_models falls back to the (empty)+ static list, so the result is [] rather than None. Before the fix this+ returned None and the wildcard was never expanded.+ """+ import litellm+ from litellm.proxy.auth.model_checks import get_provider_models+ from litellm.types.router import LiteLLM_Params++ assert "litellm_proxy" not in litellm.models_by_provider+ assert "hosted_vllm" not in litellm.models_by_provider++ result = get_provider_models(+ "litellm_proxy",+ litellm_params=LiteLLM_Params(+ model="litellm_proxy/*",+ api_base="http://upstream:4000",+ api_key="sk-upstream",+ ),+ )++ assert result == []+++def test_get_complete_model_list_discovers_litellm_proxy_wildcard_models(monkeypatch):+ """A litellm_proxy/* deployment lists the upstream proxy's models when endpoint discovery is on."""+ import litellm+ from litellm import Router+ from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig+ from litellm.proxy.auth.model_checks import get_complete_model_list++ monkeypatch.setattr(litellm, "check_provider_endpoint", True)+ captured = {}++ def fake_get_models(self, api_key=None, api_base=None):+ captured["api_key"] = api_key+ captured["api_base"] = api_base+ return ["gpt-4o", "claude-sonnet"]++ monkeypatch.setattr(OpenAIGPTConfig, "get_models", fake_get_models)++ router = Router(+ model_list=[+ {+ "model_name": "litellm_proxy/*",+ "litellm_params": {+ "model": "litellm_proxy/*",+ "api_base": "http://upstream:4000",+ "api_key": "sk-upstream",+ },+ }+ ]+ )+ result = get_complete_model_list(+ key_models=[],+ team_models=[],+ proxy_model_list=["litellm_proxy/*"],+ user_model=None,+ infer_model_from_keys=False,+ llm_router=router,+ )++ assert captured == {"api_key": "sk-upstream", "api_base": "http://upstream:4000"}+ assert "litellm_proxy/gpt-4o" in result+ assert "litellm_proxy/claude-sonnet" in result+++def test_get_provider_models_returns_none_for_an_unknown_provider():+ from litellm.proxy.auth.model_checks import get_provider_models++ assert get_provider_models("not-a-real-provider") is None++ def test_transcribe_is_a_known_provider_for_wildcard_expansion(): import litellm from litellm.proxy.auth.model_checks import (