Backtestslitellmrun-20261018-182052#38547
[Bug]: /v1/models never expands litellm_proxy/* wildcards — provider-endpoint discovery is gated on a static dict that excludes litellm_proxy
Our patch was judged better than the merged fix.
In this judgment, Patch B = our agent, Patch A = the merged fix.
Both patches fix the root cause identified in the issue: `get_provider_models` returning None before discovery is attempted for providers absent from the static `models_by_provider` catalog. Both admit providers that have a registered model-info config via `ProviderConfigManager.get_provider_model_info`, and all three independent tests pass on both. The difference is in how the gate is shaped. Patch B implements exactly the issue's suggested fix: discovery is admitted only when `litellm.check_provider_endpoint` is on, isolated in a small, documented helper `_supports_provider_endpoint_discovery`. Patch A drops the `check_provider_endpoint` condition entirely, so any provider with a model-info config now routes through `get_valid_models` even when discovery is off, changing the return value from None to [] for those providers. Patch A's own test documents this as a deliberate behavioral change, and it is probably harmless downstream, but it is a wider change than the issue asks for and relies on `get_valid_models` honoring the global flag internally rather than the gate doing so explicitly. Patch A's test for the discovery-off case also depends on the global `check_provider_endpoint` default rather than monkeypatching it, which is a minor fragility. On tests, Patch B's end-to-end test is stronger: it flushes the model cache, drives the real `LiteLLMProxyChatConfig.get_models` path by stubbing the HTTP client, and asserts the exact upstream URL and Authorization header built from the deployment's api_base/api_key, matching the issue's expectation that discovery uses the wildcard deployment's credentials. It also adds a hosted_vllm case and a negative test proving no upstream request is made when discovery is off. Patch A's test patches `OpenAIGPTConfig.get_models` at the base class, which verifies the credentials reach the fetcher but exercises less of the real path. Scope and craft favor B: it stays within the issue's ask, names and documents the new condition clearly, and keeps the original control flow intact.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
12m 4s
Wall-clock time in the container
Full test suite
No new failures
29 passed · 0 skipped · 29 total
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
133vs 86 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
What happened?
In a chained setup (LiteLLM in front of LiteLLM) using the documented wildcard config for the litellm_proxy provider with check_provider_endpoint: true, the downstream proxy's GET /v1/models returns the literal string litellm_proxy/* as its only model entry instead of the upstream's expanded model list. It fails silently: no HTTP request to the upstream is made and nothing is logged. Routing is unaffected — model: litellm_proxy/<name> requests forward to the upstream correctly — so only discovery is broken.
Expected: one litellm_proxy/<name> entry per model listed by the upstream, discovered via the wildcard deployment's api_base/api_key, as described in the model discovery docs, which list LiteLLM Proxy as a supported provider.
Root cause (from the source, main @ 6e569ee0): get_provider_models in litellm/proxy/auth/model_checks.py gates provider-endpoint discovery on membership in the static litellm.models_by_provider dict:
if provider in litellm.models_by_provider:
provider_models = get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params)
return provider_models
return None
litellm_proxy is not a key in that dict (litellm/__init__.py, _build_models_by_provider()), so the function returns None before discovery is attempted, and get_known_models_from_wildcard silently returns []. The wildcard-expansion path upstream of this point works correctly — _get_wildcard_models finds the deployment and extracts its litellm_params — and the provider's model-list fetcher exists and works (LiteLLMProxyChatConfig.get_models(), registered in ProviderConfigManager.get_provider_model_info), but it is unreachable through this gate. The effect is ironic: the providers that most need endpoint discovery (litellm_proxy, hosted_vllm — dynamic model lists with no static catalog) are exactly the ones a static-catalog membership test excludes. Setting LITELLM_PROXY_API_BASE/LITELLM_PROXY_API_KEY does not help this path; the code returns before credentials are consulted.
Suggested fix: admit providers that have a registered model-info config when check_provider_endpoint is enabled, rather than requiring membership in the static catalog:
if provider in litellm.models_by_provider or (
litellm.check_provider_endpoint
and ProviderConfigManager.get_provider_model_info(model=None, provider=LlmProviders(provider)) is not None
):
return get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params)
This fixes the same class of failure for any dynamically-listed provider, not just litellm_proxy.
Closest existing issues, none of which cover this: #13643 — same symptom family, stale-closed without a fix; #20064 — open, one comment describes this exact chained-proxy case; #13078 — closed "unable to repro", but the repro attempt used an openai config while the reporter hit it with litellm_proxy/hosted_vllm; #36945 — same symptom, different cause (STORE_MODEL_IN_DB drops the flag), for vertex_ai; #28264 — discovery failing to resolve stored litellm_credential_name credentials, fixed by #28419 (that fix is present and doesn't help here — this bug bails out before credentials are consulted at all); #31856 — open PR fixing the sibling gate for custom_openai only; #11784 — fixed a different variant.
User Flow
Before a (hypothetical) fix: a developer whose app lists models from a chained LLM gateway (a downstream LiteLLM proxy forwarding to a central upstream LiteLLM proxy) sees only a wildcard string instead of usable model names, so their model picker is broken.
- The developer sends GET http://litellm-downstream:4000/v1/models with their gateway key
- The response's
dataarray contains a single entry whose id is the literal stringlitellm_proxy/*— not a callable model name — no matter how many models the upstream serves - Their client app's model picker, which populates from that endpoint, shows only
litellm_proxy/*; picking it fails, and the developer has to learn real model names out-of-band - If the developer already knows a name, POST http://litellm-downstream:4000/v1/chat/completions with
"model": "litellm_proxy/gpt-4o"works — the request forwards through to the upstream and on to the provider
After a (hypothetical) fix: the same gateway's model list shows every model the upstream serves, so the picker just works.
- The developer sends GET http://litellm-downstream:4000/v1/models with their gateway key
- The response's
dataarray contains one entry per upstream model — e.g.litellm_proxy/gpt-4o,litellm_proxy/claude-sonnet - Their client app's model picker shows those names; the developer picks one
- POST http://litellm-downstream:4000/v1/chat/completions with the picked name works exactly as before
Proof the bug occurs
Version: v1.98.0 (pip litellm[proxy]==1.98.0), locally on Linux, no database.
Config the upstream proxy ran with (upstream.yaml, port 4010):
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: sk-fake-upstream-provider-key
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-sonnet-4-5
api_key: sk-fake-upstream-provider-key
general_settings:
master_key: sk-upstream-1234
Config the downstream proxy ran with (downstream.yaml, port 4020), no extra env vars:
model_list:
- model_name: "litellm_proxy/*"
litellm_params:
model: "litellm_proxy/*"
api_base: http://127.0.0.1:4010
api_key: sk-upstream-1234
litellm_settings:
check_provider_endpoint: true
general_settings:
master_key: sk-downstream-1234
Started with litellm --config upstream.yaml --port 4010 and litellm --config downstream.yaml --port 4020.
The upstream lists its models fine:
$ curl -s http://127.0.0.1:4010/v1/models -H "Authorization: Bearer sk-upstream-1234"
{"data":[{"id":"gpt-4o","object":"model","created":1677610602,"owned_by":"openai","mode":"chat","max_input_tokens":128000,"max_output_tokens":16384},{"id":"claude-sonnet","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}
The downstream returns only the literal wildcard — this is the bug:
$ curl -s http://127.0.0.1:4020/v1/models -H "Authorization: Bearer sk-downstream-1234"
{"data":[{"id":"litellm_proxy/*","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}
Routing through the same chain works (the request traverses downstream → upstream → OpenAI's real API, which rejects the deliberately fake provider key — proving the forwarding path is fully functional):
$ curl -s http://127.0.0.1:4020/v1/chat/completions -H "Authorization: Bearer sk-downstream-1234" \
-H "Content-Type: application/json" \
-d '{"model":"litellm_proxy/gpt-4o","messages":[{"role":"user","content":"hi"}]}'
{"error":{"message":"litellm.AuthenticationError: Error code: 401 - {'error': {'message': 'litellm.AuthenticationError: AuthenticationError: OpenAIException - Incorrect API key provided: sk-fake-*****************-key. ...'}}. Received Model Group=litellm_proxy/gpt-4o\nAvailable Model Group Fallbacks=None","type":null,"param":null,"code":"401"}}
During the /v1/models request the upstream's access log shows no incoming request — the discovery fetch is never attempted.
Workaround confirming the diagnosis: the only change below is adding infer_model_from_keys: true and the two env vars — this reaches the same provider fetcher through the env-inference path, which does not go through the models_by_provider gate:
general_settings:
master_key: sk-downstream-1234
infer_model_from_keys: true
$ LITELLM_PROXY_API_BASE=http://127.0.0.1:4010 LITELLM_PROXY_API_KEY=sk-upstream-1234 \
litellm --config downstream-workaround.yaml --port 4020
$ curl -s http://127.0.0.1:4020/v1/models -H "Authorization: Bearer sk-downstream-1234"
{"data":[{"id":"litellm_proxy/*","object":"model","created":1677610602,"owned_by":"openai"},{"id":"litellm_proxy/gpt-4o","object":"model","created":1677610602,"owned_by":"openai","mode":"chat","max_input_tokens":128000,"max_output_tokens":16384},{"id":"litellm_proxy/claude-sonnet","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}
(The literal litellm_proxy/* entry still leaking into the successful listing is the separate known issue being addressed in #29165/#36398.)
What part of LiteLLM is this about?
Proxy
What LiteLLM version are you on ?
v1.98.0
Twitter / LinkedIn details
_No response_
The two patches
litellm/proxy/auth/model_checks.pyindex f48f5dd716..0a2df02431 100644@@ -13,7 +13,7 @@ from litellm.router import Router from litellm.router_utils.fallback_event_handlers import get_fallback_model_group from litellm.types.router import CredentialLiteLLMParams, LiteLLM_Params from litellm.types.utils import LlmProviders-from litellm.utils import get_valid_models+from litellm.utils import ProviderConfigManager, get_valid_models _CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields) @@ -42,12 +42,29 @@ def get_provider_models(provider: str, litellm_params: LiteLLM_Params | None = N if provider == "*": return get_valid_models(litellm_params=litellm_params) - if provider in litellm.models_by_provider:+ if provider in litellm.models_by_provider or _supports_provider_endpoint_discovery(provider): provider_models: Final = get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params) return provider_models return None +def _supports_provider_endpoint_discovery(provider: str) -> bool:+ """+ Returns True if `check_provider_endpoint` is enabled and the provider has a registered+ model-info config that can list its models live.++ Providers with dynamic model lists (e.g. `litellm_proxy`, `hosted_vllm`) have no entry in+ the static `litellm.models_by_provider` catalog, so the endpoint is their only source.+ """+ if not litellm.check_provider_endpoint:+ return False+ try:+ llm_provider: Final = LlmProviders(provider)+ except ValueError:+ return False+ return ProviderConfigManager.get_provider_model_info(model=None, provider=llm_provider) is not None++ def _get_models_from_access_groups( model_access_groups: dict[str, list[str]], all_models: list[str],tests/unit/proxy/auth/test_model_checks.pyindex 13171a42cd..73fe288790 100644@@ -980,3 +980,115 @@ def test_transcribe_is_a_known_provider_for_wildcard_expansion(): assert get_known_models_from_wildcard("transcribe/*") == [ "transcribe/StartTranscriptionJob" ]+++class _FakeModelsResponse:+ status_code = 200+ text = ""++ def __init__(self, model_ids):+ self._model_ids = model_ids++ def json(self):+ return {"data": [{"id": model_id, "object": "model"} for model_id in self._model_ids]}++ def raise_for_status(self):+ return None+++def _patch_upstream_models(monkeypatch, model_ids):+ import litellm++ calls = []++ def fake_get(url, headers=None, **kwargs):+ calls.append({"url": url, "headers": headers})+ return _FakeModelsResponse(model_ids)++ monkeypatch.setattr(litellm.module_level_client, "get", fake_get)+ return calls+++def test_get_complete_model_list_expands_litellm_proxy_wildcard_via_provider_endpoint(monkeypatch):+ """+ Chained proxy: a `litellm_proxy/*` wildcard with check_provider_endpoint must list the+ upstream proxy's models, using the deployment's api_base/api_key. `litellm_proxy` has no+ entry in the static models_by_provider catalog, which previously short-circuited discovery.+ """+ import litellm+ from litellm import Router+ from litellm.proxy.auth.model_checks import get_complete_model_list+ from litellm.utils import _model_cache++ _model_cache.flush_cache()+ monkeypatch.setattr(litellm, "check_provider_endpoint", True)+ calls = _patch_upstream_models(monkeypatch, ["gpt-4o", "claude-sonnet"])++ router = Router(+ model_list=[+ {+ "model_name": "litellm_proxy/*",+ "litellm_params": {+ "model": "litellm_proxy/*",+ "api_base": "http://upstream-litellm-proxy-discovery.test:4010",+ "api_key": "sk-upstream-1234",+ },+ }+ ]+ )++ result = get_complete_model_list(+ key_models=[],+ team_models=[],+ proxy_model_list=["litellm_proxy/*"],+ user_model=None,+ infer_model_from_keys=False,+ llm_router=router,+ )++ assert result == ["litellm_proxy/gpt-4o", "litellm_proxy/claude-sonnet"]+ assert calls == [+ {+ "url": "http://upstream-litellm-proxy-discovery.test:4010/v1/models",+ "headers": {"Authorization": "Bearer sk-upstream-1234"},+ }+ ]+++def test_get_provider_models_admits_hosted_vllm_when_check_provider_endpoint(monkeypatch):+ """Same gate, another dynamically-listed provider with no static catalog entry."""+ import litellm+ from litellm.proxy.auth import model_checks+ from litellm.types.router import LiteLLM_Params++ monkeypatch.setattr(litellm, "check_provider_endpoint", True)+ captured = {}++ def fake_get_valid_models(custom_llm_provider=None, litellm_params=None):+ captured["provider"] = custom_llm_provider+ captured["api_base"] = litellm_params.api_base+ return ["hosted_vllm/meta-llama/Llama-3-8B"]++ monkeypatch.setattr(model_checks, "get_valid_models", fake_get_valid_models)++ assert "hosted_vllm" not in litellm.models_by_provider+ result = model_checks.get_provider_models(+ provider="hosted_vllm",+ litellm_params=LiteLLM_Params(model="hosted_vllm/*", api_base="http://vllm.test:8000"),+ )++ assert result == ["hosted_vllm/meta-llama/Llama-3-8B"]+ assert captured == {"provider": "hosted_vllm", "api_base": "http://vllm.test:8000"}+++def test_litellm_proxy_wildcard_not_discovered_without_check_provider_endpoint(monkeypatch):+ """Without check_provider_endpoint, no upstream request is made for providers with no static catalog."""+ import litellm+ from litellm.proxy.auth.model_checks import get_provider_models++ monkeypatch.setattr(litellm, "check_provider_endpoint", False)+ calls = _patch_upstream_models(monkeypatch, ["gpt-4o"])++ assert get_provider_models(provider="litellm_proxy") is None+ assert get_provider_models(provider="not-a-real-provider") is None+ assert calls == []
litellm/proxy/auth/model_checks.pyindex f48f5dd7162..133e95eadba 100644@@ -13,7 +13,7 @@ from litellm.router import Router from litellm.router_utils.fallback_event_handlers import get_fallback_model_group from litellm.types.router import CredentialLiteLLMParams, LiteLLM_Params from litellm.types.utils import LlmProviders-from litellm.utils import get_valid_models+from litellm.utils import ProviderConfigManager, get_valid_models _CREDENTIAL_LITELLM_PARAM_FIELDS = set(CredentialLiteLLMParams.model_fields) @@ -45,7 +45,14 @@ def get_provider_models(provider: str, litellm_params: LiteLLM_Params | None = N if provider in litellm.models_by_provider: provider_models: Final = get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params) return provider_models- return None++ try:+ llm_provider: Final = LlmProviders(provider)+ except ValueError:+ return None+ if ProviderConfigManager.get_provider_model_info(model=None, provider=llm_provider) is None:+ return None+ return get_valid_models(custom_llm_provider=provider, litellm_params=litellm_params) def _get_models_from_access_groups(tests/unit/proxy/auth/test_model_checks.pyindex 13171a42cda..88c2d9c684a 100644@@ -967,6 +967,81 @@ def test_get_complete_model_list_sentinel_only_grants_nothing(): assert result == [] +def test_get_provider_models_admits_providers_without_a_static_catalog():+ """Providers without a static model list are no longer rejected up front.++ With endpoint discovery off, get_valid_models falls back to the (empty)+ static list, so the result is [] rather than None. Before the fix this+ returned None and the wildcard was never expanded.+ """+ import litellm+ from litellm.proxy.auth.model_checks import get_provider_models+ from litellm.types.router import LiteLLM_Params++ assert "litellm_proxy" not in litellm.models_by_provider+ assert "hosted_vllm" not in litellm.models_by_provider++ result = get_provider_models(+ "litellm_proxy",+ litellm_params=LiteLLM_Params(+ model="litellm_proxy/*",+ api_base="http://upstream:4000",+ api_key="sk-upstream",+ ),+ )++ assert result == []+++def test_get_complete_model_list_discovers_litellm_proxy_wildcard_models(monkeypatch):+ """A litellm_proxy/* deployment lists the upstream proxy's models when endpoint discovery is on."""+ import litellm+ from litellm import Router+ from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig+ from litellm.proxy.auth.model_checks import get_complete_model_list++ monkeypatch.setattr(litellm, "check_provider_endpoint", True)+ captured = {}++ def fake_get_models(self, api_key=None, api_base=None):+ captured["api_key"] = api_key+ captured["api_base"] = api_base+ return ["gpt-4o", "claude-sonnet"]++ monkeypatch.setattr(OpenAIGPTConfig, "get_models", fake_get_models)++ router = Router(+ model_list=[+ {+ "model_name": "litellm_proxy/*",+ "litellm_params": {+ "model": "litellm_proxy/*",+ "api_base": "http://upstream:4000",+ "api_key": "sk-upstream",+ },+ }+ ]+ )+ result = get_complete_model_list(+ key_models=[],+ team_models=[],+ proxy_model_list=["litellm_proxy/*"],+ user_model=None,+ infer_model_from_keys=False,+ llm_router=router,+ )++ assert captured == {"api_key": "sk-upstream", "api_base": "http://upstream:4000"}+ assert "litellm_proxy/gpt-4o" in result+ assert "litellm_proxy/claude-sonnet" in result+++def test_get_provider_models_returns_none_for_an_unknown_provider():+ from litellm.proxy.auth.model_checks import get_provider_models++ assert get_provider_models("not-a-real-provider") is None++ def test_transcribe_is_a_known_provider_for_wildcard_expansion(): import litellm from litellm.proxy.auth.model_checks import (