Backtestslitellmrun-20261111-220245#34367
[Feature]: Model Group Budget Limits for API Key Usage
Our patch was judged better than the merged fix.
In this judgment, Patch A = our agent, Patch B = the merged fix.
Both patches make a shared cap across a family of models enforceable for keys, teams, users and end users, and both pass the core independent tests (combined spend across Opus versions blocks every member, non-members unaffected). They diverge on the scenario where a member also has its own roomier per-model entry. Patch A checks a model against its own entry and every group listing it, so the group's combined cap still binds; the independent test for this passes. Patch B uses a precedence model (exact entry, then access group, then most-specific glob) in which the per-model entry wins and the group counter is never charged or consulted, so the test fails with no BudgetExceededError. That precedence design lets a single per-model entry quietly carve a model out of the family cap, which is exactly the overspend the issue describes, and B's own tests confirm this is intentional behaviour rather than an oversight. On scope and risk, A is a contained change: a `models` field on BudgetConfig with validation and backward-compatible serialization, a multi-budget resolver, a tpm/rpm guard, a small team-authority extension, and focused tests. B bundles a very large redesign: glob matching with precedence rules, router access-group lookups keyed on team ids, a new database-backed shared-window subsystem with raw upsert SQL for Redis-less replicas, Bedrock cross-region request-form enumeration for team-admin authority checks, router API additions, docstring edits across endpoints, and UI combobox changes. Much of that is unrelated to the ask, adds considerable surface for regressions (new SQL path, read-replica races, in-memory 'unshared' spend bookkeeping), and makes the actual fix hard to locate. B's tests are extensive but largely exercise that adjacent machinery. A's tests target the issue directly (combined cap per entity type, group plus own-entry interaction, group named like a model, malformed groups, validation, serialization, team authority). A resolves the issue at its root with less risk and better matches the stated intent that combined spend governs regardless of per-model entries.
Four judges ruled on this ticket in parallel, and the verdict comes from counting their votes. Each saw the issue and the two patches, labelled A and B in random order (two of them saw ours as A), with issue links and version numbers removed. Where tests could check the fix, they also saw the results of tests a separate agent wrote for the issue, run on both patches. None was told which patch was ours. The reasoning above is one judge's, shown verbatim.
Solve
56m 4s
Wall-clock time in the container
Full test suite
Not run
The harness did not reach the test stage
Contamination audit
Clean
No sign the agent saw the real fix
Patch size
503vs 2,434 merged
Changed lines, ours vs merged
The issue
View on GitHubCheck for existing issues
- I have searched the existing issues and checked that my issue is not a duplicate.
The Feature
Add support for defining a shared budget at the model_group level for an API key. All models within the group should draw from the same combined budget.
Motivation, pitch
Teams may want to limit developer spending on a family of expensive models, such as multiple Claude Opus versions. Today, budgets can be set per individual model, but each model receives its own limit. This means developers can exceed the intended total cap by using several models in the same expensive family.
A model-group budget would allow configuring one shared limit across models—for example, a single monthly budget covering all Opus model versions—so usage is enforced against the combined spend rather than per-model spend.
What part of LiteLLM is this about?
Proxy
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
No
Twitter / LinkedIn details
_No response_
The two patches
litellm/proxy/auth/auth_utils.pyindex aa6ddae438..01bccc1313 100644@@ -1246,7 +1246,11 @@ def get_key_own_model_rate_limit( model_limit: Final = { model: budget[budget_key] for model, budget in user_api_key_dict.model_max_budget.items()- if isinstance(budget, dict) and budget.get(budget_key) is not None+ # A model group entry is a shared spend cap, not a per-model rate limit+ # (validate_model_max_budget rejects tpm/rpm on one).+ if isinstance(budget, dict)+ and budget.get(budget_key) is not None+ and ("models" not in budget or budget["models"] is None) } return model_limit or None litellm/proxy/hooks/model_max_budget_limiter.pyindex bc9cee1cb1..866c793a29 100644@@ -66,6 +66,8 @@ class ResolvedModelBudget: budget_model: str budget_config: BudgetConfig+ # The entry lists `models`: one cap shared by every model it names.+ is_model_group: bool = False def team_member_budget_entity_id(user_id: str, team_id: str) -> str:@@ -152,16 +154,72 @@ def resolve_model_budget(model: str, model_max_budget: Mapping[str, object]) -> candidate, ) continue- return ResolvedModelBudget(budget_model=candidate, budget_config=budget_config)+ return ResolvedModelBudget(+ budget_model=candidate,+ budget_config=budget_config,+ is_model_group=budget_config.models is not None,+ ) return None +def resolve_model_budgets(model: str, model_max_budget: Mapping[str, object]) -> tuple[ResolvedModelBudget, ...]:+ """Every `model_max_budget` entry that governs `model`: its own entry, then each model group listing it.++ A model group entry (one with ``models``) is keyed on the group's name, so+ every model it lists shares one spend counter and one cap. A model may sit+ under its own entry and several groups at once, and must stay within all of+ them, otherwise spreading usage across a group's models would exceed the+ combined cap the group exists to enforce.+ """+ # A group governs only the models it lists, never a model that merely shares+ # its name, so the model's own entry is looked up among per-model entries+ # alone. Otherwise a group named like a model would capture that model and+ # hide the model's own cap.+ direct: Final = resolve_model_budget(+ model=model,+ model_max_budget={name: raw for name, raw in model_max_budget.items() if not _raw_group_models(raw)},+ )+ candidates: Final = frozenset(_budget_model_candidates(model))+ groups: Final = tuple(+ ResolvedModelBudget(budget_model=group_name, budget_config=budget_config, is_model_group=True)+ for group_name, raw_budget_config in model_max_budget.items()+ if not candidates.isdisjoint(_raw_group_models(raw_budget_config))+ for budget_config in (_usable_group_budget_config(group_name, raw_budget_config),)+ if budget_config is not None+ )+ return groups if direct is None else (direct, *groups)+++def _raw_group_models(raw_budget_config: object) -> tuple[str, ...]:+ """The ``models`` an entry lists, read without validating the entry, so a request only validates its own groups."""+ if isinstance(raw_budget_config, BudgetConfig):+ return tuple(raw_budget_config.models or ())+ if not isinstance(raw_budget_config, Mapping):+ return ()+ raw_models: Final = cast(Mapping[str, object], raw_budget_config).get("models") # cast-ok: JSON object entry+ if not isinstance(raw_models, (list, tuple)):+ return ()+ return tuple(model for model in cast(Sequence[object], raw_models) if isinstance(model, str)) # cast-ok: JSON list+++def _usable_group_budget_config(group_name: str, raw_budget_config: object) -> BudgetConfig | None:+ budget_config: Final = _usable_budget_config(raw_budget_config)+ if budget_config is None:+ verbose_proxy_logger.warning(+ "Ignoring unusable model_max_budget group entry %s; it cannot be enforced or tracked",+ group_name,+ )+ return budget_config++ def team_model_budget_applies(model: str, key_model_max_budget: Mapping[str, object] | None) -> bool: """A key entry that spend-gates `model` overrides the team cap: it is then gated on and billed to the key alone.""" if not key_model_max_budget: return True- resolved: Final = resolve_model_budget(model=model, model_max_budget=key_model_max_budget)- return resolved is None or not _spend_gated(resolved.budget_config)+ return not any(+ _spend_gated(resolved.budget_config)+ for resolved in resolve_model_budgets(model=model, model_max_budget=key_model_max_budget)+ ) def _spend_gated(budget_config: BudgetConfig) -> bool:@@ -263,6 +321,8 @@ async def build_model_max_budget_usage( budget_duration=budget_config.budget_duration, now=current_time, ),+ # A model group's spend is the combined spend of every model it lists.+ **({"models": list(budget_config.models)} if budget_config.models is not None else {}), } for (budget_model, budget_config), window in zip(budgets, windows, strict=True) }@@ -338,13 +398,13 @@ def _resolve_entity_model_budgets( model: str, entity_budgets: Iterable[tuple[Litellm_EntityType, str | None, object]], ) -> tuple[tuple[Litellm_EntityType, str, ResolvedModelBudget], ...]:- """Drop the scopes that do not budget `model`, keeping only what can be incremented."""+ """Drop the scopes that do not budget `model`, keeping every budget (own entry and groups) to increment.""" return tuple( (entity_type, entity_id, resolved) for entity_type, entity_id, model_max_budget in entity_budgets if entity_id is not None and isinstance(model_max_budget, Mapping) and model_max_budget- for resolved in (resolve_model_budget(model=model, model_max_budget=model_max_budget),)- if resolved is not None and resolved.budget_config.budget_duration is not None+ for resolved in resolve_model_budgets(model=model, model_max_budget=model_max_budget)+ if resolved.budget_config.budget_duration is not None ) @@ -511,29 +571,33 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): ) -> bool: if not model_max_budget: return True- resolved: Final = resolve_model_budget(model=model, model_max_budget=model_max_budget)- if resolved is None:+ resolved_budgets: Final = resolve_model_budgets(model=model, model_max_budget=model_max_budget)+ if not resolved_budgets: verbose_proxy_logger.debug("Model %s not found in %s model_max_budget", model, entity_type.value) return True - max_budget: Final = resolved.budget_config.max_budget- if max_budget is None or max_budget < 0:- return True-- current_spend: Final = await self._get_spend_for_model_budget(- entity_type=entity_type,- entity_id=entity_id,- model=model,- resolved=resolved,- )- if current_spend >= max_budget:- raise litellm.BudgetExceededError(- message=exceeded_message,- current_cost=current_spend,- max_budget=max_budget,- entity_type=entity_type.value,+ for resolved in resolved_budgets:+ max_budget = resolved.budget_config.max_budget+ if max_budget is None or max_budget < 0:+ continue+ current_spend = await self._get_spend_for_model_budget(+ entity_type=entity_type, entity_id=entity_id,+ model=model,+ resolved=resolved, )+ if current_spend >= max_budget:+ raise litellm.BudgetExceededError(+ message=(+ f"{exceeded_message} (model group budget={resolved.budget_model})"+ if resolved.is_model_group+ else exceeded_message+ ),+ current_cost=current_spend,+ max_budget=max_budget,+ entity_type=entity_type.value,+ entity_id=entity_id,+ ) return True async def _get_spend_for_model_budget(@@ -562,7 +626,9 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): resolved=resolved, ) current_spend: Final = _as_spend(await self._cached_spend(spend_key))- if legacy_spend_key is None or legacy_spend_key == spend_key:+ # A legacy counter is one model's own spend; a group never had one, so+ # adding it would charge the group a model's pre-upgrade per-model spend.+ if legacy_spend_key is None or legacy_spend_key == spend_key or resolved.is_model_group: return current_spend return current_spend + _as_spend(await self._cached_spend(legacy_spend_key)) litellm/proxy/management_endpoints/key_management_endpoints.pyindex f7b72b263d..62b6f96a8b 100644@@ -8429,7 +8429,14 @@ def validate_model_max_budget(model_max_budget: dict | None) -> None: # /CRUD endpoints can pass budget_limit as a string, so we need to convert it to a float if "budget_limit" in _info: _info["budget_limit"] = float(_info["budget_limit"])- BudgetConfig(**_info)+ _budget_config = BudgetConfig(**_info)+ if _budget_config.models is not None and (+ _budget_config.tpm_limit is not None or _budget_config.rpm_limit is not None+ ):+ raise ValueError(+ f"model group budget {_model!r} cannot set tpm_limit/rpm_limit; "+ "set per-model rate limits on per-model entries instead"+ ) except Exception as e: raise ValueError( f"Invalid model_max_budget: {e}. Example of valid model_max_budget: https://docs.litellm.ai/docs/proxy/users"litellm/proxy/management_endpoints/team_endpoints.pyindex 1fa18e9b2b..e5a4922c49 100644@@ -1476,7 +1476,21 @@ def _check_team_model_budget_update_authority( requested: Final[Mapping[str, BudgetConfig]] = data.model_max_budget or {} for model_name, raw_existing in existing_model_max_budget.items(): existing = _existing_model_cap(raw_existing)- if existing is None or existing.max_budget is None or model_name in requested:+ if existing is None or existing.max_budget is None:+ continue+ if model_name in requested:+ # Dropping a model from a capped group frees it from the shared cap, which is a removal too.+ released = sorted(set(existing.models or ()) - set(requested[model_name].models or ()))+ if existing.models is not None and released:+ raise HTTPException(+ status_code=403,+ detail={+ "error": (+ f"Only a proxy admin can remove models {released} from a team's model group budget "+ f"{model_name!r}. Current max_budget={existing.max_budget}."+ )+ },+ ) continue raise HTTPException( status_code=403,@@ -1491,6 +1505,10 @@ def _check_team_model_budget_update_authority( governing = resolve_model_budget(model=model_name, model_max_budget=existing_model_max_budget) if governing is None: continue+ if proposed.models is not None and governing.budget_model != model_name:+ # A new model group's name is not a model, so no existing cap governs+ # it; it only adds a cap on top of the ones its models already have.+ continue cap = governing.budget_config if cap.max_budget is None: continuelitellm/types/utils.pyindex b21bf0e14d..a0f29851ec 100644@@ -43,9 +43,11 @@ from pydantic import ( GetCoreSchemaHandler, JsonValue, PrivateAttr,+ SerializerFunctionWrapHandler, SkipValidation, field_serializer, field_validator,+ model_serializer, ) from pydantic.main import IncEx from pydantic_core import core_schema@@ -4400,6 +4402,9 @@ class BudgetConfig(LiteLLMBaseModel): budget_duration: str | None = None tpm_limit: int | None = None rpm_limit: int | None = None+ # Makes the entry a model group budget: every model listed draws from one+ # shared cap, counted under the entry's own name rather than per model.+ models: list[str] | None = None def __init__(self, **data: Any) -> None: # Map time_period to budget_duration if present@@ -4412,6 +4417,28 @@ class BudgetConfig(LiteLLMBaseModel): super().__init__(**data) + @field_validator("models")+ @classmethod+ def _models_name_a_non_empty_group(cls, models: list[str] | None) -> list[str] | None:+ if models is None:+ return None+ if not models or any(not model.strip() for model in models):+ raise ValueError("models must be a non-empty list of non-empty model names")+ return models++ @model_serializer(mode="wrap")+ def _omit_absent_models( # noqa: ANN202 # annotating it replaces the model's serialization schema+ self, handler: SerializerFunctionWrapHandler+ ):+ # A per-model entry serializes exactly as it did before groups existed,+ # so stored rows and API responses for those entries do not change.+ # Read `models` off the output, not `self`: a field typed BudgetConfig+ # can hold a raw dict (model_construct), which reaches here as `self`.+ serialized: Final[object] = handler(self)+ if not isinstance(serialized, Mapping) or serialized.get("models") is not None:+ return serialized+ return {key: value for key, value in serialized.items() if key != "models"}+ GenericBudgetConfigType = dict[str, BudgetConfig] tests/unit/proxy/auth/test_auth_utils.pyindex 10a0f974a4..3018ec1260 100644@@ -717,6 +717,20 @@ class TestGetKeyModelRpmLimit: assert get_key_own_model_rate_limit(from_budget, "model_rpm_limit") == {"gpt-4": 10} assert get_key_own_model_rate_limit(from_budget, "model_tpm_limit") == {"gpt-3.5-turbo": 1000} + def test_a_model_group_budget_entry_is_not_a_per_model_rate_limit(self):+ key = UserAPIKeyAuth(+ api_key="sk-123",+ model_max_budget={+ "opus-family": {"rpm_limit": 5, "models": ["claude-opus-4-1", "claude-opus-4-5"]},+ "gpt-4": {"rpm_limit": 10},+ },+ )+ assert get_key_own_model_rate_limit(key, "model_rpm_limit") == {"gpt-4": 10}++ def test_a_per_model_entry_with_null_models_keeps_its_rate_limit(self):+ key = UserAPIKeyAuth(api_key="sk-123", model_max_budget={"gpt-4": {"rpm_limit": 10, "models": None}})+ assert get_key_own_model_rate_limit(key, "model_rpm_limit") == {"gpt-4": 10}+ def test_returns_key_metadata_when_present(self): """Key metadata takes priority over team metadata.""" user_api_key_dict = UserAPIKeyAuth(tests/unit/proxy/hooks/test_unit_test_max_model_budget_limiter.pyindex c6369a1f3d..52ad838b57 100644@@ -2066,3 +2066,223 @@ async def test_key_entry_without_a_spend_cap_does_not_lift_the_team_cap(key_entr key_model_max_budget=key_model_max_budget, model="openai/gpt-4", )+++_OPUS_GROUP_BUDGET: Final = MappingProxyType(+ {"opus-family": {"budget_limit": 1.0, "time_period": "30d", "models": ["claude-opus-4-1", "claude-opus-4-5"]}}+)+++async def _check_entity(limiter, entity_type, model_max_budget, model):+ if entity_type is Litellm_EntityType.KEY:+ return await limiter.is_key_within_model_budget(+ UserAPIKeyAuth(token="vk-hash", model_max_budget=model_max_budget), model+ )+ if entity_type is Litellm_EntityType.TEAM:+ return await limiter.is_team_within_model_budget(+ team_id="team-1", team_model_max_budget=model_max_budget, key_model_max_budget=None, model=model+ )+ if entity_type is Litellm_EntityType.USER:+ return await limiter.is_user_within_model_budget(+ user_id="user-1", user_model_max_budget=model_max_budget, model=model+ )+ return await limiter.is_end_user_within_model_budget(+ end_user_id="end-user-1", end_user_model_max_budget=model_max_budget, model=model+ )+++def _scope_kwargs(entity_type, model_max_budget):+ return {+ Litellm_EntityType.KEY: {"key_hash": "vk-hash", "key_model_max_budget": model_max_budget},+ Litellm_EntityType.TEAM: {"team_id": "team-1", "team_model_max_budget": model_max_budget},+ Litellm_EntityType.USER: {"user_id": "user-1", "user_model_max_budget": model_max_budget},+ Litellm_EntityType.END_USER: {"end_user_id": "end-user-1", "end_user_model_max_budget": model_max_budget},+ }[entity_type]+++_ENTITY_IDS: Final = MappingProxyType(+ {+ Litellm_EntityType.KEY: "vk-hash",+ Litellm_EntityType.TEAM: "team-1",+ Litellm_EntityType.USER: "user-1",+ Litellm_EntityType.END_USER: "end-user-1",+ }+)+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ "entity_type",+ [Litellm_EntityType.KEY, Litellm_EntityType.TEAM, Litellm_EntityType.USER, Litellm_EntityType.END_USER],+)+async def test_model_group_budget_is_one_combined_cap_across_its_models(entity_type):+ """+ Spend on one model of a group draws down the cap every other model in the+ group is gated on: two models each under the cap must not add up past it.+ """+ dual_cache = DualCache()+ limiter = PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=dual_cache)+ model_max_budget = dict(_OPUS_GROUP_BUDGET)++ await _log_success(+ limiter,+ model_group="claude-opus-4-1",+ response_cost=0.6,+ **_scope_kwargs(entity_type, model_max_budget),+ )+ assert await _check_entity(limiter, entity_type, model_max_budget, "anthropic/claude-opus-4-5") is True+ await _log_success(+ limiter,+ model_group="anthropic/claude-opus-4-5",+ response_cost=0.6,+ **_scope_kwargs(entity_type, model_max_budget),+ )++ for model in ("claude-opus-4-1", "claude-opus-4-5"):+ with pytest.raises(litellm.BudgetExceededError) as exc:+ await _check_entity(limiter, entity_type, model_max_budget, model)+ assert exc.value.entity_type == entity_type.value+ assert "model group budget=opus-family" in exc.value.message+ assert await _check_entity(limiter, entity_type, model_max_budget, "claude-sonnet-4-5") is True+ assert await build_model_max_budget_usage(+ entity_type=entity_type,+ entity_id=_ENTITY_IDS[entity_type],+ model_max_budget=model_max_budget,+ cache=dual_cache,+ ) == {+ "opus-family": {+ "current_spend": pytest.approx(1.2),+ "budget_limit": 1.0,+ "time_period": "30d",+ "budget_reset_at": await _reset_at_of_open_window(+ dual_cache, entity_type, _ENTITY_IDS[entity_type], "opus-family", "30d"+ ),+ "models": ["claude-opus-4-1", "claude-opus-4-5"],+ }+ }+++@pytest.mark.asyncio+async def test_team_member_model_group_budget_is_one_combined_cap():+ dual_cache = DualCache()+ limiter = PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=dual_cache)+ member_caps = dict(_OPUS_GROUP_BUDGET)+ kwargs = _success_kwargs(model_group="claude-opus-4-1", response_cost=1.5, user_id="user-1", team_id="team-1")+ kwargs["litellm_params"]["metadata"]["user_api_key_team_member_model_max_budget"] = member_caps++ await limiter.async_log_success_event(kwargs, response_obj=None, start_time=None, end_time=None)++ with pytest.raises(litellm.BudgetExceededError) as exc:+ await limiter.is_team_member_within_model_budget(+ user_id="user-1", team_id="team-1", team_member_model_max_budget=member_caps, model="claude-opus-4-5"+ )+ assert exc.value.entity_type == Litellm_EntityType.TEAM_MEMBER.value+++@pytest.mark.asyncio+async def test_a_model_under_its_own_cap_is_still_refused_once_its_group_is_spent():+ dual_cache = DualCache()+ limiter = PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=dual_cache)+ model_max_budget = {+ **_OPUS_GROUP_BUDGET,+ "claude-opus-4-5": {"budget_limit": 100.0, "time_period": "30d"},+ }++ await _log_success(+ limiter,+ model_group="claude-opus-4-5",+ response_cost=1.5,+ key_hash="vk-hash",+ key_model_max_budget=model_max_budget,+ )++ assert await dual_cache.async_get_cache(key="virtual_key_spend:vk-hash:claude-opus-4-5:30d") == 1.5+ assert await dual_cache.async_get_cache(key="virtual_key_spend:vk-hash:opus-family:30d") == 1.5+ with pytest.raises(litellm.BudgetExceededError) as exc:+ await limiter.is_key_within_model_budget(+ UserAPIKeyAuth(token="vk-hash", model_max_budget=model_max_budget), "claude-opus-4-5"+ )+ assert exc.value.max_budget == 1.0+++@pytest.mark.asyncio+async def test_a_key_model_group_budget_overrides_the_team_cap_for_its_models():+ dual_cache = DualCache()+ limiter = PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=dual_cache)+ team_model_max_budget = {"claude-opus-4-1": {"budget_limit": 1.0, "time_period": "30d"}}+ await dual_cache.async_set_cache(key="team_model_spend:team-1:claude-opus-4-1:30d", value=9.0)++ assert (+ await limiter.is_team_within_model_budget(+ team_id="team-1",+ team_model_max_budget=team_model_max_budget,+ key_model_max_budget=dict(_OPUS_GROUP_BUDGET),+ model="claude-opus-4-1",+ )+ is True+ )+++def test_a_per_model_budget_entry_serializes_without_a_models_field():+ from litellm.types.utils import BudgetConfig++ assert BudgetConfig(max_budget=1.0, budget_duration="1d").model_dump() == {+ "max_budget": 1.0,+ "budget_duration": "1d",+ "tpm_limit": None,+ "rpm_limit": None,+ }+ assert BudgetConfig(max_budget=1.0, budget_duration="1d", models=["a", "b"]).model_dump() == {+ "max_budget": 1.0,+ "budget_duration": "1d",+ "tpm_limit": None,+ "rpm_limit": None,+ "models": ["a", "b"],+ }+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ "models", [[], ["claude-opus-4-1", " "], "claude-opus-4-1"], ids=["empty", "blank_name", "not_a_list"]+)+async def test_a_malformed_model_group_is_not_enforced_or_charged(models):+ dual_cache = DualCache()+ limiter = PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=dual_cache)+ model_max_budget = {"opus-family": {"budget_limit": 0.0, "time_period": "30d", "models": models}}++ assert (+ await limiter.is_key_within_model_budget(+ UserAPIKeyAuth(token="vk-hash", model_max_budget=model_max_budget), "claude-opus-4-1"+ )+ is True+ )+ await _log_success(+ limiter, model_group="claude-opus-4-1", key_hash="vk-hash", key_model_max_budget=model_max_budget+ )+ assert await dual_cache.async_get_cache(key="virtual_key_spend:vk-hash:opus-family:30d") is None+++@pytest.mark.asyncio+async def test_model_group_named_like_a_model_does_not_capture_that_model():+ """+ A group governs only the models it lists: one named like a model must not+ gate that model, nor hide the model's own per-model cap.+ """+ dual_cache = DualCache()+ limiter = PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=dual_cache)+ model_max_budget = {+ "openai/gpt-4o": {"budget_limit": 100.0, "time_period": "1d", "models": ["o3"]},+ "gpt-4o": {"budget_limit": 1.0, "time_period": "1d"},+ }++ await _log_success(+ limiter,+ model_group="openai/gpt-4o",+ response_cost=1.5,+ key_hash="vk-hash",+ key_model_max_budget=model_max_budget,+ )++ with pytest.raises(litellm.BudgetExceededError):+ await _check_entity(limiter, Litellm_EntityType.KEY, model_max_budget, "openai/gpt-4o")+ assert await _check_entity(limiter, Litellm_EntityType.KEY, model_max_budget, "o3") is Truetests/unit/proxy/management_endpoints/test_key_management_endpoints.pyindex 9634f1efce..61953f6fdc 100644@@ -23372,3 +23372,36 @@ def test_key_info_database_not_connected_sanitized_500( assert "Database not connected" in caplog.text finally: proxy_server.app.dependency_overrides.pop(user_api_key_auth, None)+++@pytest.mark.parametrize(+ ("budget_entry", "error_fragment"),+ [+ ({"budget_limit": 10.0, "time_period": "30d", "models": []}, "non-empty list"),+ ({"budget_limit": 10.0, "time_period": "30d", "models": ["claude-opus-4-1"], "rpm_limit": 5}, "rpm_limit"),+ ],+ ids=["empty_group", "rate_limit_on_group"],+)+def test_validate_model_max_budget_rejects_an_unenforceable_model_group(budget_entry, error_fragment):+ from litellm.proxy.management_endpoints.key_management_endpoints import validate_model_max_budget++ with patch("litellm.proxy.proxy_server.premium_user", True), pytest.raises(ValueError, match=error_fragment):+ validate_model_max_budget({"opus-family": budget_entry})+++def test_validate_model_max_budget_accepts_a_model_group():+ from litellm.proxy.management_endpoints.key_management_endpoints import validate_model_max_budget++ with patch("litellm.proxy.proxy_server.premium_user", True):+ assert (+ validate_model_max_budget(+ {+ "opus-family": {+ "budget_limit": "10",+ "time_period": "30d",+ "models": ["claude-opus-4-1", "claude-opus-4-5"],+ }+ }+ )+ is None+ )tests/unit/proxy/management_endpoints/test_team_endpoints.pyindex f727448f3a..8a489f3702 100644@@ -17593,6 +17593,64 @@ def test_team_admin_can_tighten_or_keep_team_model_caps(requested) -> None: ) +_EXISTING_TEAM_GROUP_CAP: Final = MappingProxyType(+ {"opus-family": {"max_budget": 10.0, "budget_duration": "30d", "models": ["claude-opus-4-1", "claude-opus-4-5"]}}+)+++@pytest.mark.parametrize(+ ("requested_models", "allowed"),+ [+ (["claude-opus-4-1"], False),+ (["claude-opus-4-1", "claude-opus-4-5", "claude-opus-4-6"], True),+ (["claude-opus-4-5", "claude-opus-4-1"], True),+ ],+ ids=["drop_a_model_from_the_group", "add_a_model_to_the_group", "reorder"],+)+def test_team_admin_cannot_release_a_model_from_a_team_model_group_cap(requested_models, allowed) -> None:+ from litellm.proxy.management_endpoints.team_endpoints import _check_team_model_budget_update_authority++ def update() -> None:+ _check_team_model_budget_update_authority(+ data=UpdateTeamRequest(+ team_id="t1",+ model_max_budget={+ "opus-family": {"max_budget": 10.0, "budget_duration": "30d", "models": requested_models}+ },+ ),+ user_api_key_dict=UserAPIKeyAuth(user_role=LitellmUserRoles.INTERNAL_USER, user_id="team-admin"),+ existing_model_max_budget=_EXISTING_TEAM_GROUP_CAP,+ )++ if allowed:+ assert update() is None+ return+ with pytest.raises(HTTPException) as exc:+ update()+ assert exc.value.status_code == 403+ assert "claude-opus-4-5" in exc.value.detail["error"]+++def test_team_admin_can_add_a_model_group_named_like_a_capped_model() -> None:+ """A new group's name is not a model, so a per-model cap sharing that name must not govern it."""+ from litellm.proxy.management_endpoints.team_endpoints import _check_team_model_budget_update_authority++ assert (+ _check_team_model_budget_update_authority(+ data=UpdateTeamRequest(+ team_id="t1",+ model_max_budget={+ "gpt-4o": {"max_budget": 10.0, "budget_duration": "30d"},+ "openai/gpt-4o": {"max_budget": 1000.0, "budget_duration": "1d", "models": ["o3"]},+ },+ ),+ user_api_key_dict=UserAPIKeyAuth(user_role=LitellmUserRoles.INTERNAL_USER, user_id="team-admin"),+ existing_model_max_budget={"gpt-4o": {"max_budget": 10.0, "budget_duration": "30d"}},+ )+ is None+ )++ def test_team_model_cap_authority_skips_omitted_field_malformed_rows_and_proxy_admins() -> None: from litellm.proxy.management_endpoints.team_endpoints import _check_team_model_budget_update_authority
litellm/integrations/otel/model/spans.pyindex e423b6698e6..4907bd4368d 100644@@ -436,6 +436,7 @@ _POSTGRES_OPERATION_BY_CALL_TYPE: Final[Mapping[str, PostgresOperation]] = Mappi "reset_budget_windows": PostgresOperation("select", None), "write_budget_windows": PostgresOperation("update", None), "roll_window_spend_row": PostgresOperation("update", None),+ "open_window_spend_row": PostgresOperation("upsert", None), "seed_window_spend": PostgresOperation("select", None), "select_window_spend_rows": PostgresOperation("select", None), "commit_window_spend_updates": PostgresOperation("upsert", None),litellm/llms/anthropic/messages/caller_model_checks.pyindex 7eb6a8f9266..3822f075ccf 100644@@ -268,6 +268,7 @@ async def caller_within_model_budget( user_id=user_id, user_model_max_budget=user_model_max_budget, model=model,+ team_id=user_api_key_auth.team_id, ) except litellm.BudgetExceededError: return "denied"@@ -309,6 +310,7 @@ async def caller_within_model_budget( end_user_id=end_user_id, end_user_model_max_budget=end_user_model_max_budget, model=model,+ team_id=user_api_key_auth.team_id, ) except litellm.BudgetExceededError: return "denied"litellm/proxy/auth/user_api_key_auth.pyindex 99edfb940d7..79ad4824c92 100644@@ -370,7 +370,7 @@ class _KeyModelBudgetLimiter(Protocol): class _UserModelBudgetLimiter(Protocol): async def is_user_within_model_budget(- self, user_id: str, user_model_max_budget: Mapping[str, object], model: str+ self, user_id: str, user_model_max_budget: Mapping[str, object], model: str, team_id: str | None = None ) -> bool: ... @@ -388,11 +388,15 @@ class _EveryModelBudgetLimiter(Protocol): async def is_key_within_model_budget(self, user_api_key_dict: UserAPIKeyAuth, model: str) -> bool: ... async def is_user_within_model_budget(- self, user_id: str, user_model_max_budget: Mapping[str, object], model: str+ self, user_id: str, user_model_max_budget: Mapping[str, object], model: str, team_id: str | None = None ) -> bool: ... async def is_end_user_within_model_budget(- self, end_user_id: str, end_user_model_max_budget: Mapping[str, object], model: str+ self,+ end_user_id: str,+ end_user_model_max_budget: Mapping[str, object],+ model: str,+ team_id: str | None = None, ) -> bool: ... async def is_team_within_model_budget(@@ -503,6 +507,7 @@ async def _check_user_model_budget( user_id=valid_token.user_id, user_model_max_budget=user_model_max_budget, model=model_name,+ team_id=valid_token.team_id, ) @@ -547,6 +552,7 @@ async def _is_within_every_model_budget( user_id=valid_token.user_id, user_model_max_budget=user_model_max_budget, model=model,+ team_id=valid_token.team_id, ) if ( valid_token.end_user_id is not None@@ -557,6 +563,7 @@ async def _is_within_every_model_budget( end_user_id=valid_token.end_user_id, end_user_model_max_budget=end_user_model_max_budget, model=model,+ team_id=valid_token.team_id, ) if valid_token.team_id is not None and isinstance(team_model_max_budget, Mapping) and team_model_max_budget: await model_max_budget_limiter.is_team_within_model_budget(@@ -3154,6 +3161,7 @@ async def validate_resolved_virtual_key( # noqa: C901 # Preserve ordering of e end_user_id=valid_token.end_user_id, end_user_model_max_budget=end_user_mmb, model=model_name,+ team_id=valid_token.team_id, ) # Check 6: Additional Common Checks across jwt + key auth@@ -4733,6 +4741,7 @@ async def _run_post_custom_auth_checks( end_user_id=valid_token.end_user_id, end_user_model_max_budget=end_user_mmb, model=model_name,+ team_id=valid_token.team_id, ) # team / user / end_user / project context objects are fetched bylitellm/proxy/db/budget_window_spend_writer.pyindex 82c95cfe998..99b49331932 100644@@ -64,6 +64,16 @@ _ROLL_WINDOW_SPEND_SQL: Final = ( "AND window_start < ($4::timestamptz AT TIME ZONE 'UTC')" ) +_OPEN_WINDOW_SPEND_SQL: Final = (+ 'INSERT INTO "LiteLLM_BudgetWindowSpend" '+ "(entity_type, entity_id, window_duration, window_start, spend, created_at, updated_at) "+ "VALUES ($1, $2, $3, ($4::timestamptz AT TIME ZONE 'UTC'), 0, "+ "($4::timestamptz AT TIME ZONE 'UTC'), ($4::timestamptz AT TIME ZONE 'UTC')) "+ "ON CONFLICT (entity_type, entity_id, window_duration) DO UPDATE SET "+ "window_start = EXCLUDED.window_start, spend = 0, updated_at = EXCLUDED.updated_at "+ "WHERE \"LiteLLM_BudgetWindowSpend\".window_start <= ($5::timestamptz AT TIME ZONE 'UTC')"+)+ _SEED_FROM_SPEND_LOGS_KEY_SQL: Final = ( "SELECT COALESCE(SUM(spend), 0.0) AS total, " "COALESCE(SUM(spend) FILTER (WHERE \"startTime\" < ($3::timestamptz AT TIME ZONE 'UTC')), 0.0) AS before_batch "@@ -389,3 +399,29 @@ async def roll_window_spend_row( to_naive_utc(new_window_start), to_naive_utc(datetime.now(timezone.utc)), )+++async def open_window_spend_row(+ prisma_client: "PrismaClient",+ entity_type: str,+ entity_id: str,+ window_duration: str,+ window_start: datetime,+ replaces_window_start: datetime | None,+) -> None:+ """Start a window at `window_start` unless another writer already started one.++ The row is created when it does not exist, and moved onto the new window only+ while it still holds `replaces_window_start`, the ended window the caller saw.+ A writer that lost the race leaves the winner's window in place, so reading+ the row back gives every writer the same window start.+ """+ async with db_span("open_window_spend_row", "LiteLLM_BudgetWindowSpend"):+ await prisma_client.db.execute_raw(+ _OPEN_WINDOW_SPEND_SQL,+ entity_type,+ entity_id,+ window_duration,+ to_naive_utc(window_start),+ None if replaces_window_start is None else to_naive_utc(replaces_window_start),+ )litellm/proxy/hooks/model_budget_shared_windows.pynew file mode 100644index 00000000000..880dd0eb218@@ -0,0 +1,176 @@+from collections.abc import Awaitable, Callable+from dataclasses import dataclass+from datetime import datetime, timezone+from typing import TYPE_CHECKING, Final, Protocol, TypeAlias++from litellm._logging import verbose_proxy_logger+from litellm.caching.in_memory_cache import InMemoryCache+from litellm.proxy.db.budget_window_spend_writer import open_window_spend_row+from litellm.proxy.db.db_transaction_queue.window_spend_update_queue import (+ WindowSpendTransaction,+ build_window_spend_transaction,+)+from litellm.repositories.table_repositories import BudgetWindowSpendRepository+from litellm.router_strategy.budget_limiter import budget_window_seconds++if TYPE_CHECKING:+ from litellm.proxy.utils import PrismaClient++MODEL_BUDGET_WINDOW_ENTITY_TYPE: Final = "model_max_budget"+_READ_TTL_SECONDS: Final = 5+++@dataclass(frozen=True, slots=True)+class SharedWindow:+ started_at: float+ spend: float++ def is_open(self, budget_duration: str, now: float) -> bool:+ return now < self.started_at + budget_window_seconds(self.started_at, budget_duration)+++class LocalTTLCache(Protocol):+ def get_cache(self, key: str) -> object: ...++ def set_cache(self, key: str, value: object, *, ttl: float) -> None: ...++ def delete_cache(self, key: str) -> None: ...+++class SharedWindows(Protocol):+ async def read(self, spend_key: str, budget_duration: str) -> SharedWindow | None: ...++ async def open(self, spend_key: str, budget_duration: str, now: float) -> float | None: ...++ async def add(self, spend_key: str, budget_duration: str, started_at: float, spend: float) -> None: ...+++PrismaClientSource: TypeAlias = Callable[[], "PrismaClient | None"]+WindowSpendSink: TypeAlias = Callable[[WindowSpendTransaction], Awaitable[None]]+++def _proxy_prisma_client() -> "PrismaClient | None":+ try:+ from litellm.proxy.proxy_server import prisma_client+ except ImportError:+ return None+ return prisma_client+++async def _queue_on_proxy_spend_writer(transaction: WindowSpendTransaction) -> None:+ from litellm.proxy.proxy_server import proxy_logging_obj++ await proxy_logging_obj.db_spend_update_writer.window_spend_update_queue.add_update(transaction)+++@dataclass(frozen=True, slots=True)+class _Reading:+ window: SharedWindow | None+++class DatabaseSharedWindows:+ def __init__(+ self,+ prisma_client_source: PrismaClientSource = _proxy_prisma_client,+ sink: WindowSpendSink = _queue_on_proxy_spend_writer,+ read_ttl_seconds: int = _READ_TTL_SECONDS,+ ) -> None:+ self._prisma_client_source = prisma_client_source+ self._sink = sink+ self._read_ttl_seconds = read_ttl_seconds+ self._readings: LocalTTLCache = InMemoryCache(max_size_in_memory=10_000, default_ttl=read_ttl_seconds)++ async def read(self, spend_key: str, budget_duration: str) -> SharedWindow | None:+ cached: Final = self._readings.get_cache(key=spend_key)+ if isinstance(cached, _Reading):+ return cached.window+ window: Final = await self._read_row(spend_key=spend_key, budget_duration=budget_duration)+ self._readings.set_cache(key=spend_key, value=_Reading(window=window), ttl=self._read_ttl_seconds)+ return window++ async def open(self, spend_key: str, budget_duration: str, now: float) -> float | None:+ """When the window every replica shares started, opening one at `now` if none is open.++ Without a database every window is this replica's own, so it starts at `now`. None means the database did+ not answer, so this replica cannot know which window the others are in yet.+ """+ prisma_client: Final = self._prisma_client_source()+ if prisma_client is None:+ return now+ try:+ seen: Final = await _fetch_row(+ prisma_client=prisma_client, spend_key=spend_key, budget_duration=budget_duration, use_writer=True+ )+ if seen is not None and seen.is_open(budget_duration=budget_duration, now=now):+ return seen.started_at+ await open_window_spend_row(+ prisma_client=prisma_client,+ entity_type=MODEL_BUDGET_WINDOW_ENTITY_TYPE,+ entity_id=spend_key,+ window_duration=budget_duration,+ window_start=datetime.fromtimestamp(now, tz=timezone.utc),+ replaces_window_start=None+ if seen is None+ else datetime.fromtimestamp(seen.started_at, tz=timezone.utc),+ )+ opened: Final = await _fetch_row(+ prisma_client=prisma_client, spend_key=spend_key, budget_duration=budget_duration, use_writer=True+ )+ except Exception: # noqa: BLE001 # an unreachable database leaves this replica on its own window+ verbose_proxy_logger.exception("Unable to open the shared model budget window")+ return None+ if opened is None or not opened.is_open(budget_duration=budget_duration, now=now):+ return None+ return opened.started_at++ async def add(self, spend_key: str, budget_duration: str, started_at: float, spend: float) -> None:+ if self._prisma_client_source() is None:+ return+ try:+ await self._sink(+ build_window_spend_transaction(+ entity_type=MODEL_BUDGET_WINDOW_ENTITY_TYPE,+ entity_id=spend_key,+ window_duration=budget_duration,+ window_start=datetime.fromtimestamp(started_at, tz=timezone.utc),+ spend=spend,+ )+ )+ except Exception: # noqa: BLE001 # spend tracking must never fail the success callback+ verbose_proxy_logger.debug("Unable to queue shared model budget spend", exc_info=True)++ async def _read_row(self, spend_key: str, budget_duration: str) -> SharedWindow | None:+ prisma_client: Final = self._prisma_client_source()+ if prisma_client is None:+ return None+ try:+ return await _fetch_row(+ prisma_client=prisma_client, spend_key=spend_key, budget_duration=budget_duration, use_writer=False+ )+ except Exception: # noqa: BLE001 # an unreadable row leaves enforcement on this replica's own counter+ verbose_proxy_logger.exception("Unable to read shared model budget spend")+ return None+++async def _fetch_row(+ prisma_client: "PrismaClient", spend_key: str, budget_duration: str, use_writer: bool+) -> SharedWindow | None:+ row: Final = await BudgetWindowSpendRepository(prisma_client, use_writer=use_writer).table.find_unique(+ where={+ "entity_type_entity_id_window_duration": {+ "entity_type": MODEL_BUDGET_WINDOW_ENTITY_TYPE,+ "entity_id": spend_key,+ "window_duration": budget_duration,+ }+ }+ )+ if row is None:+ return None+ window_start: Final = row.window_start+ started_at: Final = (+ window_start if window_start.tzinfo is not None else window_start.replace(tzinfo=timezone.utc)+ ).timestamp()+ return SharedWindow(started_at=started_at, spend=float(row.spend or 0.0))+++PROXY_SHARED_WINDOWS: Final = DatabaseSharedWindows()litellm/proxy/hooks/model_max_budget_limiter.pyindex bc9cee1cb1d..11a47b2d424 100644@@ -1,11 +1,18 @@ import asyncio+import functools+import itertools import json import time-from collections.abc import Callable, Iterable, Mapping, Sequence+from collections.abc import Callable, Collection, Iterable, Iterator, Mapping, Sequence from dataclasses import dataclass from datetime import datetime, timezone from types import MappingProxyType-from typing import Final, cast # noqa: TID251 # untyped callback payload requires explicit narrowing+from typing import (+ TYPE_CHECKING,+ Final,+ TypeAlias,+ cast, # noqa: TID251 # untyped callback payload requires explicit narrowing+) import litellm from litellm._internal_context import with_service_target@@ -14,12 +21,21 @@ from litellm.caching.caching import DualCache from litellm.integrations.custom_logger import Span from litellm.litellm_core_utils.duration_parser import duration_in_seconds from litellm.litellm_core_utils.litellm_logging import StandardLoggingPayloadSetup-from litellm.llms.bedrock.common_utils import get_bedrock_base_model+from litellm.llms.bedrock.common_utils import get_bedrock_base_model, get_bedrock_cross_region_inference_regions from litellm.proxy._types import Litellm_EntityType, UserAPIKeyAuth+from litellm.proxy.hooks.model_budget_shared_windows import (+ PROXY_SHARED_WINDOWS,+ LocalTTLCache,+ SharedWindow,+ SharedWindows,+) from litellm.router_strategy.budget_limiter import RouterBudgetLimiting, budget_window_seconds from litellm.router_utils.batch_utils import batch_charged_once_marker_key, batch_id_to_charge_once from litellm.types.llms.openai import AllMessageValues-from litellm.types.utils import BudgetConfig, StandardLoggingPayload+from litellm.types.utils import BudgetConfig, LlmProviders, StandardLoggingPayload++if TYPE_CHECKING:+ from litellm.router import Router VIRTUAL_KEY_SPEND_CACHE_KEY_PREFIX: Final = "virtual_key_spend" END_USER_SPEND_CACHE_KEY_PREFIX: Final = "end_user_model_spend"@@ -39,10 +55,18 @@ _SPEND_CACHE_KEY_PREFIXES: Final = MappingProxyType( _LEGACY_REQUEST_MODEL_SCOPES: Final = frozenset({Litellm_EntityType.KEY, Litellm_EntityType.END_USER}) +_PER_REPLICA_SCOPES: Final = frozenset({Litellm_EntityType.END_USER})+ _PROCESS_STARTED_AT: Final = time.monotonic() _LATEST_RESET_AT: Final = datetime(9999, 12, 31, tzinfo=timezone.utc).timestamp() +_GLOB_WILDCARD: Final = "*"++ModelAccessGroups: TypeAlias = Callable[[str, str | None], Collection[str]]++_PROVIDER_PREFIXES: Final = tuple(f"{provider.value}/" for provider in LlmProviders)+ _BUDGET_START_TIME_KEY_PREFIXES: Final = MappingProxyType( { Litellm_EntityType.KEY: "virtual_key_budget_start_time",@@ -68,6 +92,15 @@ class ResolvedModelBudget: budget_config: BudgetConfig +@dataclass(frozen=True, slots=True)+class RoutedModels:+ served_names: frozenset[str]+ group_members: Mapping[str, frozenset[str]]++ def members(self, access_group: str) -> frozenset[str]:+ return self.group_members.get(access_group, frozenset())++ def team_member_budget_entity_id(user_id: str, team_id: str) -> str: return f"{len(user_id)}:{user_id}:{team_id}" @@ -131,43 +164,403 @@ def model_budget_start_time_cache_key( return f"{_BUDGET_START_TIME_KEY_PREFIXES[entity_type]}:{entity_id}:{budget_model}:{budget_duration}" -def resolve_model_budget(model: str, model_max_budget: Mapping[str, object]) -> ResolvedModelBudget | None:- """Find the `model_max_budget` entry that governs `model`, or None."""- for candidate in _budget_model_candidates(model):- raw_budget_config = model_max_budget.get(candidate)- if raw_budget_config is None:- continue- if (budget_config := _usable_budget_config(raw_budget_config)) is None:- # An entry that will not validate cannot be keyed, so it cannot be- # enforced or incremented. Skip to the next candidate rather than- # raising: raising would abort every other scope's increment and turn- # a config typo into a 500, and stopping here would let one malformed- # specific entry disable a perfectly good bare-family budget beside- # it. The candidate chain already falls through an ABSENT entry, and- # an unparseable one is indistinguishable from absent to enforcement.- # `validate_model_max_budget` rejects these on the write path, so- # reaching here means config.yaml or a direct DB edit.- verbose_proxy_logger.warning(- "Ignoring unusable model_max_budget entry for %s; it cannot be enforced or tracked",- candidate,- )- continue- return ResolvedModelBudget(budget_model=candidate, budget_config=budget_config)- return None+def model_access_groups_of(router: "Router", model: str, team_id: str | None = None) -> frozenset[str]:+ """Access groups serving `model`, or the team's own model of that name, leaving out any named after a model the+ router serves.++ An entry with that name already budgets that model, so reading it as the group too would quietly widen it.+ """+ return frozenset(+ group+ for group in router.get_model_access_groups(model_name=model, team_id=team_id)+ if group not in router.model_names+ )+++def routed_models_of(router: "Router", team_id: str | None = None) -> RoutedModels:+ serves_every_name: Final = (+ router.default_deployment is not None or router.router_general_settings.pass_through_all_models+ )+ served_names: Final = frozenset(router.model_names) | (+ frozenset({_GLOB_WILDCARD}) if serves_every_name else frozenset()+ )+ team_members: Final = _team_public_names_by_group(router=router, team_id=team_id)+ return RoutedModels(+ served_names=served_names,+ group_members=MappingProxyType(+ {+ group: frozenset(members) | team_members.get(group, frozenset())+ for group, members in router.get_model_access_groups().items()+ if group not in served_names+ }+ ),+ )+++def _team_public_names_by_group(router: "Router", team_id: str | None) -> Mapping[str, frozenset[str]]:+ if team_id is None:+ return MappingProxyType({})+ groups_by_name: Final = MappingProxyType(+ {+ name: model_access_groups_of(router=router, model=name, team_id=team_id)+ for name in router.team_public_model_names_of(team_id)+ }+ )+ return MappingProxyType(+ {+ group: frozenset(name for name, groups in groups_by_name.items() if group in groups)+ for group in frozenset[str]().union(*groups_by_name.values())+ }+ )+++def proxy_model_access_groups(model: str, team_id: str | None = None) -> frozenset[str]:+ router: Final = _proxy_router()+ return frozenset() if router is None else model_access_groups_of(router=router, model=model, team_id=team_id)+++def proxy_routed_models(team_id: str | None = None) -> RoutedModels:+ router: Final = _proxy_router()+ if router is None:+ return RoutedModels(served_names=frozenset(), group_members=MappingProxyType({}))+ return routed_models_of(router=router, team_id=team_id)+++def _proxy_router() -> "Router | None":+ try:+ from litellm.proxy.proxy_server import llm_router+ except ImportError:+ return None+ return llm_router+++def resolve_model_budget(+ model: str,+ model_max_budget: Mapping[str, object],+ model_access_groups: ModelAccessGroups = proxy_model_access_groups,+ team_id: str | None = None,+) -> ResolvedModelBudget | None:+ """Find the `model_max_budget` entry that governs `model`, or None.++ An entry named after the model wins, then one named after a model access+ group serving it, then the most specific glob (`claude-opus*`). Every model+ an access group or glob entry covers draws from that entry's one counter.+ """+ resolved: Final = (+ _usable_entry(entry=entry, raw_budget_config=model_max_budget[entry])+ for entry in _governing_entry_names(+ model=model, model_max_budget=model_max_budget, model_access_groups=model_access_groups, team_id=team_id+ )+ )+ return next((entry for entry in resolved if entry is not None), None)+++def _governing_entry_names(+ model: str,+ model_max_budget: Mapping[str, object],+ model_access_groups: ModelAccessGroups,+ team_id: str | None,+) -> Iterator[str]:+ candidates: Final = _budget_model_candidates(model)+ yield from (candidate for candidate in candidates if candidate in model_max_budget)+ group_entries: Final = tuple(+ sorted(entry for entry in model_max_budget if _GLOB_WILDCARD not in entry and entry not in candidates)+ )+ if group_entries:+ groups: Final = model_access_groups(model, team_id)+ yield from (entry for entry in group_entries if entry in groups)+ yield from sorted(+ (entry for entry in model_max_budget if _GLOB_WILDCARD in entry and _glob_matches(entry, candidates)),+ key=_glob_precedence,+ reverse=True,+ )+++def _glob_matches(pattern: str, names: Iterable[str]) -> bool:+ return any(_glob_matches_name(pattern=pattern, name=name) for name in names)+++def _glob_matches_name(pattern: str, name: str) -> bool:+ """Whole-name match where `*` is the only wildcard, in linear time whatever the pattern holds.++ Taking each literal piece between two wildcards at its first occurrence is+ always safe, since a later occurrence only leaves less room for the rest, so+ the match never needs to backtrack the way a `.*` regex does.+ """+ if _GLOB_WILDCARD not in pattern:+ return pattern == name+ head, *middle, tail = pattern.split(_GLOB_WILDCARD)+ middle_end: Final = len(name) - len(tail)+ if middle_end < len(head) or not name.startswith(head) or not name.endswith(tail):+ return False+ return functools.reduce(_after_piece(name=name, end=middle_end), middle, len(head)) >= 0+++def _after_piece(name: str, end: int) -> Callable[[int, str], int]:+ def step(position: int, piece: str) -> int:+ found: Final = name.find(piece, position, end) if position >= 0 else -1+ return found + len(piece) if found >= 0 else -1++ return step+++def _glob_precedence(pattern: str) -> tuple[int, int, str]:+ """More literal characters first, then fewer wildcards, so padding a pattern with `*` never makes it win."""+ wildcards: Final = pattern.count(_GLOB_WILDCARD)+ return len(pattern) - wildcards, -wildcards, pattern+++@dataclass(frozen=True, slots=True)+class _RequestForm:+ """Request names that an entry governs.++ Every such name matches all of ``name_globs`` and also resolves through+ ``candidate_globs``. Its last ``/`` segment is among them unless+ ``last_segment_open``.+ """++ name_globs: tuple[str, ...]+ candidate_globs: tuple[str, ...] = ()+ last_segment_open: bool = False+++def entries_outranked_by(entry: str, model_max_budget: Collection[str], routed_models: RoutedModels) -> Iterator[str]:+ """Entries that `entry` can take precedence over for some model a request can name.++ A model or group entry is resolved before any pattern, and a pattern before+ every less specific one, so `entry` outranks a pattern that some request it+ governs also resolves to. A group entry with a wildcard route among its+ members outranks every pattern.+ """+ patterns: Final = tuple(other for other in model_max_budget if _GLOB_WILDCARD in other)+ if _GLOB_WILDCARD in entry:+ entry_precedence: Final = _glob_precedence(entry)+ pattern_forms: Final = tuple(_pattern_request_forms(pattern=entry, routed_models=routed_models))+ yield from (+ other+ for other in patterns+ if _glob_precedence(other) < entry_precedence and _resolves_any(pattern=other, forms=pattern_forms)+ )+ return+ entry_members: Final = routed_models.members(entry)+ covers_wildcard_route: Final = any(_GLOB_WILDCARD in member for member in entry_members)+ entry_forms: Final = tuple(+ _entry_request_forms(entry=entry, entry_members=entry_members, routed_models=routed_models)+ )+ yield from (other for other in patterns if covers_wildcard_route or _resolves_any(pattern=other, forms=entry_forms))+ yield from (+ other+ for other in model_max_budget+ if _GLOB_WILDCARD not in other+ and _ranks_ahead_of_group(+ entry=entry, entry_members=entry_members, group=other, group_members=routed_models.members(other)+ )+ )+++def _resolves_any(pattern: str, forms: Iterable[_RequestForm]) -> bool:+ return any(_resolves(pattern=pattern, form=form) for form in forms)+++def _resolves(pattern: str, form: _RequestForm) -> bool:+ return (+ _globs_share_a_name(pattern, *form.name_globs)+ or any(_globs_share_a_name(pattern, candidate) for candidate in form.candidate_globs)+ or (form.last_segment_open and "/" not in pattern and _globs_share_a_name(f"*/{pattern}", *form.name_globs))+ )+++def _globs_share_a_name(*globs: str) -> bool:+ """Whether one name matches every glob.++ Globs that each hold a `*` share a name exactly when their literal heads+ are prefixes of one another and so are their literal tails: the longest+ head, every middle piece and the longest tail, in that order, matches all+ of them.+ """+ literal: Final = next((glob for glob in globs if _GLOB_WILDCARD not in glob), None)+ if literal is not None:+ return all(_glob_matches_name(pattern=glob, name=literal) for glob in globs)+ heads: Final = sorted((glob.partition(_GLOB_WILDCARD)[0] for glob in globs), key=len)+ tails: Final = sorted((glob.rpartition(_GLOB_WILDCARD)[2] for glob in globs), key=len)+ return all(longer.startswith(shorter) for shorter, longer in itertools.pairwise(heads)) and all(+ longer.endswith(shorter) for shorter, longer in itertools.pairwise(tails)+ )+++def _served_form(name: str) -> _RequestForm:+ return _RequestForm(name_globs=(name,), candidate_globs=_budget_model_candidates(name))+++def _wildcard_routes(routed_models: RoutedModels) -> tuple[str, ...]:+ return tuple(sorted(name for name in routed_models.served_names if _GLOB_WILDCARD in name))+++def _entry_request_forms(+ entry: str, entry_members: frozenset[str], routed_models: RoutedModels+) -> Iterator[_RequestForm]:+ yield from (+ _served_form(name)+ for name in _names_governed_by(entry=entry, entry_members=entry_members, routed_models=routed_models)+ )+ routes: Final = _wildcard_routes(routed_models)+ yield from (+ _RequestForm(name_globs=(route, f"*/{entry}"), candidate_globs=_budget_model_candidates(entry))+ for route in routes+ )+ yield from _bedrock_request_forms(+ bedrock_models=tuple(model for model in _bedrock_base_models() if entry in _bedrock_names(model)),+ routes=routes,+ )+++def _pattern_request_forms(pattern: str, routed_models: RoutedModels) -> Iterator[_RequestForm]:+ yield _RequestForm(name_globs=(pattern,))+ yield from (+ _served_form(name)+ for name in routed_models.served_names+ if _GLOB_WILDCARD not in name and _glob_matches(pattern, _budget_model_candidates(name))+ )+ routes: Final = _wildcard_routes(routed_models)+ yield from (_RequestForm(name_globs=(route, pattern), last_segment_open=True) for route in routes)+ if "/" not in pattern:+ yield from (_RequestForm(name_globs=(route, f"*/{pattern}"), candidate_globs=(pattern,)) for route in routes)+ yield from (+ _narrowed_to(pattern=pattern, form=form)+ for form in _bedrock_request_forms(bedrock_models=_bedrock_base_models(), routes=routes)+ if _resolves(pattern=pattern, form=form)+ )+++def _narrowed_to(pattern: str, form: _RequestForm) -> _RequestForm:+ if any(_globs_share_a_name(pattern, candidate) for candidate in form.candidate_globs):+ return form+ return _RequestForm(name_globs=(*form.name_globs, pattern), candidate_globs=form.candidate_globs)+++def _bedrock_request_forms(bedrock_models: Iterable[str], routes: Sequence[str]) -> Iterator[_RequestForm]:+ """Wildcard-routed request names that resolve to a Bedrock model: its id, under a cross-region prefix or with a+ context or throughput suffix, as the whole name or after any routing prefix, region path or ARN."""+ forms: Final = itertools.chain.from_iterable(+ _bedrock_model_request_forms(model=model, routes=routes) for model in bedrock_models+ )+ yield from (form for form in forms if _globs_share_a_name(*form.name_globs))+++def _bedrock_model_request_forms(model: str, routes: Sequence[str]) -> Iterator[_RequestForm]:+ names: Final = _bedrock_names(model)+ return (+ _RequestForm(name_globs=(route, f"{prefix}{last_segment}"), candidate_globs=(last_segment, *names))+ for route, (last_segment, prefix) in itertools.product(+ routes, itertools.product(_bedrock_last_segments(model), ("", "*/"))+ )+ )+++def _bedrock_last_segments(model: str) -> tuple[str, ...]:+ regioned: Final = (model, *(f"{region}.{model}" for region in get_bedrock_cross_region_inference_regions()))+ _, version_separator, version = model.rpartition(":")+ suffixes: Final = ("", "[*]", *((":*k",) if version_separator and version.isdigit() else ()))+ return tuple(f"{name}{suffix}" for name, suffix in itertools.product(regioned, suffixes))+++def _model_cost() -> Mapping[str, object]:+ return litellm.model_cost+++def _bedrock_base_models() -> tuple[str, ...]:+ return tuple(+ model+ for model, cost_entry in _model_cost().items()+ if _priced_as_bedrock(cost_entry) and get_bedrock_base_model(model) == model+ )+++def _names_governed_by(entry: str, entry_members: frozenset[str], routed_models: RoutedModels) -> Iterator[str]:+ yield entry+ yield from (f"{prefix}{entry}" for prefix in _PROVIDER_PREFIXES)+ yield from (+ name+ for name in routed_models.served_names+ if _GLOB_WILDCARD not in name and entry in _budget_model_candidates(name)+ )+ yield from (member for member in entry_members if _GLOB_WILDCARD not in member)+++def _ranks_ahead_of_group(entry: str, entry_members: frozenset[str], group: str, group_members: Iterable[str]) -> bool:+ return any(+ _GLOB_WILDCARD in member+ or entry in _budget_model_candidates(member)+ or (member in entry_members and entry < group)+ for member in group_members+ ) -def team_model_budget_applies(model: str, key_model_max_budget: Mapping[str, object] | None) -> bool:- """A key entry that spend-gates `model` overrides the team cap: it is then gated on and billed to the key alone."""+def _usable_entry(entry: str, raw_budget_config: object) -> ResolvedModelBudget | None:+ if (budget_config := _usable_budget_config(raw_budget_config)) is None:+ # An entry that will not validate cannot be keyed, so it cannot be+ # enforced or incremented. Skip to the next candidate rather than+ # raising: raising would abort every other scope's increment and turn+ # a config typo into a 500, and stopping here would let one malformed+ # specific entry disable a perfectly good bare-family budget beside+ # it. The candidate chain already falls through an ABSENT entry, and+ # an unparseable one is indistinguishable from absent to enforcement.+ # `validate_model_max_budget` rejects these on the write path, so+ # reaching here means config.yaml or a direct DB edit.+ verbose_proxy_logger.warning(+ "Ignoring unusable model_max_budget entry for %s; it cannot be enforced or tracked",+ entry,+ )+ return None+ return ResolvedModelBudget(budget_model=entry, budget_config=budget_config)+++def team_model_budget_entries(+ model: str, team_model_max_budget: Mapping[str, object], key_model_max_budget: Mapping[str, object] | None+) -> Mapping[str, object]:+ """The team entries that budget `model` for this key.++ A key entry named after `model` that spend-gates it replaces the team entry named after it, so that model is+ gated on and billed to the key's entry. The team's pattern and group entries still apply beside it.+ """+ exact_candidates: Final = _budget_model_candidates(model)+ if not _key_entry_replaces_team_entry(+ model=model, exact_candidates=exact_candidates, key_model_max_budget=key_model_max_budget+ ):+ return team_model_max_budget+ return MappingProxyType(+ {entry: config for entry, config in team_model_max_budget.items() if entry not in exact_candidates}+ )+++def _key_entry_replaces_team_entry(+ model: str, exact_candidates: tuple[str, ...], key_model_max_budget: Mapping[str, object] | None+) -> bool: if not key_model_max_budget:- return True- resolved: Final = resolve_model_budget(model=model, model_max_budget=key_model_max_budget)- return resolved is None or not _spend_gated(resolved.budget_config)+ return False+ resolved: Final = resolve_model_budget(+ model=model,+ model_max_budget=MappingProxyType(+ {+ candidate: key_model_max_budget[candidate]+ for candidate in exact_candidates+ if candidate in key_model_max_budget+ }+ ),+ model_access_groups=lambda _model, _team_id: frozenset(),+ )+ return resolved is not None and _spend_gated(resolved.budget_config) def _spend_gated(budget_config: BudgetConfig) -> bool: return budget_config.max_budget is not None and budget_config.max_budget >= 0 +def entry_names_model(entry: str, model: str) -> bool:+ """Whether `entry` budgets `model` under the model's own name, rather than as a pattern or an access group."""+ return entry in _budget_model_candidates(model)++ def _budget_model_candidates(model: str) -> tuple[str, ...]: """Names a budget may be configured under for a request on `model`, most specific first. @@ -189,13 +582,18 @@ def _bedrock_candidates(model: str) -> tuple[str, ...]: of those would produce a garbage candidate. """ base_model: Final = get_bedrock_base_model(model)- cost_entry: Final = litellm.model_cost.get(base_model)- if not isinstance(cost_entry, dict) or not str(cost_entry.get("litellm_provider", "")).startswith("bedrock"):- return ()+ return _bedrock_names(base_model) if _priced_as_bedrock(_model_cost().get(base_model)) else ()+++def _bedrock_names(base_model: str) -> tuple[str, ...]: _, _, without_vendor = base_model.partition(".") return (base_model, without_vendor) if without_vendor else (base_model,) +def _priced_as_bedrock(cost_entry: object) -> bool:+ return str(_metadata_value(cost_entry, "litellm_provider") or "").startswith("bedrock")++ @dataclass(frozen=True, slots=True) class _WindowReading: current_spend: float@@ -208,6 +606,7 @@ async def build_model_max_budget_usage( model_max_budget: Mapping[str, object] | None, cache: DualCache | None, now: datetime | None = None,+ shared_windows: SharedWindows = PROXY_SHARED_WINDOWS, ) -> dict[str, dict[str, object]]: """Current-window spend per configured budget model, as `/key/info` and `/user/info` report it. @@ -231,17 +630,18 @@ async def build_model_max_budget_usage( ) if not budgets: return {}+ spend_keys: Final = tuple(+ model_budget_spend_cache_key(+ entity_type=entity_type,+ entity_id=entity_id,+ budget_model=budget_model,+ budget_duration=budget_config.budget_duration,+ )+ for budget_model, budget_config in budgets+ ) windows: Final = await _read_windows( cache=cache,- spend_keys=tuple(- model_budget_spend_cache_key(- entity_type=entity_type,- entity_id=entity_id,- budget_model=budget_model,- budget_duration=budget_config.budget_duration,- )- for budget_model, budget_config in budgets- ),+ spend_keys=spend_keys, start_time_keys=tuple( model_budget_start_time_cache_key( entity_type=entity_type,@@ -253,21 +653,57 @@ async def build_model_max_budget_usage( ), ) current_time: Final = (now or datetime.now(timezone.utc)).timestamp()+ shared: Final = tuple(+ [+ (+ await _open_shared_window(+ shared_windows=shared_windows,+ spend_key=spend_key,+ budget_duration=budget_config.budget_duration,+ now=current_time,+ )+ if cache.redis_cache is None+ else None+ )+ for spend_key, (_, budget_config) in zip(spend_keys, budgets, strict=True)+ ]+ )+ readings: Final = tuple(+ _fuller_reading(local=window, shared=shared_window)+ for window, shared_window in zip(windows, shared, strict=True)+ ) return { budget_model: {- "current_spend": round(window.current_spend, 4),+ "current_spend": round(reading.current_spend, 4), "budget_limit": budget_config.max_budget, "time_period": budget_config.budget_duration, "budget_reset_at": _window_reset_at(- started_at=window.started_at,+ started_at=reading.started_at, budget_duration=budget_config.budget_duration, now=current_time, ), }- for (budget_model, budget_config), window in zip(budgets, windows, strict=True)+ for (budget_model, budget_config), reading in zip(budgets, readings, strict=True) } +async def _open_shared_window(+ shared_windows: SharedWindows, spend_key: str, budget_duration: str | None, now: float+) -> SharedWindow | None:+ if budget_duration is None:+ return None+ window: Final = await shared_windows.read(spend_key=spend_key, budget_duration=budget_duration)+ if window is None or not window.is_open(budget_duration=budget_duration, now=now):+ return None+ return window+++def _fuller_reading(local: _WindowReading, shared: SharedWindow | None) -> _WindowReading:+ if shared is None or shared.spend <= local.current_spend:+ return local+ return _WindowReading(current_spend=shared.spend, started_at=shared.started_at)++ @with_service_target("model_budgets") async def _read_windows( cache: DualCache, spend_keys: Sequence[str], start_time_keys: Sequence[str]@@ -307,6 +743,10 @@ def _window_reset_at(started_at: float | None, budget_duration: str | None, now: return datetime.fromtimestamp(ends_at, tz=timezone.utc).isoformat() +def is_enforced_budget_entry(raw_budget_config: object) -> bool:+ return _usable_budget_config(raw_budget_config) is not None++ def _usable_budget_config(raw_budget_config: object) -> BudgetConfig | None: try: budget_config: Final = BudgetConfig.model_validate(raw_budget_config)@@ -337,18 +777,44 @@ def _as_float(value: object) -> float | None: def _resolve_entity_model_budgets( model: str, entity_budgets: Iterable[tuple[Litellm_EntityType, str | None, object]],+ model_access_groups: ModelAccessGroups,+ team_id: str | None, ) -> tuple[tuple[Litellm_EntityType, str, ResolvedModelBudget], ...]: """Drop the scopes that do not budget `model`, keeping only what can be incremented.""" return tuple( (entity_type, entity_id, resolved) for entity_type, entity_id, model_max_budget in entity_budgets if entity_id is not None and isinstance(model_max_budget, Mapping) and model_max_budget- for resolved in (resolve_model_budget(model=model, model_max_budget=model_max_budget),)+ for resolved in (+ resolve_model_budget(+ model=model,+ model_max_budget=model_max_budget,+ model_access_groups=model_access_groups,+ team_id=team_id,+ ),+ ) if resolved is not None and resolved.budget_config.budget_duration is not None ) +def _requested_model_name(+ metadata: Mapping[str, object], standard_logging_payload: StandardLoggingPayload+) -> str | None:+ """The model name the request was gated on: a team-scoped deployment's public name, since the router renames+ its model group to a unique internal one, else the model group, else the deployment's model."""+ team_public_model_name: Final = _metadata_value(metadata.get("model_info"), "team_public_model_name")+ if (+ isinstance(team_public_model_name, str)+ and team_public_model_name+ and _GLOB_WILDCARD not in team_public_model_name+ ):+ return team_public_model_name+ return standard_logging_payload.get("model_group") or standard_logging_payload.get("model")++ def _metadata_value(metadata: object, key: str) -> object | None:+ if not isinstance(metadata, Mapping):+ return None metadata_mapping: Final[Mapping[str, object]] = cast( # cast-ok: callback metadata has string keys Mapping[str, object], metadata )@@ -367,9 +833,17 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): Example: key=$LITELLM_API_KEY, model=gpt-4o, max_budget=100, time_period=1d """ - def __init__(self, dual_cache: DualCache, clock: Callable[[], float] = time.time):+ def __init__(+ self,+ dual_cache: DualCache,+ clock: Callable[[], float] = time.time,+ model_access_groups: ModelAccessGroups = proxy_model_access_groups,+ shared_windows: SharedWindows = PROXY_SHARED_WINDOWS,+ ): self.dual_cache = dual_cache self._clock = clock+ self._model_access_groups = model_access_groups+ self._shared_windows = shared_windows self.redis_increment_operation_queue = [] self._redis_increment_queue_lock = asyncio.Lock() self._redis_increment_flush_lock = asyncio.Lock()@@ -393,6 +867,7 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): return await self._is_entity_within_model_budget( entity_type=Litellm_EntityType.KEY, entity_id=user_api_key_dict.token,+ team_id=user_api_key_dict.team_id, model_max_budget=user_api_key_dict.model_max_budget, model=model, exceeded_message=(@@ -422,6 +897,7 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): user_id: str, user_model_max_budget: Mapping[str, object], model: str,+ team_id: str | None = None, ) -> bool: """ Check if the internal user is within the model budget@@ -432,6 +908,7 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): return await self._is_entity_within_model_budget( entity_type=Litellm_EntityType.USER, entity_id=user_id,+ team_id=team_id, model_max_budget=user_model_max_budget, model=model, exceeded_message=f"LiteLLM User: {user_id}, exceeded budget for model={model}",@@ -443,6 +920,7 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): end_user_id: str, end_user_model_max_budget: Mapping[str, object], model: str,+ team_id: str | None = None, ) -> bool: """ Check if the end_user is within the model budget@@ -453,6 +931,7 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): return await self._is_entity_within_model_budget( entity_type=Litellm_EntityType.END_USER, entity_id=end_user_id,+ team_id=team_id, model_max_budget=end_user_model_max_budget, model=model, exceeded_message=f"LiteLLM End User: {end_user_id}, exceeded budget for model={model}",@@ -467,18 +946,19 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): model: str, ) -> bool: """- Check if the team is within the model budget, unless the key's own- `model_max_budget` overrides it for `model`+ Check if the team is within the model budget, leaving out a team entry named after `model` that the key's+ own `model_max_budget` replaces Raises: BudgetExceededError: If the team has exceeded the model budget """- if not team_model_budget_applies(model=model, key_model_max_budget=key_model_max_budget):- return True return await self._is_entity_within_model_budget( entity_type=Litellm_EntityType.TEAM, entity_id=team_id,- model_max_budget=team_model_max_budget,+ team_id=team_id,+ model_max_budget=team_model_budget_entries(+ model=model, team_model_max_budget=team_model_max_budget, key_model_max_budget=key_model_max_budget+ ), model=model, exceeded_message=f"LiteLLM Team: {team_id}, exceeded budget for model={model}", )@@ -494,6 +974,7 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): return await self._is_entity_within_model_budget( entity_type=Litellm_EntityType.TEAM_MEMBER, entity_id=team_member_budget_entity_id(user_id=user_id, team_id=team_id),+ team_id=team_id, model_max_budget=team_member_model_max_budget, model=model, exceeded_message=(@@ -505,13 +986,19 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): self, entity_type: Litellm_EntityType, entity_id: str | None,+ team_id: str | None, model_max_budget: Mapping[str, object] | None, model: str, exceeded_message: str, ) -> bool: if not model_max_budget: return True- resolved: Final = resolve_model_budget(model=model, model_max_budget=model_max_budget)+ resolved: Final = resolve_model_budget(+ model=model,+ model_max_budget=model_max_budget,+ model_access_groups=self._model_access_groups,+ team_id=team_id,+ ) if resolved is None: verbose_proxy_logger.debug("Model %s not found in %s model_max_budget", model, entity_type.value) return True@@ -520,15 +1007,26 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): if max_budget is None or max_budget < 0: return True - current_spend: Final = await self._get_spend_for_model_budget(+ local_spend: Final = await self._get_spend_for_model_budget( entity_type=entity_type, entity_id=entity_id, model=model, resolved=resolved, )+ current_spend: Final = (+ local_spend+ if local_spend >= max_budget+ else await self._with_shared_spend(+ entity_type=entity_type, entity_id=entity_id, resolved=resolved, local_spend=local_spend+ )+ ) if current_spend >= max_budget: raise litellm.BudgetExceededError(- message=exceeded_message,+ message=(+ exceeded_message+ if resolved.budget_model in _budget_model_candidates(model)+ else f"{exceeded_message}, budget_model={resolved.budget_model}"+ ), current_cost=current_spend, max_budget=max_budget, entity_type=entity_type.value,@@ -566,6 +1064,30 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): return current_spend return current_spend + _as_spend(await self._cached_spend(legacy_spend_key)) + async def _with_shared_spend(+ self,+ entity_type: Litellm_EntityType,+ entity_id: str | None,+ resolved: ResolvedModelBudget,+ local_spend: float,+ ) -> float:+ """Without Redis each replica's counter holds only its own share, so the window every replica adds to+ in the database is the floor."""+ if not self._shares_windows_in_database(entity_type):+ return local_spend+ shared: Final = await _open_shared_window(+ shared_windows=self._shared_windows,+ spend_key=model_budget_spend_cache_key(+ entity_type=entity_type,+ entity_id=entity_id,+ budget_model=resolved.budget_model,+ budget_duration=resolved.budget_config.budget_duration,+ ),+ budget_duration=resolved.budget_config.budget_duration,+ now=self._clock(),+ )+ return local_spend if shared is None else max(local_spend, shared.spend)+ async def _cached_spend(self, spend_key: str) -> float | None: redis_cache: Final = self.dual_cache.redis_cache if redis_cache is None:@@ -604,13 +1126,10 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): ) payload_metadata: Final = standard_logging_payload.get("metadata") or {} - # Use model_group (the user-facing model alias, e.g. "gpt-4o") when- # available. The enforcement path receives the model name from- # request_data["model"] which is the model group alias, so the spend- # tracking cache key must resolve from the same name. Falling back to- # the deployment-level "model" field preserves behaviour for non-proxy- # or non-router deployments where model_group is None.- model: Final = standard_logging_payload.get("model_group") or standard_logging_payload.get("model")+ model: Final = _requested_model_name(+ metadata=_metadata, # pyright: ignore[reportUnknownArgumentType] # callback metadata is partially untyped+ standard_logging_payload=standard_logging_payload,+ ) if model is None: return @@ -624,6 +1143,10 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): payload_metadata, # pyright: ignore[reportUnknownArgumentType] # callback metadata is partially untyped "user_api_key_team_id", )+ team_model_max_budget: Final = _metadata_value(+ _metadata, # pyright: ignore[reportUnknownArgumentType] # callback metadata is partially untyped+ "user_api_key_team_model_max_budget",+ ) team_member_entity_id: Final = ( team_member_budget_entity_id(user_id=team_member_user_id, team_id=team_member_team_id) if team_member_user_id is not None and team_member_team_id is not None@@ -637,16 +1160,17 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): ), ( Litellm_EntityType.TEAM,- payload_metadata.get("user_api_key_team_id"),+ team_member_team_id, (- _metadata.get("user_api_key_team_model_max_budget")- if team_model_budget_applies(+ team_model_budget_entries( model=model,+ team_model_max_budget=team_model_max_budget, key_model_max_budget=( key_model_max_budget if isinstance(key_model_max_budget, Mapping) else None ), )- else None+ if isinstance(team_model_max_budget, Mapping)+ else team_model_max_budget ), ), (@@ -669,7 +1193,12 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): ), ) - resolved_budgets: Final = _resolve_entity_model_budgets(model=model, entity_budgets=entity_budgets)+ resolved_budgets: Final = _resolve_entity_model_budgets(+ model=model,+ entity_budgets=entity_budgets,+ model_access_groups=self._model_access_groups,+ team_id=team_member_team_id,+ ) if not resolved_budgets: verbose_proxy_logger.debug( "Not running _PROXY_VirtualKeyModelMaxBudgetLimiter.async_log_success_event: "@@ -723,16 +1252,113 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): ttl_seconds=duration_in_seconds(budget_duration), ): return+ start_time_key: Final = model_budget_start_time_cache_key(+ entity_type=entity_type,+ entity_id=entity_id,+ budget_model=resolved.budget_model,+ budget_duration=budget_duration,+ )+ shares_window: Final = self._shares_windows_in_database(entity_type)+ if shares_window:+ await self._join_shared_window(+ spend_key=spend_key, start_time_key=start_time_key, budget_duration=budget_duration+ ) await self._increment_spend_for_key( budget_config=resolved.budget_config, spend_key=spend_key,- start_time_key=model_budget_start_time_cache_key(- entity_type=entity_type,- entity_id=entity_id,- budget_model=resolved.budget_model,+ start_time_key=start_time_key,+ response_cost=response_cost,+ )+ if shares_window:+ await self._share_spend(+ spend_key=spend_key,+ start_time_key=start_time_key,+ budget_duration=budget_duration,+ response_cost=response_cost,+ )++ def _shares_windows_in_database(self, entity_type: Litellm_EntityType) -> bool:+ """End-user ids are whatever the client sends, so their windows stay on each replica rather than adding a+ database row per id."""+ return self.dual_cache.redis_cache is None and entity_type not in _PER_REPLICA_SCOPES++ async def _join_shared_window(self, spend_key: str, start_time_key: str, budget_duration: str) -> None:+ """Without Redis, start this replica's window where the database says every replica's window starts.++ Every replica then sends the same window start with its spend, so no flush reads as a newer window that+ replaces the spend the others already added. While the database cannot say, this replica keeps its spend+ unshared and sends it once a later charge joins the shared window.+ """+ now: Final = self._clock()+ local: Final = _local_windows(self.dual_cache)+ unshared_key: Final = _unshared_spend_key(start_time_key)+ if local.get_cache(key=unshared_key) is None and _open_local_window_start(+ local=local, start_time_key=start_time_key, budget_duration=budget_duration, now=now+ ):+ return+ started_at: Final = await self._shared_windows.open(+ spend_key=spend_key, budget_duration=budget_duration, now=now+ )+ unshared_spend: Final = _as_float(local.get_cache(key=unshared_key))+ local_window_open: Final = _open_local_window_start(+ local=local, start_time_key=start_time_key, budget_duration=budget_duration, now=now+ )+ if local_window_open and unshared_spend is None:+ return+ if not local_window_open:+ _start_local_window(+ local=local,+ spend_key=spend_key,+ start_time_key=start_time_key,+ started_at=now if started_at is None else started_at, budget_duration=budget_duration,+ now=now,+ spend=0.0,+ )+ if started_at is None:+ local.set_cache(key=unshared_key, value=0.0, ttl=budget_window_seconds(now, budget_duration))+ else:+ local.delete_cache(key=unshared_key)+ return+ if started_at is None or unshared_spend is None:+ return+ _start_local_window(+ local=local,+ spend_key=spend_key,+ start_time_key=start_time_key,+ started_at=started_at,+ budget_duration=budget_duration,+ now=now,+ spend=_as_float(local.get_cache(key=spend_key)) or 0.0,+ )+ local.delete_cache(key=unshared_key)+ await self._shared_windows.add(+ spend_key=spend_key, budget_duration=budget_duration, started_at=started_at, spend=unshared_spend+ )++ async def _share_spend(+ self, spend_key: str, start_time_key: str, budget_duration: str, response_cost: float+ ) -> None:+ now: Final = self._clock()+ local: Final = _local_windows(self.dual_cache)+ unshared_key: Final = _unshared_spend_key(start_time_key)+ unshared_spend: Final = _as_float(local.get_cache(key=unshared_key))+ if unshared_spend is not None:+ local.set_cache(+ key=unshared_key,+ value=unshared_spend + response_cost,+ ttl=budget_window_seconds(now, budget_duration),+ )+ return+ await self._shared_windows.add(+ spend_key=spend_key,+ budget_duration=budget_duration,+ started_at=await self._get_or_set_budget_start_time(+ start_time_key=start_time_key,+ current_time=now,+ ttl_seconds=budget_window_seconds(now, budget_duration), ),- response_cost=response_cost,+ spend=response_cost, ) async def _claim_batch_charge(self, spend_key: str, batch_id: str, ttl_seconds: int) -> bool:@@ -743,4 +1369,33 @@ class _PROXY_VirtualKeyModelMaxBudgetLimiter(RouterBudgetLimiting): return polls == 1 +def _local_windows(dual_cache: DualCache) -> LocalTTLCache:+ return dual_cache.in_memory_cache+++def _open_local_window_start(local: LocalTTLCache, start_time_key: str, budget_duration: str, now: float) -> bool:+ started_at: Final = _as_float(local.get_cache(key=start_time_key))+ return started_at is not None and now - started_at <= budget_window_seconds(started_at, budget_duration)+++def _start_local_window(+ local: LocalTTLCache,+ spend_key: str,+ start_time_key: str,+ started_at: float,+ budget_duration: str,+ now: float,+ spend: float,+) -> None:+ ttl: Final = started_at + budget_window_seconds(started_at, budget_duration) - now+ local.delete_cache(key=spend_key)+ local.delete_cache(key=start_time_key)+ local.set_cache(key=spend_key, value=spend, ttl=ttl)+ local.set_cache(key=start_time_key, value=started_at, ttl=ttl)+++def _unshared_spend_key(start_time_key: str) -> str:+ return f"{start_time_key}:unshared"++ PROXY_VirtualKeyModelMaxBudgetLimiter = _PROXY_VirtualKeyModelMaxBudgetLimiterlitellm/proxy/management_endpoints/internal_user_endpoints.pyindex d6bf9220551..f2f0bc5be0b 100644@@ -499,7 +499,7 @@ async def new_user( - permissions: Optional[dict] - [Not Implemented Yet] User-specific permissions, eg. turning off pii masking. - metadata: Optional[dict] - Metadata for user, store information for user. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" } - max_parallel_requests: Optional[int] - Rate limit a user based on the number of parallel requests. Raises 429 error, if user's parallel requests > x.- - model_max_budget: Optional[dict] - Model-specific max budget for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys)+ - model_max_budget: Optional[dict] - Model-specific max budget for user, keyed by model name, `*` pattern, or model access group. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys) - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. - model_rpm_limit: Optional[float] - Model-specific rpm limit for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-limits-to-keys) - mcp_rpm_limit: Optional[dict] - Per-MCP-server rpm limit, keyed by MCP server name {"github": 100, "slack": 200}. Enforced for keys and teams only; values set on a user are stored but not enforced per user.@@ -1702,7 +1702,7 @@ async def user_update( - permissions: Optional[dict] - [Not Implemented Yet] User-specific permissions, eg. turning off pii masking. - metadata: Optional[dict] - Metadata for user, store information for user. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" } - max_parallel_requests: Optional[int] - Rate limit a user based on the number of parallel requests. Raises 429 error, if user's parallel requests > x.- - model_max_budget: Optional[dict] - Model-specific max budget for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys)+ - model_max_budget: Optional[dict] - Model-specific max budget for user, keyed by model name, `*` pattern, or model access group. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys) - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. - model_rpm_limit: Optional[float] - Model-specific rpm limit for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-limits-to-keys) - mcp_rpm_limit: Optional[dict] - Per-MCP-server rpm limit, keyed by MCP server name {"github": 100, "slack": 200}. Enforced for keys and teams only; values set on a user are stored but not enforced per user.litellm/proxy/management_endpoints/key_management_endpoints.pyindex f7b72b263d4..a19ecb8a9e9 100644@@ -2328,7 +2328,7 @@ async def generate_key_fn( - throttle_on_budget_exceeded: Optional[bool] - When the key exceeds its max_budget, throttle its tpm/rpm to the global budget_exceeded_throttle_percentage instead of blocking the key entirely. - enable_prompt_caching: Optional[bool] - Auto-inject prompt caching breakpoints (Anthropic cache_control markers) on requests made with this key. Supported Claude models on Anthropic, Bedrock, Vertex AI, and Azure AI only. - permissions: Optional[dict] - key-specific permissions. Currently just used for turning off pii masking (if connected). Example - {"pii": false}- - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget.+ - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget. - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. - model_rpm_limit: Optional[dict] - key-specific model rpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific rpm limit. - model_tpm_limit: Optional[dict] - key-specific model tpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific tpm limit.@@ -2538,7 +2538,7 @@ async def generate_service_account_key_fn( - metadata: Optional[dict] - Metadata for key, store information for key. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" } - guardrails: Optional[List[str]] - List of active guardrails for the key - permissions: Optional[dict] - key-specific permissions. Currently just used for turning off pii masking (if connected). Example - {"pii": false}- - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget.+ - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget. - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. - model_rpm_limit: Optional[dict] - key-specific model rpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific rpm limit. - model_tpm_limit: Optional[dict] - key-specific model tpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific tpm limit.@@ -3855,7 +3855,7 @@ async def update_key_fn( - enforced_params: Optional[List[str]] - List of enforced params for the key (Enterprise only). [Docs](https://docs.litellm.ai/docs/proxy/enterprise#enforce-required-params-for-llm-requests) - spend: Optional[float] - Amount spent by key - max_budget: Optional[float] - Max budget for key- - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}+ - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}} - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. - budget_duration: Optional[str] - Budget reset period ("30d", "1h", etc.) - soft_budget: Optional[float] - Soft budget limit (warning vs. hard stop). Will trigger a slack alert when this soft budget is reached. Set to null to remove the soft budget.@@ -6303,7 +6303,7 @@ async def regenerate_key_fn( - tags: Optional[List[str]] - Tags for organizing keys (Enterprise only) - spend: Optional[float] - Amount spent by key - max_budget: Optional[float] - Max budget for key- - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}+ - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}} - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. - budget_duration: Optional[str] - Budget reset period ("30d", "1h", etc.) - soft_budget: Optional[float] - Soft budget limit (warning vs. hard stop). Will trigger a slack alert when this soft budget is reached.litellm/proxy/management_endpoints/team_endpoints.pyindex 1fa18e9b2b8..a256f417dfa 100644@@ -11,6 +11,7 @@ All /team management endpoints import asyncio import copy+import itertools import json import math import traceback@@ -139,7 +140,12 @@ from litellm.proxy.common_utils.user_api_key_cache import ( from litellm.proxy.db.db_span import db_span from litellm.proxy.hooks.key_management_event_hooks import KeyManagementEventHooks from litellm.proxy.hooks.model_max_budget_limiter import (+ RoutedModels, build_model_max_budget_usage,+ entries_outranked_by,+ entry_names_model,+ is_enforced_budget_entry,+ proxy_routed_models, resolve_model_budget, ) from litellm.proxy.management.teams.authz import TEAM_OR_ORG_ADMIN, TeamRole, is_team_admin, team_access_denied@@ -1467,8 +1473,10 @@ def _check_team_model_budget_update_authority( data: UpdateTeamRequest, user_api_key_dict: UserAPIKeyAuth, existing_model_max_budget: Mapping[str, object] | None,+ routed_models_source: Callable[[str | None], RoutedModels] = proxy_routed_models, ) -> None:- """Like `_check_team_budget_update_authority`: only a proxy admin may raise, re-window or drop a per-model cap."""+ """Like `_check_team_budget_update_authority`: only a proxy admin may raise, re-window or drop a per-model cap, or+ add an entry that takes precedence over one.""" if user_api_key_dict.user_role == LitellmUserRoles.PROXY_ADMIN: return if "model_max_budget" not in data.model_fields_set or not existing_model_max_budget:@@ -1487,29 +1495,97 @@ def _check_team_model_budget_update_authority( ) }, )- for model_name, proposed in requested.items():- governing = resolve_model_budget(model=model_name, model_max_budget=existing_model_max_budget)- if governing is None:- continue- cap = governing.budget_config- if cap.max_budget is None:- continue- if (- proposed.max_budget is None- or proposed.max_budget > cap.max_budget- or proposed.budget_duration != cap.budget_duration- ):- raise HTTPException(- status_code=403,- detail={- "error": (- f"Only a proxy admin can raise a team's model_max_budget for {model_name!r} or change its "- f"budget_duration. Current max_budget={cap.max_budget} per {cap.budget_duration} "- f"(entry {governing.budget_model!r}), requested={proposed.max_budget} per "- f"{proposed.budget_duration}."- )- },+ routed_models: Final = routed_models_source(data.team_id)+ for entry, proposed in requested.items():+ existing = _existing_model_cap(existing_model_max_budget.get(entry))+ if existing is not None:+ _check_existing_model_cap_kept(entry=entry, cap_entry=entry, proposed=proposed, existing=existing)+ if is_enforced_budget_entry(existing) or not is_enforced_budget_entry(proposed):+ continue+ governing = resolve_model_budget(model=entry, model_max_budget=existing_model_max_budget, team_id=data.team_id)+ if governing is not None and entry_names_model(entry=governing.budget_model, model=entry):+ _check_existing_model_cap_kept(+ entry=entry, cap_entry=governing.budget_model, proposed=proposed, existing=governing.budget_config+ )+ _check_new_entry_outranks_no_cap(+ entry=entry,+ existing_model_max_budget=existing_model_max_budget,+ routed_models=routed_models,+ team_id=data.team_id,+ )+++def _check_existing_model_cap_kept(entry: str, cap_entry: str, proposed: BudgetConfig, existing: BudgetConfig) -> None:+ if existing.max_budget is None or not _looser_model_cap(proposed=proposed, cap=existing):+ return+ raise HTTPException(+ status_code=403,+ detail={+ "error": (+ f"Only a proxy admin can raise a team's model_max_budget for {entry!r} or change its "+ f"budget_duration. Current max_budget={existing.max_budget} per {existing.budget_duration} "+ f"(entry {cap_entry!r}), requested={proposed.max_budget} per {proposed.budget_duration}."+ )+ },+ )+++def _check_new_entry_outranks_no_cap(+ entry: str,+ existing_model_max_budget: Mapping[str, object],+ routed_models: RoutedModels,+ team_id: str | None,+) -> None:+ """A new entry gets its own counter, so the models it takes from a capped entry stop drawing from that cap."""+ outranked: Final = next(+ _capped_entries_outranked_by(+ entry=entry,+ existing_model_max_budget=existing_model_max_budget,+ routed_models=routed_models,+ team_id=team_id,+ ),+ None,+ )+ if outranked is None:+ return+ outranked_name, cap = outranked+ raise HTTPException(+ status_code=403,+ detail={+ "error": (+ f"Only a proxy admin can add a team model_max_budget entry {entry!r} that takes precedence over "+ f"{outranked_name!r}, since the models they share would stop drawing from its max_budget="+ f"{cap.max_budget} per {cap.budget_duration}." )+ },+ )+++def _capped_entries_outranked_by(+ entry: str,+ existing_model_max_budget: Mapping[str, object],+ routed_models: RoutedModels,+ team_id: str | None,+) -> Iterator[tuple[str, BudgetConfig]]:+ governing: Final = resolve_model_budget(model=entry, model_max_budget=existing_model_max_budget, team_id=team_id)+ outranked_names: Final = itertools.chain(+ ()+ if governing is None or entry_names_model(entry=governing.budget_model, model=entry)+ else (governing.budget_model,),+ entries_outranked_by(entry=entry, model_max_budget=existing_model_max_budget, routed_models=routed_models),+ )+ outranked_caps: Final = ((name, _existing_model_cap(existing_model_max_budget[name])) for name in outranked_names)+ yield from (+ (name, cap) for name, cap in outranked_caps if name != entry and cap is not None and cap.max_budget is not None+ )+++def _looser_model_cap(proposed: BudgetConfig, cap: BudgetConfig) -> bool:+ return (+ proposed.max_budget is None+ or (cap.max_budget is not None and proposed.max_budget > cap.max_budget)+ or (is_enforced_budget_entry(cap) and proposed.budget_duration != cap.budget_duration)+ ) def _should_auto_add_team_creator(@@ -1572,7 +1648,7 @@ async def new_team( - prompts: Optional[List[str]] - List of prompts that the team is allowed to use. - organization_id: Optional[str] - The organization id of the team. Default is None. Create via `/organization/new`. - model_aliases: Optional[dict] - Model aliases for the team. [Docs](https://docs.litellm.ai/docs/proxy/team_based_routing#create-team-with-model-alias)- - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}}+ - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model, keyed by model name, `*` pattern, or model access group. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}} - guardrails: Optional[List[str]] - Guardrails for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails) - policies: Optional[List[str]] - Policies for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_policies) - disable_global_guardrails: Optional[bool] - Whether to disable global guardrails for the team. Proxy admin only.@@ -2350,7 +2426,7 @@ async def update_team( - tags: Optional[List[str]] - Tags for [tracking spend](https://litellm.vercel.app/docs/proxy/enterprise#tracking-spend-for-custom-tags) and/or doing [tag-based routing](https://litellm.vercel.app/docs/proxy/tag_routing). - organization_id: Optional[str] - The organization id of the team. Default is None. Create via `/organization/new`. - model_aliases: Optional[dict] - Model aliases for the team. [Docs](https://docs.litellm.ai/docs/proxy/team_based_routing#create-team-with-model-alias)- - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}}+ - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model, keyed by model name, `*` pattern, or model access group. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}} - guardrails: Optional[List[str]] - Guardrails for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails) - policies: Optional[List[str]] - Policies for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_policies) - disable_global_guardrails: Optional[bool] - Whether to disable global guardrails for the team. Proxy admin only.litellm/router.pyindex 5b18d36316f..3e6d82d9b07 100644@@ -11850,6 +11850,11 @@ class Router: indices: Final = self.model_name_to_deployment_indices.get(model_name) return None if indices is None else tuple(self.model_list[idx] for idx in indices) + def team_public_model_names_of(self, team_id: str) -> frozenset[str]:+ """The public names `team_id` serves its own deployments under."""+ with self._deployment_index_lock:+ return frozenset(name for owner, name in self.team_model_to_deployment_indices if owner == team_id)+ def _team_deployments_named(self, team_id: str, public_model_name: str) -> tuple[DeploymentTypedDict, ...] | None: """The deployments `team_id` serves under `public_model_name`, or None when that pair is not indexed.""" with self._deployment_index_lock:tests/unit/llms/anthropic/messages/context_management/test_compact.pyindex 159c3586c53..dff931a6074 100644@@ -1523,6 +1523,7 @@ async def test_summary_model_denied_when_user_over_model_budget(): user_id="user-over-budget", user_model_max_budget={"claude-haiku-4-5": {"budget_limit": 5}}, model="claude-haiku-4-5",+ team_id=None, ) import inspect tests/unit/proxy/auth/test_user_api_key_auth.pyindex 092805a9cac..117428ee27b 100644@@ -5,7 +5,7 @@ import litellm.proxy import litellm.proxy.proxy_server -from typing import Dict, Final, List, Optional+from typing import Dict, Final, List, Mapping, Optional from unittest.mock import AsyncMock, MagicMock, patch import pytest@@ -1367,7 +1367,9 @@ async def test_check_user_model_budget(user_id, user_model_max_budget, expected_ calls = [] class _Limiter:- async def is_user_within_model_budget(self, user_id, user_model_max_budget, model):+ async def is_user_within_model_budget(+ self, user_id: str, user_model_max_budget: Mapping[str, object], model: str, team_id: str | None = None+ ) -> bool: calls.append((user_id, user_model_max_budget, model)) return True tests/unit/proxy/db/test_budget_window_spend_writer.pyindex 6fc438feee8..27f1472c183 100644@@ -9,6 +9,7 @@ import pytest from litellm.proxy.db.budget_window_spend_writer import ( WindowSeedTotals, commit_window_spend_updates,+ open_window_spend_row, roll_window_spend_row, spend_logs_seed_totals, )@@ -405,6 +406,29 @@ async def test_roll_window_spend_row_is_conditional_on_the_stored_window_being_o assert params[:4] == ("team", "t1", "30d", datetime(2026, 8, 31)) +@pytest.mark.asyncio+@pytest.mark.parametrize(+ ("replaces_window_start", "replaced_param"), [(None, None), (WINDOW_A, datetime(2026, 8, 1))]+)+async def test_open_window_spend_row_moves_only_the_window_the_caller_saw_end(replaces_window_start, replaced_param):+ db = _FakeDB()++ await open_window_spend_row(+ prisma_client=_FakePrismaClient(db),+ entity_type="model_max_budget",+ entity_id="k1",+ window_duration="1d",+ window_start=WINDOW_B,+ replaces_window_start=replaces_window_start,+ )++ ((query, params),) = db.execute_raw_calls+ normalized = " ".join(query.split())+ assert "ON CONFLICT (entity_type, entity_id, window_duration) DO UPDATE" in normalized+ assert normalized.endswith("WHERE \"LiteLLM_BudgetWindowSpend\".window_start <= ($5::timestamptz AT TIME ZONE 'UTC')")+ assert params == ("model_max_budget", "k1", "1d", datetime(2026, 8, 31), replaced_param)++ @pytest.mark.asyncio async def test_seed_receives_the_batch_earliest_start_as_its_cutoff(): db = _FakeDB(existing_rows=[])tests/unit/proxy/hooks/test_model_budget_shared_windows.pynew file mode 100644index 00000000000..7002f602fca@@ -0,0 +1,199 @@+from datetime import datetime, timedelta, timezone+from types import SimpleNamespace+from typing import Final++import pytest++from litellm.proxy.db.db_transaction_queue.window_spend_update_queue import WindowSpendTransaction+from litellm.proxy.hooks.model_budget_shared_windows import DatabaseSharedWindows, SharedWindow++SPEND_KEY: Final = "virtual_key_spend:key-hash:claude-opus*:1d"+WINDOW_START: Final = datetime(2026, 10, 20, 9, 30, tzinfo=timezone.utc)+++class _Table:+ def __init__(self, row: object | None = None, error: Exception | None = None) -> None:+ self.row = row+ self.error = error+ self.reads = 0++ async def find_unique(self, where: object) -> object | None:+ self.reads = self.reads + 1+ if self.error is not None:+ raise self.error+ return self.row+++class _Database:+ def __init__(self, table: _Table) -> None:+ self.litellm_budgetwindowspend = table+ self.opened_with: tuple[object, ...] = ()++ async def execute_raw(self, query: str, *params: object) -> int:+ self.opened_with = params+ self.litellm_budgetwindowspend.row = SimpleNamespace(window_start=params[3], spend=0.0)+ return 1+++class _OutracedDatabase:+ def __init__(self, table: _Table, winner: object) -> None:+ self.litellm_budgetwindowspend = table+ self._winner = winner++ async def execute_raw(self, query: str, *params: object) -> int:+ self.litellm_budgetwindowspend.row = self._winner+ return 0+++class _Sink:+ def __init__(self) -> None:+ self.transactions: tuple[WindowSpendTransaction, ...] = ()++ async def __call__(self, transaction: WindowSpendTransaction) -> None:+ self.transactions = (*self.transactions, transaction)+++class _LaggingReplica:+ def __init__(self, writer: _Database) -> None:+ self.litellm_budgetwindowspend = _Table()+ self._writer = writer++ async def execute_raw(self, query: str, *params: object) -> int:+ return await self._writer.execute_raw(query, *params)+++def _database_windows(table: _Table | None, sink: _Sink, lagging_replica: bool = False) -> DatabaseSharedWindows:+ writer: Final = None if table is None else _Database(table)+ prisma: Final = (+ None+ if writer is None+ else SimpleNamespace(db=_LaggingReplica(writer) if lagging_replica else writer, writer_db=writer)+ )+ return DatabaseSharedWindows(+ prisma_client_source=lambda: prisma, # pyright: ignore[reportArgumentType] # duck-typed prisma double+ sink=sink,+ )+++def test_a_window_is_open_until_its_period_has_passed():+ window: Final = SharedWindow(started_at=WINDOW_START.timestamp(), spend=1.0)+ day: Final = 24 * 60 * 60++ assert window.is_open(budget_duration="1d", now=WINDOW_START.timestamp() + day - 1)+ assert not window.is_open(budget_duration="1d", now=WINDOW_START.timestamp() + day)+++@pytest.mark.asyncio+async def test_read_returns_the_stored_window_and_rereads_only_after_the_cache_expires():+ table: Final = _Table(row=SimpleNamespace(window_start=WINDOW_START.replace(tzinfo=None), spend=2.5))+ windows: Final = _database_windows(table, _Sink())++ first: Final = await windows.read(spend_key=SPEND_KEY, budget_duration="1d")+ second: Final = await windows.read(spend_key=SPEND_KEY, budget_duration="1d")++ assert first == SharedWindow(started_at=WINDOW_START.timestamp(), spend=2.5)+ assert second == first+ assert table.reads == 1+++@pytest.mark.asyncio+async def test_an_unreadable_table_reads_as_no_shared_window():+ table: Final = _Table(error=RuntimeError("database is down"))+ windows: Final = _database_windows(table, _Sink())++ assert await windows.read(spend_key=SPEND_KEY, budget_duration="1d") is None+++@pytest.mark.asyncio+async def test_add_queues_the_charge_for_the_window_it_belongs_to():+ sink: Final = _Sink()+ windows: Final = _database_windows(_Table(), sink)++ await windows.add(spend_key=SPEND_KEY, budget_duration="1d", started_at=WINDOW_START.timestamp(), spend=0.25)++ assert len(sink.transactions) == 1+ queued: Final = sink.transactions[0]+ assert (queued["entity_id"], queued["window_duration"], queued["spend"]) == (SPEND_KEY, "1d", 0.25)+ assert datetime.fromisoformat(queued["window_start"]).replace(tzinfo=timezone.utc) == WINDOW_START+++@pytest.mark.asyncio+async def test_nothing_is_queued_without_a_database():+ sink: Final = _Sink()+ windows: Final = _database_windows(None, sink)++ await windows.add(spend_key=SPEND_KEY, budget_duration="1d", started_at=WINDOW_START.timestamp(), spend=0.25)++ assert sink.transactions == ()+ assert await windows.read(spend_key=SPEND_KEY, budget_duration="1d") is None+++@pytest.mark.asyncio+async def test_open_joins_the_window_another_replica_already_started():+ table: Final = _Table(row=SimpleNamespace(window_start=WINDOW_START.replace(tzinfo=None), spend=2.5))+ windows: Final = _database_windows(table, _Sink())++ started_at: Final = await windows.open(+ spend_key=SPEND_KEY, budget_duration="1d", now=WINDOW_START.timestamp() + 3600+ )++ assert started_at == WINDOW_START.timestamp()+ assert table.row.spend == 2.5+++@pytest.mark.asyncio+async def test_open_reads_the_window_from_the_writer_while_the_read_replica_lags():+ table: Final = _Table(row=SimpleNamespace(window_start=WINDOW_START.replace(tzinfo=None), spend=2.5))+ windows: Final = _database_windows(table, _Sink(), lagging_replica=True)++ started_at: Final = await windows.open(+ spend_key=SPEND_KEY, budget_duration="1d", now=WINDOW_START.timestamp() + 3600+ )++ assert started_at == WINDOW_START.timestamp()+++@pytest.mark.asyncio+async def test_open_joins_the_window_a_racing_replica_opened_first():+ winner: Final = SimpleNamespace(window_start=WINDOW_START.replace(tzinfo=None), spend=2.5)+ database: Final = _OutracedDatabase(_Table(), winner)+ windows: Final = DatabaseSharedWindows(+ prisma_client_source=lambda: SimpleNamespace(db=database, writer_db=database), # pyright: ignore[reportArgumentType] # duck-typed prisma double+ sink=_Sink(),+ )++ started_at: Final = await windows.open(spend_key=SPEND_KEY, budget_duration="1d", now=WINDOW_START.timestamp() + 60)++ assert started_at == WINDOW_START.timestamp()+++@pytest.mark.asyncio+@pytest.mark.parametrize("stored_start", [None, WINDOW_START - timedelta(days=2)])+async def test_open_starts_a_window_at_now_when_none_is_open(stored_start: datetime | None):+ table: Final = _Table(+ row=None if stored_start is None else SimpleNamespace(window_start=stored_start.replace(tzinfo=None), spend=9.0)+ )+ windows: Final = _database_windows(table, _Sink())++ started_at: Final = await windows.open(spend_key=SPEND_KEY, budget_duration="1d", now=WINDOW_START.timestamp())++ assert started_at == WINDOW_START.timestamp()+ assert await windows.read(spend_key=SPEND_KEY, budget_duration="1d") == SharedWindow(+ started_at=WINDOW_START.timestamp(), spend=0.0+ )+++@pytest.mark.asyncio+async def test_open_starts_this_replicas_own_window_without_a_database():+ windows: Final = _database_windows(None, _Sink())++ assert await windows.open(spend_key=SPEND_KEY, budget_duration="1d", now=WINDOW_START.timestamp()) == (+ WINDOW_START.timestamp()+ )+++@pytest.mark.asyncio+async def test_open_reports_no_shared_window_while_the_database_is_unreachable():+ windows: Final = _database_windows(_Table(error=RuntimeError("database is down")), _Sink())++ assert await windows.open(spend_key=SPEND_KEY, budget_duration="1d", now=WINDOW_START.timestamp()) is Nonetests/unit/proxy/hooks/test_model_max_budget_limiter.pyindex 2678f4205d5..f5a2ccf7b76 100644@@ -1,19 +1,30 @@ import asyncio+import functools+import re import time from collections.abc import Callable, Mapping-from types import MappingProxyType+from dataclasses import dataclass+from datetime import datetime, timezone+from types import MappingProxyType, SimpleNamespace from typing import Final import pytest import litellm from litellm.caching.caching import DualCache-from litellm.proxy._types import Litellm_EntityType+from litellm.proxy._types import Litellm_EntityType, UserAPIKeyAuth+from litellm.proxy.db.db_transaction_queue.window_spend_update_queue import WindowSpendTransaction+from litellm.proxy.hooks.model_budget_shared_windows import DatabaseSharedWindows, SharedWindows from litellm.proxy.hooks.model_max_budget_limiter import ( PROXY_VirtualKeyModelMaxBudgetLimiter,+ build_model_max_budget_usage,+ model_access_groups_of, model_budget_spend_cache_key,+ resolve_model_budget,+ routed_models_of, team_member_budget_entity_id, )+from litellm.router import Router from litellm.types.caching import RedisPipelineIncrementOperation from litellm.types.utils import LiteLLMBatch, Usage @@ -74,10 +85,14 @@ def _event( } +async def _log_success(+ limiter: PROXY_VirtualKeyModelMaxBudgetLimiter, event: dict[str, object], response_obj: object = None+) -> None:+ await limiter.async_log_success_event(event, response_obj=response_obj, start_time=None, end_time=None)++ async def _poll(limiter: PROXY_VirtualKeyModelMaxBudgetLimiter, batch: LiteLLMBatch, response_cost: float) -> None:- await limiter.async_log_success_event(- _event("aretrieve_batch", response_cost), response_obj=batch, start_time=None, end_time=None- )+ await _log_success(limiter, _event("aretrieve_batch", response_cost), response_obj=batch) async def _chat(@@ -86,7 +101,8 @@ async def _chat( user_id: str = USER_ID, team_id: str = TEAM_ID, ) -> None:- await limiter.async_log_success_event(+ await _log_success(+ limiter, _event( "acompletion", CHAT_COST,@@ -94,9 +110,6 @@ async def _chat( user_id=user_id, team_id=team_id, ),- response_obj=None,- start_time=None,- end_time=None, ) @@ -311,3 +324,568 @@ async def test_a_batch_polled_within_every_budget_window_is_never_charged_again( await _poll(limiter, finished, BATCH_COST) assert _local_spend(limiter, KEY_SPEND_KEY) == pytest.approx(BATCH_COST)+++OPUS_GROUP: Final = "claude-opus"+OPUS_GROUP_MEMBERS: Final = MappingProxyType(+ {"claude-opus-4-8": frozenset({OPUS_GROUP}), "claude-opus-5-5": frozenset({OPUS_GROUP})}+)+FAMILY_CAP: Final = {"budget_limit": CHAT_COST, "time_period": "1d"} # mutable-ok: BudgetConfig validates dicts only+ROOMY_CAP: Final = {+ "budget_limit": CHAT_COST * 100,+ "time_period": "1d",+} # mutable-ok: BudgetConfig validates dicts only+WINDOW_EPOCH: Final = datetime(2026, 10, 20, 9, 30, tzinfo=timezone.utc).timestamp()+++def _model_access_groups(+ groups_by_model: Mapping[str, frozenset[str]],+) -> Callable[[str, str | None], frozenset[str]]:+ return lambda model, _team_id: groups_by_model.get(model, frozenset())+++@dataclass(frozen=True, slots=True)+class _WindowRow:+ window_start: datetime+ spend: float+++class _WindowSpendDatabase:+ def __init__(self) -> None:+ self.rows: Mapping[str, _WindowRow] = MappingProxyType({})+ self.queued: tuple[WindowSpendTransaction, ...] = ()+ self.reachable = True++ @property+ def litellm_budgetwindowspend(self) -> "_WindowSpendDatabase":+ return self++ async def find_unique(self, where: Mapping[str, Mapping[str, str]]) -> _WindowRow | None:+ await asyncio.sleep(0)+ self._answer()+ return self.rows.get(where["entity_type_entity_id_window_duration"]["entity_id"])++ async def execute_raw(+ self,+ query: str,+ entity_type: str,+ entity_id: str,+ window_duration: str,+ window_start: datetime,+ replaces_window_start: datetime | None,+ ) -> int:+ await asyncio.sleep(0)+ self._answer()+ stored: Final = self.rows.get(entity_id)+ if stored is not None and (replaces_window_start is None or stored.window_start > replaces_window_start):+ return 0+ self.rows = MappingProxyType({**self.rows, entity_id: _WindowRow(window_start=window_start, spend=0.0)})+ return 1++ async def queue(self, transaction: WindowSpendTransaction) -> None:+ self.queued = (*self.queued, transaction)++ def flush(self) -> None:+ self.rows = functools.reduce(_upsert_window_spend, self.queued, self.rows)+ self.queued = ()++ def replica(self) -> DatabaseSharedWindows:+ return DatabaseSharedWindows(+ prisma_client_source=lambda: SimpleNamespace(db=self, writer_db=self), # pyright: ignore[reportArgumentType] # duck-typed prisma double+ sink=self.queue,+ )++ def _answer(self) -> None:+ if not self.reachable:+ raise ConnectionError("database is unreachable")+++def _upsert_window_spend(+ rows: Mapping[str, _WindowRow], transaction: WindowSpendTransaction+) -> Mapping[str, _WindowRow]:+ window_start: Final = datetime.fromisoformat(transaction["window_start"])+ stored: Final = rows.get(transaction["entity_id"])+ row: Final = (+ _WindowRow(window_start=stored.window_start, spend=stored.spend + transaction["spend"])+ if stored is not None and stored.window_start >= window_start+ else _WindowRow(window_start=window_start, spend=transaction["spend"])+ )+ return MappingProxyType({**rows, transaction["entity_id"]: row})+++def _family_limiter(+ groups_by_model: Mapping[str, frozenset[str]] = OPUS_GROUP_MEMBERS,+ dual_cache: DualCache | None = None,+ shared_windows: SharedWindows | None = None,+ clock: Callable[[], float] = time.time,+) -> PROXY_VirtualKeyModelMaxBudgetLimiter:+ return PROXY_VirtualKeyModelMaxBudgetLimiter(+ dual_cache=dual_cache or DualCache(),+ clock=clock,+ model_access_groups=_model_access_groups(groups_by_model),+ shared_windows=shared_windows or _WindowSpendDatabase().replica(),+ )+++def _key(model_max_budget: Mapping[str, object]) -> UserAPIKeyAuth:+ return UserAPIKeyAuth(token=KEY_HASH, key_alias="opus-family", model_max_budget=dict(model_max_budget))+++async def _spend_on(+ limiter: PROXY_VirtualKeyModelMaxBudgetLimiter, model: str, model_max_budget: Mapping[str, object]+) -> None:+ await _log_success(+ limiter,+ {+ "call_type": "acompletion",+ "standard_logging_object": {+ "call_type": "acompletion",+ "response_cost": CHAT_COST,+ "model": f"anthropic/{model}",+ "model_group": model,+ "metadata": {"user_api_key_hash": KEY_HASH},+ },+ "litellm_params": {"metadata": {"user_api_key_model_max_budget": dict(model_max_budget)}},+ },+ )+++@pytest.mark.asyncio+@pytest.mark.parametrize("family_entry", ["claude-opus*", OPUS_GROUP])+async def test_every_opus_version_draws_from_one_family_entry(family_entry: str):+ budget: Final = {family_entry: FAMILY_CAP}+ limiter: Final = _family_limiter()+ assert await limiter.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-opus-5-5")++ await _spend_on(limiter, model="claude-opus-4-8", model_max_budget=budget)++ with pytest.raises(+ litellm.BudgetExceededError, match=rf"model=claude-opus-5-5, budget_model={re.escape(family_entry)}$"+ ):+ await limiter.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-opus-5-5")+ assert await limiter.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-sonnet-5")+++@pytest.mark.asyncio+async def test_a_model_added_to_the_access_group_later_is_covered_with_no_key_change():+ budget: Final = {OPUS_GROUP: FAMILY_CAP}+ shared_cache: Final = DualCache()+ before_group_change: Final = _family_limiter(dual_cache=shared_cache)+ after_group_change: Final = _family_limiter(+ groups_by_model={**OPUS_GROUP_MEMBERS, "claude-opus-6": frozenset({OPUS_GROUP})}, dual_cache=shared_cache+ )++ await _spend_on(before_group_change, model="claude-opus-4-8", model_max_budget=budget)++ assert await before_group_change.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-opus-6")+ with pytest.raises(litellm.BudgetExceededError):+ await after_group_change.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-opus-6")+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ ("budget", "checked_model", "blocked"),+ [+ ({"claude-opus-4-8": ROOMY_CAP, "claude-opus*": FAMILY_CAP}, "claude-opus-4-8", False),+ ({"claude-opus-4-8": ROOMY_CAP, "claude-opus*": FAMILY_CAP}, "claude-opus-5-5", False),+ ({OPUS_GROUP: FAMILY_CAP, "claude-opus*": ROOMY_CAP}, "claude-opus-5-5", True),+ ({"claude-opus-4*": FAMILY_CAP, "claude-opus*": ROOMY_CAP}, "claude-opus-4-7", True),+ ({"claude-opus-4*": FAMILY_CAP, "claude-opus*": ROOMY_CAP}, "claude-opus-5-5", False),+ ({"claude-opus-4*": {"budget_limit": CHAT_COST}, "claude-opus*": FAMILY_CAP}, "claude-opus-5-5", True),+ ({"claude-opus-4*": FAMILY_CAP, "c" + "*" * 15: ROOMY_CAP}, "claude-opus-4-7", True),+ ],+)+async def test_the_most_specific_entry_takes_the_spend_and_gates_the_request(+ budget: Mapping[str, object], checked_model: str, blocked: bool+):+ limiter: Final = _family_limiter()++ await _spend_on(limiter, model="claude-opus-4-8", model_max_budget=budget)++ if blocked:+ with pytest.raises(litellm.BudgetExceededError):+ await limiter.is_key_within_model_budget(user_api_key_dict=_key(budget), model=checked_model)+ else:+ assert await limiter.is_key_within_model_budget(user_api_key_dict=_key(budget), model=checked_model)+++@pytest.mark.asyncio+async def test_an_exact_entry_refusal_reads_as_before_for_a_provider_prefixed_model():+ budget: Final = {"claude-opus-4-8": FAMILY_CAP}+ limiter: Final = _family_limiter()++ await _spend_on(limiter, model="claude-opus-4-8", model_max_budget=budget)++ with pytest.raises(litellm.BudgetExceededError, match=r"model=anthropic/claude-opus-4-8$"):+ await limiter.is_key_within_model_budget(user_api_key_dict=_key(budget), model="anthropic/claude-opus-4-8")+++async def _spend_on_team_key(+ limiter: PROXY_VirtualKeyModelMaxBudgetLimiter,+ model: str,+ key_model_max_budget: Mapping[str, object],+ team_model_max_budget: Mapping[str, object],+) -> None:+ await _log_success(+ limiter,+ {+ "call_type": "acompletion",+ "standard_logging_object": {+ "call_type": "acompletion",+ "response_cost": CHAT_COST,+ "model": f"anthropic/{model}",+ "model_group": model,+ "metadata": {"user_api_key_hash": KEY_HASH, "user_api_key_team_id": TEAM_ID},+ },+ "litellm_params": {+ "metadata": {+ "user_api_key_model_max_budget": dict(key_model_max_budget),+ "user_api_key_team_model_max_budget": dict(team_model_max_budget),+ }+ },+ },+ )+++@pytest.mark.asyncio+@pytest.mark.parametrize(+ ("team_model_max_budget", "team_blocks"),+ [({"claude-opus*": FAMILY_CAP}, True), ({OPUS_GROUP: FAMILY_CAP}, True), ({"claude-opus-4-8": FAMILY_CAP}, False)],+)+async def test_a_key_entry_named_after_the_model_replaces_only_the_team_entry_of_that_name(+ team_model_max_budget: Mapping[str, object], team_blocks: bool+):+ key_model_max_budget: Final = {"claude-opus-4-8": ROOMY_CAP}+ limiter: Final = _family_limiter()++ await _spend_on_team_key(+ limiter,+ model="claude-opus-4-8",+ key_model_max_budget=key_model_max_budget,+ team_model_max_budget=team_model_max_budget,+ )+ team_check: Final = limiter.is_team_within_model_budget(+ team_id=TEAM_ID,+ team_model_max_budget=team_model_max_budget,+ key_model_max_budget=key_model_max_budget,+ model="claude-opus-4-8",+ )++ if team_blocks:+ with pytest.raises(litellm.BudgetExceededError, match=r"budget_model="):+ await team_check+ else:+ assert await team_check+++TEAM_OPUS_GROUP: Final = "team-opus"+++@pytest.mark.asyncio+@pytest.mark.parametrize("budget_entry", [TEAM_OPUS_GROUP, "claude-opus*"])+async def test_a_team_model_is_gated_and_charged_under_its_public_name(budget_entry: str):+ router: Final = Router(+ model_list=[+ {+ "model_name": f"model_name_{TEAM_ID}_opus",+ "litellm_params": {"model": "anthropic/claude-opus-4-8", "api_key": "sk-test"},+ "model_info": {+ "id": "team-opus-deployment",+ "team_id": TEAM_ID,+ "team_public_model_name": "claude-opus-4-8",+ "access_groups": [TEAM_OPUS_GROUP],+ },+ }+ ]+ )+ limiter: Final = PROXY_VirtualKeyModelMaxBudgetLimiter(+ dual_cache=DualCache(),+ model_access_groups=lambda model, team_id: model_access_groups_of(router=router, model=model, team_id=team_id),+ shared_windows=_WindowSpendDatabase().replica(),+ )+ budget: Final = {budget_entry: FAMILY_CAP}+ team_key: Final = UserAPIKeyAuth(token=KEY_HASH, team_id=TEAM_ID, model_max_budget=budget)+ assert await limiter.is_key_within_model_budget(user_api_key_dict=team_key, model="claude-opus-4-8")++ await _log_success(+ limiter,+ {+ "call_type": "acompletion",+ "standard_logging_object": {+ "call_type": "acompletion",+ "response_cost": CHAT_COST,+ "model": "anthropic/claude-opus-4-8",+ "model_group": f"model_name_{TEAM_ID}_opus",+ "metadata": {"user_api_key_hash": KEY_HASH, "user_api_key_team_id": TEAM_ID},+ },+ "litellm_params": {+ "metadata": {+ "user_api_key_model_max_budget": budget,+ "model_info": router.get_deployment(model_id="team-opus-deployment").model_info.model_dump(),+ }+ },+ },+ )++ with pytest.raises(litellm.BudgetExceededError, match=rf"budget_model={re.escape(budget_entry)}$"):+ await limiter.is_key_within_model_budget(user_api_key_dict=team_key, model="claude-opus-4-8")+++@pytest.mark.asyncio+async def test_spend_charged_while_the_database_is_unreachable_reaches_the_shared_window_without_wiping_it():+ budget: Final = {"claude-opus*": ROOMY_CAP}+ database: Final = _WindowSpendDatabase()+ clock: Final = _Clock()+ clock.advance(WINDOW_EPOCH)+ replica_a: Final = _family_limiter(shared_windows=database.replica(), clock=clock.now)+ replica_b: Final = _family_limiter(shared_windows=database.replica(), clock=clock.now)++ await _spend_on(replica_a, model="claude-opus-4-8", model_max_budget=budget)+ database.flush()+ clock.advance(60)+ database.reachable = False+ await _spend_on(replica_b, model="claude-opus-4-8", model_max_budget=budget)+ database.reachable = True+ database.flush()+ clock.advance(60)+ await _spend_on(replica_b, model="claude-opus-5-5", model_max_budget=budget)+ database.flush()++ usage: Final = await build_model_max_budget_usage(+ entity_type=Litellm_EntityType.KEY,+ entity_id=KEY_HASH,+ model_max_budget=budget,+ cache=DualCache(),+ now=datetime.fromtimestamp(clock.now(), tz=timezone.utc),+ shared_windows=database.replica(),+ )+ assert usage["claude-opus*"]["current_spend"] == round(CHAT_COST * 3, 4)+ assert usage["claude-opus*"]["budget_reset_at"] == datetime.fromtimestamp(+ WINDOW_EPOCH + 24 * 3600, tz=timezone.utc+ ).isoformat()+++@pytest.mark.asyncio+async def test_end_user_windows_stay_on_the_replica_without_redis():+ budget: Final = {"claude-opus*": FAMILY_CAP}+ table: Final = _WindowSpendDatabase()+ limiter: Final = _family_limiter(shared_windows=table.replica())++ await _log_success(+ limiter,+ {+ "call_type": "acompletion",+ "standard_logging_object": {+ "call_type": "acompletion",+ "response_cost": CHAT_COST,+ "model": "anthropic/claude-opus-4-8",+ "model_group": "claude-opus-4-8",+ "end_user": "end-user-1",+ "metadata": {},+ },+ "litellm_params": {"metadata": {"user_api_key_end_user_model_max_budget": budget}},+ },+ )+ table.flush()++ assert dict(table.rows) == {}+ with pytest.raises(litellm.BudgetExceededError, match="model=claude-opus-5-5"):+ await limiter.is_end_user_within_model_budget(+ end_user_id="end-user-1", end_user_model_max_budget=budget, model="claude-opus-5-5"+ )+++def test_an_access_group_named_after_a_served_model_never_widens_that_models_entry():+ router: Final = Router(+ model_list=[+ {+ "model_name": "gpt-4o",+ "litellm_params": {"model": "openai/gpt-4o", "api_key": "sk-test"},+ "model_info": {"access_groups": ["gpt-4o", "openai-models"]},+ },+ {+ "model_name": "gpt-4o-mini",+ "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "sk-test"},+ "model_info": {"access_groups": ["gpt-4o", "openai-models"]},+ },+ ]+ )++ def governing_entry(model_max_budget: Mapping[str, object]) -> str | None:+ resolved: Final = resolve_model_budget(+ model="gpt-4o-mini",+ model_max_budget=model_max_budget,+ model_access_groups=lambda model, team_id: model_access_groups_of(router=router, model=model, team_id=team_id),+ )+ return None if resolved is None else resolved.budget_model++ assert governing_entry({"gpt-4o": FAMILY_CAP}) is None+ assert governing_entry({"openai-models": FAMILY_CAP}) == "openai-models"+ assert routed_models_of(router).members("gpt-4o") == frozenset()+ assert routed_models_of(router).members("openai-models") == frozenset({"gpt-4o", "gpt-4o-mini"})+++@pytest.mark.asyncio+async def test_key_info_usage_reports_the_family_total():+ budget: Final = {"claude-opus*": ROOMY_CAP}+ limiter: Final = _family_limiter()++ await _spend_on(limiter, model="claude-opus-4-8", model_max_budget=budget)+ await _spend_on(limiter, model="claude-opus-5-5", model_max_budget=budget)++ usage: Final = await build_model_max_budget_usage(+ entity_type=Litellm_EntityType.KEY,+ entity_id=KEY_HASH,+ model_max_budget=budget,+ cache=limiter.dual_cache,+ )+ assert usage["claude-opus*"]["current_spend"] == round(CHAT_COST * 2, 4)+++@pytest.mark.parametrize(+ ("pattern", "model", "governs"),+ [+ ("claude-opus*", "claude-opus-4-8", True),+ ("claude-opus*", "anthropic/claude-opus-4-8", True),+ ("anthropic/claude-opus*", "anthropic/claude-opus-4-8", True),+ ("anthropic/claude-opus*", "claude-opus-4-8", False),+ ("claude-opus*", "my-claude-opus-4-8", False),+ ("claude-opus*", "claude-sonnet-5", False),+ ("gpt-4.1*", "gpt-401-mini", False),+ ("*", "claude-sonnet-5", True),+ ],+)+def test_glob_entries_match_the_whole_model_name_with_only_star_as_a_wildcard(pattern: str, model: str, governs: bool):+ resolved: Final = resolve_model_budget(+ model=model, model_max_budget={pattern: FAMILY_CAP}, model_access_groups=_model_access_groups({})+ )++ assert (resolved is not None and resolved.budget_model == pattern) is governs+++@pytest.mark.asyncio+async def test_replicas_without_redis_enforce_one_family_cap_together():+ budget: Final = {"claude-opus*": FAMILY_CAP}+ table: Final = _WindowSpendDatabase()+ replica_a: Final = _family_limiter(shared_windows=table.replica())+ replica_b: Final = _family_limiter(shared_windows=table.replica())++ await _spend_on(replica_a, model="claude-opus-4-8", model_max_budget=budget)+ table.flush()++ with pytest.raises(litellm.BudgetExceededError, match="model=claude-opus-5-5"):+ await replica_b.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-opus-5-5")+++@pytest.mark.asyncio+async def test_replicas_sharing_redis_enforce_the_family_cap_without_the_database():+ budget: Final = {"claude-opus*": FAMILY_CAP}+ redis: Final = _SharedRedisDouble()+ table: Final = _WindowSpendDatabase()+ replica_a: Final = _family_limiter(+ dual_cache=DualCache(redis_cache=redis), # pyright: ignore[reportArgumentType] # duck-typed Redis double+ shared_windows=table.replica(),+ )+ replica_b: Final = _family_limiter(+ dual_cache=DualCache(redis_cache=redis), # pyright: ignore[reportArgumentType] # duck-typed Redis double+ shared_windows=table.replica(),+ )++ await _spend_on(replica_a, model="claude-opus-4-8", model_max_budget=budget)+ await _drain_redis_pushes()+ table.flush()++ with pytest.raises(litellm.BudgetExceededError, match="model=claude-opus-5-5"):+ await replica_b.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-opus-5-5")+ assert dict(table.rows) == {}+++@pytest.mark.asyncio+async def test_a_replica_charges_the_open_shared_window_instead_of_restarting_it():+ budget: Final = {"claude-opus*": {"budget_limit": CHAT_COST * 1.5, "time_period": "1d"}}+ table: Final = _WindowSpendDatabase()+ clock: Final = _Clock()+ clock.advance(WINDOW_EPOCH)+ replica_a: Final = _family_limiter(shared_windows=table.replica(), clock=clock.now)+ replica_b: Final = _family_limiter(shared_windows=table.replica(), clock=clock.now)++ await _spend_on(replica_a, model="claude-opus-4-8", model_max_budget=budget)+ table.flush()+ clock.advance(3600)+ await _spend_on(replica_b, model="claude-opus-5-5", model_max_budget=budget)+ table.flush()++ for replica in (replica_a, replica_b):+ with pytest.raises(litellm.BudgetExceededError):+ await replica.is_key_within_model_budget(user_api_key_dict=_key(budget), model="claude-opus-5-5")+++@pytest.mark.asyncio+async def test_key_info_on_one_replica_reports_spend_from_every_replica():+ budget: Final = {"claude-opus*": ROOMY_CAP}+ table: Final = _WindowSpendDatabase()+ replica_a: Final = _family_limiter(shared_windows=table.replica())+ replica_b: Final = _family_limiter(shared_windows=table.replica())++ await _spend_on(replica_a, model="claude-opus-4-8", model_max_budget=budget)+ await _spend_on(replica_b, model="claude-opus-5-5", model_max_budget=budget)+ table.flush()++ usage: Final = await build_model_max_budget_usage(+ entity_type=Litellm_EntityType.KEY,+ entity_id=KEY_HASH,+ model_max_budget=budget,+ cache=replica_a.dual_cache,+ shared_windows=table.replica(),+ )+ assert usage["claude-opus*"]["current_spend"] == round(CHAT_COST * 2, 4)+ assert usage["claude-opus*"]["budget_reset_at"] is not None+++@pytest.mark.asyncio+@pytest.mark.parametrize("budget_model", ["claude-opus*", "claude-opus-4-8"])+async def test_replicas_starting_a_window_moments_apart_keep_each_others_spend(budget_model: str):+ budget: Final = {budget_model: ROOMY_CAP}+ table: Final = _WindowSpendDatabase()+ clock: Final = _Clock()+ clock.advance(WINDOW_EPOCH)+ replica_a: Final = _family_limiter(shared_windows=table.replica(), clock=clock.now)+ replica_b: Final = _family_limiter(shared_windows=table.replica(), clock=clock.now)++ await _spend_on(replica_a, model="claude-opus-4-8", model_max_budget=budget)+ clock.advance(1)+ await _spend_on(replica_b, model="claude-opus-4-8", model_max_budget=budget)+ table.flush()++ usage: Final = await build_model_max_budget_usage(+ entity_type=Litellm_EntityType.KEY,+ entity_id=KEY_HASH,+ model_max_budget=budget,+ cache=DualCache(),+ now=datetime.fromtimestamp(clock.now(), tz=timezone.utc),+ shared_windows=table.replica(),+ )+ assert usage[budget_model]["current_spend"] == round(CHAT_COST * 2, 4)+++@pytest.mark.asyncio+async def test_concurrent_first_charges_on_one_replica_both_count():+ budget: Final = {"claude-opus*": ROOMY_CAP}+ clock: Final = _Clock()+ clock.advance(WINDOW_EPOCH)+ limiter: Final = _family_limiter(shared_windows=_WindowSpendDatabase().replica(), clock=clock.now)++ await asyncio.gather(+ _spend_on(limiter, model="claude-opus-4-8", model_max_budget=budget),+ _spend_on(limiter, model="claude-opus-5-5", model_max_budget=budget),+ )++ usage: Final = await build_model_max_budget_usage(+ entity_type=Litellm_EntityType.KEY,+ entity_id=KEY_HASH,+ model_max_budget=budget,+ cache=limiter.dual_cache,+ now=datetime.fromtimestamp(clock.now(), tz=timezone.utc),+ shared_windows=_WindowSpendDatabase().replica(),+ )+ assert usage["claude-opus*"]["current_spend"] == round(CHAT_COST * 2, 4)tests/unit/proxy/management_endpoints/test_key_management_endpoints.pyindex 9634f1efce8..b13a7bfc33e 100644@@ -16580,6 +16580,7 @@ async def test_info_key_fn_includes_model_max_budget_usage(monkeypatch): } mock_prisma_client = AsyncMock()+ mock_prisma_client.db.litellm_budgetwindowspend.find_unique = AsyncMock(return_value=None) monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", mock_prisma_client) mock_user_api_key_cache = AsyncMock() monkeypatch.setattr(@@ -16703,6 +16704,7 @@ async def test_info_key_fn_v2_includes_model_max_budget_usage(monkeypatch): model_max_budget = {"gpt-4o": {"budget_limit": 1.00, "time_period": "7d"}} mock_prisma_client = AsyncMock()+ mock_prisma_client.db.litellm_budgetwindowspend.find_unique = AsyncMock(return_value=None) monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", mock_prisma_client) mock_user_api_key_cache = AsyncMock() monkeypatch.setattr(@@ -16767,6 +16769,7 @@ async def test_info_key_fn_budget_table_fallback(monkeypatch): } mock_prisma_client = AsyncMock()+ mock_prisma_client.db.litellm_budgetwindowspend.find_unique = AsyncMock(return_value=None) monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", mock_prisma_client) mock_user_api_key_cache = AsyncMock() monkeypatch.setattr(@@ -16840,6 +16843,7 @@ async def test_info_key_fn_v2_budget_table_fallback(monkeypatch): } mock_prisma_client = AsyncMock()+ mock_prisma_client.db.litellm_budgetwindowspend.find_unique = AsyncMock(return_value=None) monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", mock_prisma_client) mock_user_api_key_cache = AsyncMock() monkeypatch.setattr(@@ -17215,6 +17219,7 @@ async def test_info_key_fn_reads_the_configured_budget_model_key(monkeypatch): } mock_prisma_client = AsyncMock()+ mock_prisma_client.db.litellm_budgetwindowspend.find_unique = AsyncMock(return_value=None) monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", mock_prisma_client) mock_user_api_key_cache = AsyncMock() monkeypatch.setattr(tests/unit/proxy/management_endpoints/test_team_endpoints.pyindex f727448f3ac..e51f26bf1c3 100644@@ -44,6 +44,7 @@ from litellm.proxy._types import ( UpdateTeamRequest, UserAPIKeyAuth, # Import UserAPIKeyAuth )+from litellm.proxy.hooks.model_max_budget_limiter import RoutedModels, routed_models_of from litellm.proxy.management.teams.authz import TeamAccess from litellm.proxy.management_endpoints.team_admin_field_permissions import ( PROXY_CONTROLLED_TEAM_METADATA_KEYS,@@ -17570,6 +17571,26 @@ def test_team_admin_cannot_loosen_team_model_caps(requested) -> None: assert "proxy admin" in exc.value.detail["error"] +def test_team_cap_refusal_names_the_entry_holding_the_cap() -> None:+ from litellm.proxy.management_endpoints.team_endpoints import _check_team_model_budget_update_authority++ with pytest.raises(HTTPException) as exc:+ _check_team_model_budget_update_authority(+ data=UpdateTeamRequest(+ team_id="t1",+ model_max_budget={+ **_EXISTING_TEAM_MODEL_CAPS,+ "openai/gpt-4o": {"max_budget": 1000.0, "budget_duration": "1d"},+ },+ ),+ user_api_key_dict=UserAPIKeyAuth(user_role=LitellmUserRoles.INTERNAL_USER, user_id="team-admin"),+ existing_model_max_budget=_EXISTING_TEAM_MODEL_CAPS,+ )+ assert exc.value.status_code == 403+ assert "for 'openai/gpt-4o'" in exc.value.detail["error"]+ assert "(entry 'gpt-4o')" in exc.value.detail["error"]++ @pytest.mark.parametrize( "requested", [@@ -17593,6 +17614,253 @@ def test_team_admin_can_tighten_or_keep_team_model_caps(requested) -> None: ) +_TEAM_OPUS_FAMILY_CAP: Final = MappingProxyType({"claude-opus*": {"max_budget": 10.0, "budget_duration": "1d"}})+_TEAM_OPUS_GROUP_CAP: Final = MappingProxyType({"z-opus-group": {"max_budget": 10.0, "budget_duration": "1d"}})+_TEAM_ROUTED_MODELS: Final = RoutedModels(+ served_names=frozenset(+ {"claude-opus-4-8", "claude-opus-5-5", "gemini-3-pro", "gpt-4o", "my-team/gpt-4o", "team-route/*"}+ ),+ group_members=MappingProxyType(+ {+ "opus-group": frozenset({"claude-opus-4-8"}),+ "z-opus-group": frozenset({"claude-opus-4-8"}),+ "a-opus-group": frozenset({"claude-opus-4-8", "claude-opus-5-5"}),+ "zz-opus-group": frozenset({"claude-opus-4-8"}),+ "a-gemini-group": frozenset({"gemini-3-pro"}),+ "z-anthropic-group": frozenset({"anthropic/*"}),+ }+ ),+)+++def _first_bedrock_anthropic_model(model_cost: Mapping[str, Mapping[str, object]]) -> str:+ return next(+ model.removeprefix("anthropic.")+ for model, cost in model_cost.items()+ if model.startswith("anthropic.") and str(cost.get("litellm_provider", "")).startswith("bedrock")+ )+++_BEDROCK_ANTHROPIC_MODEL: Final = _first_bedrock_anthropic_model(litellm.model_cost)+_TEAM_BEDROCK_VENDOR_CAP: Final = MappingProxyType({"anthropic.*": {"max_budget": 10.0, "budget_duration": "1d"}})+_BEDROCK_ROUTE_ADDED_ENTRIES: Final = (_BEDROCK_ANTHROPIC_MODEL, f"{_BEDROCK_ANTHROPIC_MODEL[:12]}*")+++def _check_team_admin_model_caps(+ existing: Mapping[str, object],+ requested: Mapping[str, object],+ routed_models: RoutedModels = _TEAM_ROUTED_MODELS,+) -> None:+ from litellm.proxy.management_endpoints.team_endpoints import _check_team_model_budget_update_authority++ _check_team_model_budget_update_authority(+ data=UpdateTeamRequest(team_id="t1", model_max_budget=dict(requested)),+ user_api_key_dict=UserAPIKeyAuth(user_role=LitellmUserRoles.INTERNAL_USER, user_id="team-admin"),+ existing_model_max_budget=existing,+ routed_models_source=lambda _team_id: routed_models,+ )+++@pytest.mark.parametrize(+ ("existing", "added"),+ [+ (_TEAM_OPUS_FAMILY_CAP, {"opus-group": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_FAMILY_CAP, {"opus-group": {"max_budget": 10.0, "budget_duration": "1h"}}),+ (_TEAM_OPUS_FAMILY_CAP, {"*laude-opus-4-8": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_GROUP_CAP, {"a-opus-group": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_GROUP_CAP, {"claude-opus-4-8": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (+ {"z-anthropic-group": {"max_budget": 10.0, "budget_duration": "1d"}},+ {"claude-opus-4-8": {"max_budget": 1000.0, "budget_duration": "1d"}},+ ),+ (_TEAM_OPUS_FAMILY_CAP, {"claude-opus-4-8": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (+ {"anthropic/*": {"max_budget": 10.0, "budget_duration": "1d"}},+ {"claude-opus-4-8": {"max_budget": 1000.0, "budget_duration": "1d"}},+ ),+ (+ {"my-team/*": {"max_budget": 10.0, "budget_duration": "1d"}},+ {"gpt-4o": {"max_budget": 1000.0, "budget_duration": "1d"}},+ ),+ (+ {"team-route/*": {"max_budget": 10.0, "budget_duration": "1d"}},+ {"gpt-4o": {"max_budget": 1000.0, "budget_duration": "1d"}},+ ),+ (_TEAM_OPUS_FAMILY_CAP, {"claude-opus-4-8": {"max_budget": 10.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_FAMILY_CAP, {"claude-opus-4*": {"max_budget": 10.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_FAMILY_CAP, {"opus-group": {"max_budget": 5.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_FAMILY_CAP, {"opus-group": {"max_budget": 10.0, "budget_duration": "2d"}}),+ (_TEAM_OPUS_FAMILY_CAP, {"opus-group": {"max_budget": 0.01, "budget_duration": "1h"}}),+ (_TEAM_OPUS_FAMILY_CAP, {"claude-opus-4-8": {"rpm_limit": 10}}),+ ],+ ids=[+ "group_over_pattern",+ "group_with_a_faster_window",+ "more_specific_pattern",+ "earlier_group_sharing_a_model",+ "model_inside_group",+ "model_under_wildcard_group",+ "model_matching_pattern",+ "model_under_provider_pattern",+ "model_under_served_prefix",+ "model_under_wildcard_route",+ "model_carved_out_at_the_same_cap",+ "pattern_carved_out_at_the_same_cap",+ "tighter_group",+ "group_with_a_slower_window",+ "group_with_a_faster_window_within_the_cap",+ "uncapped_model",+ ],+)+def test_team_admin_cannot_add_an_entry_that_takes_precedence_over_a_cap(+ existing: Mapping[str, object], added: Mapping[str, object]+) -> None:+ with pytest.raises(HTTPException) as exc:+ _check_team_admin_model_caps(existing=existing, requested={**existing, **added})+ assert exc.value.status_code == 403+ assert "proxy admin" in exc.value.detail["error"]+ assert repr(next(iter(existing))) in exc.value.detail["error"]+++@pytest.mark.parametrize(+ ("existing", "requested"),+ [+ (_TEAM_OPUS_FAMILY_CAP, {**_TEAM_OPUS_FAMILY_CAP, "claude*": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_GROUP_CAP, {**_TEAM_OPUS_GROUP_CAP, "a-gemini-group": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_GROUP_CAP, {**_TEAM_OPUS_GROUP_CAP, "zz-opus-group": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_GROUP_CAP, {**_TEAM_OPUS_GROUP_CAP, "gpt-4o": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (_TEAM_OPUS_FAMILY_CAP, {**_TEAM_OPUS_FAMILY_CAP, "gpt-4o": {"max_budget": 1000.0, "budget_duration": "1d"}}),+ (+ _TEAM_OPUS_FAMILY_CAP,+ {**_TEAM_OPUS_FAMILY_CAP, "a-gemini-group": {"max_budget": 1000.0, "budget_duration": "1d"}},+ ),+ (+ {**_TEAM_OPUS_FAMILY_CAP, "opus-group": {"max_budget": 100.0, "budget_duration": "1d"}},+ {**_TEAM_OPUS_FAMILY_CAP, "opus-group": {"max_budget": 100.0, "budget_duration": "1d"}},+ ),+ (+ {**_TEAM_OPUS_FAMILY_CAP, "opus-group": {"max_budget": 100.0, "budget_duration": "1d"}},+ {**_TEAM_OPUS_FAMILY_CAP, "opus-group": {"max_budget": 50.0, "budget_duration": "1d"}},+ ),+ (+ {**_TEAM_OPUS_FAMILY_CAP, "opus-group": {"rpm_limit": 10}},+ {**_TEAM_OPUS_FAMILY_CAP, "opus-group": {"rpm_limit": 20}},+ ),+ (+ {**_TEAM_OPUS_FAMILY_CAP, "claude-opus-4-8": {"rpm_limit": 10, "budget_duration": "1d"}},+ {**_TEAM_OPUS_FAMILY_CAP, "claude-opus-4-8": {"max_budget": 1000.0, "budget_duration": "1d"}},+ ),+ (+ _TEAM_OPUS_FAMILY_CAP,+ {**_TEAM_OPUS_FAMILY_CAP, "gemini-3-pro*": {"max_budget": 1000.0, "budget_duration": "1d"}},+ ),+ (+ {**_TEAM_OPUS_FAMILY_CAP, "gpt-4o": {"max_budget": 10.0}},+ {**_TEAM_OPUS_FAMILY_CAP, "gpt-4o": {"max_budget": 10.0, "budget_duration": "1d"}},+ ),+ (+ {"z-anthropic-group": {"max_budget": 10.0}},+ {"z-anthropic-group": {"max_budget": 10.0, "budget_duration": "1d"}},+ ),+ ],+ ids=[+ "less_specific_pattern",+ "group_without_shared_models",+ "later_group_sharing_a_model",+ "model_outside_group",+ "model_outside_pattern",+ "group_outside_pattern",+ "unchanged_looser_entry",+ "tightened_looser_entry",+ "uncapped_entry_edited",+ "enforced_uncapped_entry_capped",+ "pattern_sharing_only_a_wildcard_route",+ "unenforced_capped_model_given_a_duration",+ "unenforced_capped_wildcard_group_given_a_duration",+ ],+)+def test_team_admin_can_keep_or_tighten_entries_and_add_ones_that_outrank_no_cap(+ existing: Mapping[str, object], requested: Mapping[str, object]+) -> None:+ assert _check_team_admin_model_caps(existing=existing, requested=requested) is None+++def test_team_admin_cannot_make_an_unenforced_entry_take_precedence_over_a_cap() -> None:+ existing: Final = {**_TEAM_OPUS_FAMILY_CAP, "claude-opus-4-8": {"rpm_limit": 10}}++ with pytest.raises(HTTPException) as exc:+ _check_team_admin_model_caps(+ existing=existing,+ requested={**existing, "claude-opus-4-8": {"max_budget": 1000.0, "budget_duration": "1d"}},+ )+ assert exc.value.status_code == 403+ assert "'claude-opus*'" in exc.value.detail["error"]+++def test_team_admin_cannot_raise_an_unenforced_cap_while_enforcing_it() -> None:+ with pytest.raises(HTTPException) as exc:+ _check_team_admin_model_caps(+ existing={"gpt-4o": {"max_budget": 10.0}},+ requested={"gpt-4o": {"max_budget": 20.0, "budget_duration": "1d"}},+ )+ assert exc.value.status_code == 403+ assert "(entry 'gpt-4o')" in exc.value.detail["error"]+++@pytest.mark.parametrize("added_entry", _BEDROCK_ROUTE_ADDED_ENTRIES, ids=["model", "pattern"])+def test_team_admin_cannot_lift_a_bedrock_vendor_cap_through_a_bedrock_wildcard_route(added_entry: str) -> None:+ with pytest.raises(HTTPException) as exc:+ _check_team_admin_model_caps(+ existing=_TEAM_BEDROCK_VENDOR_CAP,+ requested={**_TEAM_BEDROCK_VENDOR_CAP, added_entry: {"max_budget": 1000.0, "budget_duration": "1d"}},+ routed_models=RoutedModels(+ served_names=frozenset({"bedrock/*", "gpt-4o"}), group_members=MappingProxyType({})+ ),+ )+ assert exc.value.status_code == 403+ assert "'anthropic.*'" in exc.value.detail["error"]+++@pytest.mark.parametrize("added_entry", _BEDROCK_ROUTE_ADDED_ENTRIES, ids=["model", "pattern"])+def test_team_admin_can_add_entries_beside_a_bedrock_vendor_cap_no_route_serves(added_entry: str) -> None:+ assert (+ _check_team_admin_model_caps(+ existing=_TEAM_BEDROCK_VENDOR_CAP,+ requested={**_TEAM_BEDROCK_VENDOR_CAP, added_entry: {"max_budget": 1000.0, "budget_duration": "1d"}},+ routed_models=RoutedModels(served_names=frozenset({"gpt-4o"}), group_members=MappingProxyType({})),+ )+ is None+ )+++def test_team_admin_cannot_outrank_a_group_cap_through_a_team_model_both_groups_serve() -> None:+ from litellm.proxy.management_endpoints.team_endpoints import _check_team_model_budget_update_authority++ team_router: Final = Router(+ model_list=[+ {+ "model_name": f"model_name_t1_{group}",+ "litellm_params": {"model": "anthropic/claude-opus-4-8", "api_key": "sk-test"},+ "model_info": {"team_id": "t1", "team_public_model_name": "claude-opus-4-8", "access_groups": [group]},+ }+ for group in ("a-opus", "z-opus")+ ]+ )+ existing: Final = {"z-opus": {"max_budget": 10.0, "budget_duration": "1d"}}++ with pytest.raises(HTTPException) as exc:+ _check_team_model_budget_update_authority(+ data=UpdateTeamRequest(+ team_id="t1", model_max_budget={**existing, "a-opus": {"max_budget": 100.0, "budget_duration": "1d"}}+ ),+ user_api_key_dict=UserAPIKeyAuth(user_role=LitellmUserRoles.INTERNAL_USER, user_id="team-admin"),+ existing_model_max_budget=existing,+ routed_models_source=lambda team_id: routed_models_of(router=team_router, team_id=team_id),+ )+ assert exc.value.status_code == 403+ assert "'z-opus'" in exc.value.detail["error"]++ def test_team_model_cap_authority_skips_omitted_field_malformed_rows_and_proxy_admins() -> None: from litellm.proxy.management_endpoints.team_endpoints import _check_team_model_budget_update_authority tests/unit/test_internal_context.pyindex f78c6064f76..3336cce9aaf 100644@@ -74,6 +74,7 @@ _IN_MEMORY_ONLY_CALLERS: Final = frozenset( "litellm/proxy/container_endpoints/ownership.py", "litellm/proxy/discovery_endpoints/agent_skills_endpoints.py", "litellm/proxy/guardrails/guardrail_hooks/straiker/straiker.py",+ "litellm/proxy/hooks/model_budget_shared_windows.py", "litellm/proxy/spend_tracking/key_metadata_recovery.py", "litellm/proxy/ui_crud_endpoints/latest_release_endpoints.py", "litellm/responses/litellm_completion_transformation/transformation.py",tests/unit/test_router/test_router.pyindex b4a3d207bd4..3c7a222d9a1 100644@@ -1126,6 +1126,36 @@ def test_arouter_test_team_model(): assert result is not None +def test_team_public_model_names_of_lists_only_that_teams_live_public_names():+ def team_deployment(deployment_id: str, team_id: str, public_model_name: str) -> DeploymentTypedDict:+ return {+ "model_name": f"model_name_{team_id}_{deployment_id}",+ "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "sk-test"},+ "model_info": {"id": deployment_id, "team_id": team_id, "team_public_model_name": public_model_name},+ }++ router: Final = litellm.Router(+ model_list=[+ team_deployment("team-a-1", "team-a", "shared-model"),+ team_deployment("team-a-2", "team-a", "shared-model"),+ team_deployment("team-a-solo", "team-a", "solo-model"),+ team_deployment("team-b-1", "team-b", "team-b-model"),+ {+ "model_name": "plain-model",+ "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "sk-test"},+ "model_info": {"id": "plain-1"},+ },+ ],+ )++ assert router.team_public_model_names_of("team-a") == frozenset({"shared-model", "solo-model"})+ assert router.team_public_model_names_of("team-c") == frozenset()++ router.delete_deployment(id="team-a-solo")++ assert router.team_public_model_names_of("team-a") == frozenset({"shared-model"})++ def test_team_model_has_alternatives(): def team_deployment( deployment_id: str, team_id: str, public_model_name: str, blocked: bool = Falseui/litellm-dashboard/src/components/key_team_helpers/ModelMaxBudgetEditor.integration.test.tsxindex 33f09092cca..955b911ca97 100644@@ -1,5 +1,6 @@+import userEvent from "@testing-library/user-event"; import { describe, expect, it, vi } from "vitest";-import { renderWithProviders, screen } from "../../../tests/test-utils";+import { fireEvent, renderWithProviders, screen } from "../../../tests/test-utils"; import { MODEL_MAX_BUDGET_PREMIUM_HINT, ModelMaxBudgetEditor, type ModelMaxBudget } from "./ModelMaxBudgetEditor"; const STORED: ModelMaxBudget = { "gpt-4o": { budget_limit: 5, time_period: "30d" } };@@ -67,3 +68,65 @@ describe("ModelMaxBudgetEditor with an enterprise license", () => { expect(addButton()).toBeEnabled(); }); });++describe("ModelMaxBudgetEditor model families", () => {+ it("stores a typed pattern that matches no listed model as one budget entry", async () => {+ const user = userEvent.setup();+ const onChange = vi.fn();+ renderWithProviders(+ <ModelMaxBudgetEditor+ value={{}}+ onChange={onChange}+ availableModels={["gpt-4o", "claude-opus-4-8"]}+ premiumUser={true}+ />,+ );++ await user.click(addButton());+ const modelField = screen.getByPlaceholderText("Select model");+ await user.type(modelField, "claude-opus*");+ await user.click(await screen.findByRole("option", { name: /claude-opus\*/ }));+ fireEvent.change(screen.getByPlaceholderText("Max spend ($)"), { target: { value: "10" } });++ expect(onChange).toHaveBeenLastCalledWith({ "claude-opus*": { budget_limit: 10, time_period: "30d" } });+ });++ it("does not offer a typed pattern that another row already budgets", async () => {+ const user = userEvent.setup();+ renderWithProviders(+ <ModelMaxBudgetEditor+ value={{ "claude-opus*": { budget_limit: 10, time_period: "1d" } }}+ onChange={vi.fn()}+ availableModels={["gpt-4o", "claude-opus-4-8"]}+ premiumUser={true}+ />,+ );++ await user.click(addButton());+ const [, newRowModelField] = screen.getAllByPlaceholderText("Select model");+ await user.type(newRowModelField, "claude-opus*");++ expect(screen.queryByRole("option", { name: /claude-opus\*/ })).not.toBeInTheDocument();+ await user.clear(newRowModelField);+ await user.type(newRowModelField, "claude-sonnet*");+ expect(await screen.findByRole("option", { name: /claude-sonnet\*/ })).toBeInTheDocument();+ });++ it("does not offer a typed name that is neither listed nor a pattern", async () => {+ const user = userEvent.setup();+ renderWithProviders(+ <ModelMaxBudgetEditor+ value={{}}+ onChange={vi.fn()}+ availableModels={["gpt-4o", "claude-opus-4-8"]}+ premiumUser={true}+ />,+ );++ await user.click(addButton());+ await user.type(screen.getByPlaceholderText("Select model"), "claude-opsu-4-8");++ expect(screen.queryByRole("option", { name: /claude-opsu-4-8/ })).not.toBeInTheDocument();+ expect(screen.getByText("No models found")).toBeInTheDocument();+ });+});ui/litellm-dashboard/src/components/key_team_helpers/ModelMaxBudgetEditor.tsxindex a6e72068463..02fc022e881 100644@@ -50,6 +50,8 @@ export const MODEL_BUDGET_PERIOD_OPTIONS = [ const DEFAULT_PERIOD = "30d"; +const MODEL_PATTERN_VALUE = { hint: "Model pattern", accepts: (typed: string) => typed.includes("*") };+ export const entriesToModelMaxBudget = (entries: readonly ModelBudgetEntry[]): ModelMaxBudget => Object.fromEntries( entries@@ -121,7 +123,7 @@ export function ModelMaxBudgetEditor({ const blurb = ( <div className="text-xs text-muted-foreground"> {premiumUser- ? "Cap spend per model over its own window. A budget set on the bare model name also covers the provider-prefixed spelling of that model."+ ? "Cap spend per model over its own window. A budget set on the bare model name also covers the provider-prefixed spelling of that model. Pick an access group, or type a pattern like claude-opus*, to share one budget across every model it covers." : MODEL_MAX_BUDGET_PREMIUM_HINT} </div> );@@ -165,6 +167,8 @@ export function ModelMaxBudgetEditor({ onValueChange={(model) => updateEntry(entry.id, { model })} placeholder="Select model" emptyText="No models found"+ customValue={MODEL_PATTERN_VALUE}+ reservedValues={entries.flatMap((other) => (other.id !== entry.id && other.model ? [other.model] : []))} disabled={!premiumUser} /> </div>ui/litellm-dashboard/src/components/shared/SearchSelect.integration.test.tsxindex 9f320049b09..ecb8dd161f3 100644@@ -63,6 +63,24 @@ describe("SearchSelect", () => { expect(onValueChange).toHaveBeenLastCalledWith("team-3"); }); + it("offers the typed text as a value only when the caller accepts it", async () => {+ const user = userEvent.setup();+ const onValueChange = vi.fn();+ const customValue = { hint: "Custom value", accepts: (typed: string) => typed.endsWith("*") };+ const { rerender } = render(<SearchSelect options={OPTIONS} onValueChange={onValueChange} />);+ const input = screen.getByRole("combobox");+ await user.click(input);+ fireEvent.change(input, { target: { value: "team-*" } });+ expect(screen.queryByRole("option", { name: /team-\*/ })).not.toBeInTheDocument();++ rerender(<SearchSelect options={OPTIONS} onValueChange={onValueChange} customValue={customValue} />);+ fireEvent.change(input, { target: { value: "team-4" } });+ expect(screen.queryByRole("option", { name: /team-4/ })).not.toBeInTheDocument();+ fireEvent.change(input, { target: { value: "team-*" } });+ await user.click(await screen.findByRole("option", { name: /team-\*/ }));+ expect(onValueChange).toHaveBeenLastCalledWith("team-*");+ });+ it("filters the options client-side as you type", async () => { const user = userEvent.setup(); render(<SearchSelect options={OPTIONS} onValueChange={vi.fn()} />);ui/litellm-dashboard/src/components/shared/SearchSelect.tsxindex 3649114826d..e139720a2ed 100644@@ -1,6 +1,6 @@ "use client"; -import React from "react";+import React, { useState } from "react"; import { Combobox, ComboboxContent,@@ -19,6 +19,11 @@ export interface SearchSelectOption { icon?: React.ReactNode; } +interface CustomValue {+ hint: string;+ accepts: (typed: string) => boolean;+}+ interface SearchSelectProps { options: SearchSelectOption[]; value?: string | null;@@ -29,6 +34,8 @@ interface SearchSelectProps { className?: string; inputId?: string; allowClear?: boolean;+ customValue?: CustomValue;+ reservedValues?: readonly string[]; "aria-label"?: string; id?: string; }@@ -49,13 +56,22 @@ export function SearchSelect({ className, inputId, allowClear = true,+ customValue,+ reservedValues = [], "aria-label": ariaLabel, id, }: SearchSelectProps) {+ const [query, setQuery] = useState(""); const selected = value == null || value === "" ? null : options.find((option) => option.value === value) ?? { label: value, value };- const items =+ const known = selected !== null && !options.some((option) => option.value === selected.value) ? [selected, ...options] : options;+ const typed = query.trim();+ const isNewValue = typed !== "" && !reservedValues.includes(typed) && !known.some((option) => option.value === typed);+ const items =+ customValue !== undefined && isNewValue && customValue.accepts(typed)+ ? [...known, { label: typed, value: typed, sublabel: customValue.hint }]+ : known; return ( <Combobox@@ -65,6 +81,7 @@ export function SearchSelect({ isItemEqualToValue={(a: SearchSelectOption, b: SearchSelectOption) => a.value === b.value} itemToStringLabel={(item: SearchSelectOption) => item.label} filter={matchesQuery}+ onInputValueChange={setQuery} disabled={disabled} > <ComboboxInputui/litellm-dashboard/src/lib/http/schema.d.tsindex 72cc8cdc94e..a09672d7d8f 100644@@ -8475,7 +8475,7 @@ export interface paths { * - throttle_on_budget_exceeded: Optional[bool] - When the key exceeds its max_budget, throttle its tpm/rpm to the global budget_exceeded_throttle_percentage instead of blocking the key entirely. * - enable_prompt_caching: Optional[bool] - Auto-inject prompt caching breakpoints (Anthropic cache_control markers) on requests made with this key. Supported Claude models on Anthropic, Bedrock, Vertex AI, and Azure AI only. * - permissions: Optional[dict] - key-specific permissions. Currently just used for turning off pii masking (if connected). Example - {"pii": false}- * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget.+ * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget. * - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. * - model_rpm_limit: Optional[dict] - key-specific model rpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific rpm limit. * - model_tpm_limit: Optional[dict] - key-specific model tpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific tpm limit.@@ -8721,7 +8721,7 @@ export interface paths { * - tags: Optional[List[str]] - Tags for organizing keys (Enterprise only) * - spend: Optional[float] - Amount spent by key * - max_budget: Optional[float] - Max budget for key- * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}+ * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}} * - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. * - budget_duration: Optional[str] - Budget reset period ("30d", "1h", etc.) * - soft_budget: Optional[float] - Soft budget limit (warning vs. hard stop). Will trigger a slack alert when this soft budget is reached.@@ -8798,7 +8798,7 @@ export interface paths { * - metadata: Optional[dict] - Metadata for key, store information for key. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" } * - guardrails: Optional[List[str]] - List of active guardrails for the key * - permissions: Optional[dict] - key-specific permissions. Currently just used for turning off pii masking (if connected). Example - {"pii": false}- * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget.+ * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}}. IF null or {} then no model specific budget. * - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. * - model_rpm_limit: Optional[dict] - key-specific model rpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific rpm limit. * - model_tpm_limit: Optional[dict] - key-specific model tpm limit. Example - {"text-davinci-002": 1000, "gpt-3.5-turbo": 1000}. IF null or {} then no model specific tpm limit.@@ -8931,7 +8931,7 @@ export interface paths { * - enforced_params: Optional[List[str]] - List of enforced params for the key (Enterprise only). [Docs](https://docs.litellm.ai/docs/proxy/enterprise#enforce-required-params-for-llm-requests) * - spend: Optional[float] - Amount spent by key * - max_budget: Optional[float] - Max budget for key- * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}+ * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}} * - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. * - budget_duration: Optional[str] - Budget reset period ("30d", "1h", etc.) * - soft_budget: Optional[float] - Soft budget limit (warning vs. hard stop). Will trigger a slack alert when this soft budget is reached. Set to null to remove the soft budget.@@ -9020,7 +9020,7 @@ export interface paths { * - tags: Optional[List[str]] - Tags for organizing keys (Enterprise only) * - spend: Optional[float] - Amount spent by key * - max_budget: Optional[float] - Max budget for key- * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}}+ * - model_max_budget: Optional[Dict[str, BudgetConfig]] - Model-specific budgets keyed by model name, `*` pattern like "claude-opus*", or model access group, e.g. {"gpt-4": {"budget_limit": 0.0005, "time_period": "30d"}} * - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. * - budget_duration: Optional[str] - Budget reset period ("30d", "1h", etc.) * - soft_budget: Optional[float] - Soft budget limit (warning vs. hard stop). Will trigger a slack alert when this soft budget is reached.@@ -17098,7 +17098,7 @@ export interface paths { * - prompts: Optional[List[str]] - List of prompts that the team is allowed to use. * - organization_id: Optional[str] - The organization id of the team. Default is None. Create via `/organization/new`. * - model_aliases: Optional[dict] - Model aliases for the team. [Docs](https://docs.litellm.ai/docs/proxy/team_based_routing#create-team-with-model-alias)- * - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}}+ * - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model, keyed by model name, `*` pattern, or model access group. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}} * - guardrails: Optional[List[str]] - Guardrails for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails) * - policies: Optional[List[str]] - Policies for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_policies) * - disable_global_guardrails: Optional[bool] - Whether to disable global guardrails for the team. Proxy admin only.@@ -17328,7 +17328,7 @@ export interface paths { * - tags: Optional[List[str]] - Tags for [tracking spend](https://litellm.vercel.app/docs/proxy/enterprise#tracking-spend-for-custom-tags) and/or doing [tag-based routing](https://litellm.vercel.app/docs/proxy/tag_routing). * - organization_id: Optional[str] - The organization id of the team. Default is None. Create via `/organization/new`. * - model_aliases: Optional[dict] - Model aliases for the team. [Docs](https://docs.litellm.ai/docs/proxy/team_based_routing#create-team-with-model-alias)- * - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}}+ * - model_max_budget: Optional[dict] - Per-model max budget every key on the team inherits unless the key sets its own for that model, keyed by model name, `*` pattern, or model access group. Example: {"gpt-4o": {"max_budget": 10, "budget_duration": "1d"}} * - guardrails: Optional[List[str]] - Guardrails for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails) * - policies: Optional[List[str]] - Policies for the team. [Docs](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_policies) * - disable_global_guardrails: Optional[bool] - Whether to disable global guardrails for the team. Proxy admin only.@@ -18778,7 +18778,7 @@ export interface paths { * - permissions: Optional[dict] - [Not Implemented Yet] User-specific permissions, eg. turning off pii masking. * - metadata: Optional[dict] - Metadata for user, store information for user. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" } * - max_parallel_requests: Optional[int] - Rate limit a user based on the number of parallel requests. Raises 429 error, if user's parallel requests > x.- * - model_max_budget: Optional[dict] - Model-specific max budget for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys)+ * - model_max_budget: Optional[dict] - Model-specific max budget for user, keyed by model name, `*` pattern, or model access group. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys) * - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. * - model_rpm_limit: Optional[float] - Model-specific rpm limit for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-limits-to-keys) * - mcp_rpm_limit: Optional[dict] - Per-MCP-server rpm limit, keyed by MCP server name {"github": 100, "slack": 200}. Enforced for keys and teams only; values set on a user are stored but not enforced per user.@@ -18917,7 +18917,7 @@ export interface paths { * - permissions: Optional[dict] - [Not Implemented Yet] User-specific permissions, eg. turning off pii masking. * - metadata: Optional[dict] - Metadata for user, store information for user. Example metadata = {"team": "core-infra", "app": "app2", "email": "ishaan@berri.ai" } * - max_parallel_requests: Optional[int] - Rate limit a user based on the number of parallel requests. Raises 429 error, if user's parallel requests > x.- * - model_max_budget: Optional[dict] - Model-specific max budget for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys)+ * - model_max_budget: Optional[dict] - Model-specific max budget for user, keyed by model name, `*` pattern, or model access group. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-budgets-to-keys) * - budget_fallbacks: Optional[Dict[str, List[str]]] - Per-model fallback chain tried in order when that model's own `model_max_budget` is exceeded, e.g. {"gpt-4o": ["gpt-4o-mini"]}. * - model_rpm_limit: Optional[float] - Model-specific rpm limit for user. [Docs](https://docs.litellm.ai/docs/proxy/users#add-model-specific-limits-to-keys) * - mcp_rpm_limit: Optional[dict] - Per-MCP-server rpm limit, keyed by MCP server name {"github": 100, "slack": 200}. Enforced for keys and teams only; values set on a user are stored but not enforced per user.