[Bug]: provider budget never counts embedding or rerank spend — RouterBudgetLimiting raises "custom_llm_provider is required"

[danielgines]

What happened?

With router_settings.provider_budget_config enabled, every /v1/embeddings and /v1/rerank call raises inside the budget callback:

LiteLLM:ERROR: litellm_logging.py:2672 - LiteLLM.LoggingError: [Non-Blocking] Exception occurred while success logging
  File ".../litellm/litellm_core_utils/litellm_logging.py", line 2618, in async_success_handler
    await callback.async_log_success_event(...)
  File ".../litellm/router_strategy/budget_limiter.py", line 412, in async_log_success_event
    raise ValueError("custom_llm_provider is required")
ValueError: custom_llm_provider is required

The error is labelled non-blocking and the request succeeds, so it looks cosmetic. It is not: RouterBudgetLimiting.async_log_success_event raises before _increment_spend_for_key, so embedding and rerank spend is never added to the provider budget. The ceiling silently under-counts, and a deployment that is mostly embeddings would never reach it.

/v1/chat/completions is unaffected — its spend does accumulate.

Root cause

budget_limiter.py:410 reads the provider from litellm_params:

custom_llm_provider: str = kwargs.get("litellm_params", {}).get("custom_llm_provider", None)
if custom_llm_provider is None:
    raise ValueError("custom_llm_provider is required")

completion() puts it there (main.py:5289):

litellm_params = get_litellm_params(
    ...
    custom_llm_provider=custom_llm_provider,
    ...
)

embedding() does not (main.py:6028):

litellm_params_dict = get_litellm_params(**kwargs)

custom_llm_provider is a named parameter of embedding(), so it is never in **kwargs; and it has already been resolved a few lines above by get_llm_provider() — it is passed to logging.update_environment_variables(custom_llm_provider=...) on the very next statement, just not into litellm_params.

Same shape at main.py:7548 (transcription) and main.py:7825 (speech). rerank_api/main.py:168 builds rerank_litellm_params without it as well.

Because the parameter is consumed by the function signature, no configuration can work around this — passing custom_llm_provider in the request body or in the deployment's litellm_params still ends up as the named argument, never in **kwargs.

Relevant log output

Reproduced 3/3 on /v1/embeddings and /v1/rerank; 0/4 on /v1/chat/completions (streaming and non-streaming, cache hit and miss).

GET /provider/budgets on the same proxy shows spend accumulating from chat only:

{"providers":{"bedrock":{"budget_limit":1000.0,"time_period":"1mo","spend":159.18631146,...}}}

Suggested fix

Pass the already-resolved value, as completion() does:

litellm_params_dict = get_litellm_params(custom_llm_provider=custom_llm_provider, **kwargs)

Config

model_list:
  - model_name: titan-embed-v2
    litellm_params:
      model: bedrock/amazon.titan-embed-text-v2:0
      aws_region_name: us-east-1
  - model_name: cohere-rerank-v3-5
    litellm_params:
      model: bedrock/arn:aws:bedrock:us-east-1::foundation-model/cohere.rerank-v3-5:0
      aws_region_name: us-east-1

router_settings:
  provider_budget_config:
    bedrock:
      budget_limit: 1000
      time_period: 1mo
  redis_host: os.environ/REDIS_HOST

Twitter / LinkedIn details

No response

[Apricooooot]

If no one is currently working on this, I’d be happy to submit a focused fix with regression tests for the missing provider metadata in embedding and rerank calls.

[NetBr3ak]

I'd call this a correctness bug rather than a reporting one. A budget that
skips a spend category doesn't just report low, it fails open: the number
you're enforcing isn't the number you get invoiced for.

There's a second field with the same problem, if anyone is auditing this
area. On Bedrock, inputTokens leaves cached tokens out, so a cost formula
built on it misses caching entirely. The real total is inputTokens plus
cacheReadInputTokens plus cacheWriteInputTokens. With writes at 1.25 times
input and reads at 0.1, that omission runs either way depending on the hit
rate.

Different field, same ending: the tool says one thing and the invoice says
another.