[Bug]: cache_control blocks silently dropped when routing through SAP AI Core Orchestration (sap/ prefix)

[gaogegolf]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When calling ChatLiteLLM with a model on the SAP AI Core Orchestration path (sap/ prefix, e.g. sap/anthropic--claude-4.5-haiku), Anthropic's cache_control parameter is silently stripped before the request reaches Anthropic. No error or warning is raised — the API call succeeds, but cache_read_input_tokens and cache_creation_input_tokens are both 0 on every call regardless of what cache_control markers are set in the messages.

Expected: cache_creation_input_tokens > 0 on the first call with a cached prefix, cache_read_input_tokens > 0 on repeat calls with the same prefix.

User Flow

Before a fix: a developer sets cache_control on the system prompt and last message, calls ChatLiteLLM with model="sap/anthropic--claude-4.5-haiku", and expects caching to activate.

1. They construct messages with cache_control: {"type": "ephemeral"} on the system block and last user message
2. They call llm.invoke(messages) via ChatLiteLLM
3. The response metadata shows cache_creation_input_tokens: 0 and cache_read_input_tokens: 0
4. On the second identical call, cache_read_input_tokens is still 0 — caching never activated

After a fix: the same call with cache_control set produces cache_creation_input_tokens > 0 on the first call and cache_read_input_tokens > 0 on the second.

1. They construct the same messages with cache_control markers
2. They call llm.invoke(messages) via ChatLiteLLM
3. The response metadata shows cache_creation_input_tokens > 0
4. On the second call with the same prefix, cache_read_input_tokens > 0

Proof the bug occurs

Config / setup:

• litellm==1.83.14, langchain-litellm==0.6.4, Python 3.13
• Provider: SAP AI Core Orchestration (sap/ prefix routing to Anthropic)
• Not reproducible with a public LiteLLM proxy — requires SAP AI Core credentials

We verified the stripping happens in the LiteLLM/orchestration layer by bypassing it: we wrote a custom BaseChatModel that POSTs the same payload directly to the native Anthropic Messages API via an AI Core foundation-models /invoke deployment. With that path:

• cache_control is preserved
• cache_creation_input_tokens > 0 on first call
• 97.2% cache hit rate and 87% effective input-token cost reduction measured across 169 LLM calls
  (5-iteration aeval run)

The payload we send on the working path (for reference):

{
"system": [{"type": "text", "text": "...", "cache_control": {"type": "ephemeral"}}],
"messages": [
  ...,
  {"role": "user", "content": [{"type": "text", "text": "...", "cache_control": {"type": "ephemeral"}}]}
]
}

The same payload routed through ChatLiteLLM (sap/ prefix) results in cache_creation_input_tokens: 0.

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.83.14

Twitter / LinkedIn details

No response