[Bug]: stream_chunk_builder raises IndexError/KeyError in build_base_response, truncating client SSE stream and skipping SpendLogs

[ZXT-zjbiliy]

What happened?

ChunkProcessor.build_base_response() reads the assistant role without guarding either the choices array bounds or the presence of a role key:

# litellm/litellm_core_utils/streaming_chunk_builder_utils.py
first_chunk_with_choices: Final = next((c for c in chunks if c.get("choices")), chunk)
role: Final = first_chunk_with_choices["choices"][0]["delta"]["role"]

Two distinct failures follow:

1. IndexError: list index out of range — when no chunk carries a non-empty choices array, next() returns its default (the first chunk), whose choices is []. The [0] access then goes out of bounds.
2. KeyError: 'role' — when the first choice's delta omits role or is {}.

Either one propagates out of stream_chunk_builder() as:

litellm.APIError: Error building chunks for logging/streaming usage calculation

proxy_server.async_data_generator() catches it and writes it into the response stream:

error_returned = json.dumps({"error": proxy_exception.to_dict()})
yield f"data: {error_returned}\n\n"

Control then goes straight to finally, so data: [DONE] is never emitted.

Note the loop directly below the offending line already guards correctly with len(chunk["choices"]) > 0, so this looks like a simple omission rather than intent.

Impact

This is not only a logging-path bug. Two user-visible consequences:

• The client's stream is truncated. The user has already received part of the answer, then the stream stops and an error frame arrives with no [DONE]. HTTP status is still 200, because SSE headers are flushed before the failure — so this does not show up as a 5xx in proxy access logs.
• The request is missing from SpendLogs, so it is never billed.

In our production proxy this hit Anthropic/Claude streaming intermittently — a handful of requests per day across a week. Successful responses in the same second were 19KB–339KB; the failing ones returned only ~360 bytes (just the error frame) after 8–38s.

Relevant log output

LiteLLM:ERROR: main.py:7815 - litellm.main.py::stream_chunk_builder() - Exception occurred - list index out of range
Traceback (most recent call last):
  File "litellm/litellm_core_utils/streaming_handler.py", line 2147, in __anext__
    raise StopAsyncIteration
StopAsyncIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "litellm/main.py", line 7561, in stream_chunk_builder
    response = processor.build_base_response(chunks)
  File "litellm/litellm_core_utils/streaming_chunk_builder_utils.py", line 123, in build_base_response
    role = first_chunk_with_choices["choices"][0]["delta"]["role"]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

LiteLLM Proxy:ERROR: proxy_server.py:7081 - litellm.proxy.proxy_server.async_data_generator(): Exception occured - litellm.APIError: Error building chunks for logging/streaming usage calculation

Reproduction

from litellm.litellm_core_utils.streaming_chunk_builder_utils import ChunkProcessor

def mk(choices, **kw):
    d = {"id": "c1", "object": "chat.completion.chunk", "created": 1,
         "model": "claude-opus-4-8", "choices": choices}
    d.update(kw)
    return d

cases = {
    "all-empty-choices":  [mk([]), mk([])],
    "usage-only frames":  [mk([], usage={"prompt_tokens": 10}), mk([], usage={"completion_tokens": 0})],
    "delta without role": [mk([{"index": 0, "delta": {"content": "Hi"}, "finish_reason": None}])],
    "delta empty dict":   [mk([{"index": 0, "delta": {}, "finish_reason": None}])],
}

for name, chunks in cases.items():
    try:
        ChunkProcessor(chunks=list(chunks)).build_base_response(list(chunks))
        print(f"[ok]   {name}")
    except Exception as e:
        print(f"[fail] {name} -> {type(e).__name__}: {e}")

Output on v1.88.1 and on current litellm_internal_staging:

[fail] all-empty-choices  -> IndexError: list index out of range
[fail] usage-only frames  -> IndexError: list index out of range
[fail] delta without role -> KeyError: 'role'
[fail] delta empty dict   -> KeyError: 'role'

Relation to existing issues

#32051 and #32951 report KeyError('choices') — a missing choices key. #34382 and #32203 address that by switching to c.get("choices"). Those changes do not fix this report: here choices is present but empty, so .get("choices") returns [] (falsy), next() skips it, and the default fallback lands on a chunk that still indexes out of range. Neither PR touches the ["choices"][0] access itself.

Are you a ML Ops Team?

No

What LiteLLM version are you on?

v1.88.1, and the same code is present unchanged on litellm_internal_staging as of 2026-08-21 (latest release v1.97.0).

Twitter / LinkedIn details

No response