Skip to content

Preserve streaming timings from OpenAI-compatible providers.#1507

Open
dmsnell wants to merge 1 commit into
simonw:mainfrom
dmsnell:fix/streaming-timings
Open

Preserve streaming timings from OpenAI-compatible providers.#1507
dmsnell wants to merge 1 commit into
simonw:mainfrom
dmsnell:fix/streaming-timings

Conversation

@dmsnell

@dmsnell dmsnell commented Jul 2, 2026

Copy link
Copy Markdown

llama.cpp's OpenAI-compatible streaming endpoint can include token/sec metrics in a top-level timings field on the final streamed chunk. llm was already preserving usage from that chunk, but combine_chunks() discarded provider-specific timing metadata, so llm logs --json could not show the throughput data even when the provider returned it.

In this patch, timings is captured while combining streamed chunks and include it in the persisted response JSON. Add a regression test using a llama.cpp-style streaming response.

Co-Authored-By: Codex (GPT-5.5)

llama.cpp's OpenAI-compatible streaming endpoint can include token/sec
metrics in a top-level `timings` field on the final streamed chunk.
`llm` was already preserving usage from that chunk, but
`combine_chunks()` discarded provider-specific timing metadata, so `llm
logs --json` could not show the throughput data even when the provider
returned it.

In this patch, `timings` is captured while combining streamed chunks and
include it in the persisted response JSON. Add a regression test using a
llama.cpp-style streaming response.

Co-Authored-By: Codex (GPT-5.5)
@dmsnell

dmsnell commented Jul 2, 2026

Copy link
Copy Markdown
Author

I’ve seen this before, where llm would show no timing information from OpenAI-compatible API calls, but it wasn’t until working with llama.cpp locally where I examined the streaming chunks that I discovered that the information was there, but llm was discarding it.

This returns the API timing information into the logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant