Cloud turn detector timeout and late EOU detection warnings after switching to LiveKit Inference VAD/turn detector

[kaushal-aubie]

Bug Description

After switching from the plugin-based VAD and turn detector to LiveKit Inference VAD + LiveKit Inference audio turn detector, I started seeing warnings/errors that did not occur with the previous setup.

Current issues

1. Late EOU detection:

eou detection ran after the audio eot turn was already flushed (likely a late stt final).
consider raising `min_delay` in the endpointing options to accommodate slow stt.

2. Cloud turn detector timeout/failure:

cloud turn detector failed (eot prediction timed out); falling back to local mini model

3. Higher worker/job memory usage compared to previous setup around 970+mb which was much lower below 750mb.

Previous setup

• Silero VAD plugin
• LiveKit plugin turn detector

No warnings were observed.

Current setup

• LiveKit Inference VAD
• LiveKit Inference audio turn detector

Configuration

VAD: (Turn detector throw error when below this value, should be actually documented in website)

MIN_SILENCE_DURATION=0.25

Endpointing: (I understand they are aggressive but i need that to avoid as much lower latency as possible)

MIN_ENDPOINTING_DELAY=0.15
MAX_ENDPOINTING_DELAY=1.2
ENDPOINTING_MODE=fixed

Deepgram stt settings, so default endpointing that livekit has is 25ms.

    kwargs: dict = {
        "model": (c and c.model) or "nova-3",
        "language": (c and c.language) or "en",
        "filler_words": True,
        "smart_format": True,
    }

Observations

The warning suggests that EOU detection is completing after the turn has already been flushed, possibly because the endpointing delay is too aggressive or the cloud turn detector response arrives late.

I reduced the delays for lower latency, but after moving to the inference models these warnings started appearing.

Questions

1. Is this expected behaviour when using LiveKit Inference turn detector compared to the plugin version?
2. What is the recommended min_delay/max_delay range when using cloud turn detection with external STT providers?
3. Is there any known reason for increased memory usage after enabling LiveKit Inference VAD + turn detector?

Would appreciate if these issues are fixed.

Expected Behavior

1. There should not be increase of 200mb+ as its same vad and turn detector?
2. There should be no failure/timeouts
3. Proper documentation in website for developers to know minimum settings and ideal settings and reason/cause it could increase ram usage.

Reproduction Steps

Just switched the livekit plugin silero, turn detector to livekit inference ones.

Operating System

Livekit cloud

Models Used

Deepgram nova3, gpt 5.4 mini, cartesia tts

Package Versions

"livekit-agents>=1.6.1",
"livekit-api>=1.1.0"

Session/Room/Call IDs

RM_HQMqnZ3afh2U, RM_GfoXD29Yfto6,RM_7BcQza5BEoAA

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response

[nightcityblade]

Hi, I'd like to work on this. I'll submit a PR shortly.

[nightcityblade]

I took a closer look and this appears to need maintainers' guidance rather than a small client-side fix: the reported timeout/fallback behavior is tied to LiveKit Inference/cloud turn detection and endpointing recommendations, plus memory usage under LiveKit Cloud. I don't want to send a speculative PR for this, so I'm going to unclaim it. Thanks!

[chenghao-mou]

Thanks for opening an issue!

The warning suggests that EOU detection is completing after the turn has already been flushed, possibly because the endpointing delay is too aggressive or the cloud turn detector response arrives late.

The audio turn detector has decided that it is an end of turn, and has committed the turn. But late STT transcript triggered another EOU detection call. It becomes possible exactly because the audio turn detector doesn't need to wait for transcripts to make a decision while previous text-based eot models need to run after STT transcripts.

I will rephrase the warning message to avoid the confusion.

MIN_ENDPOINTING_DELAY=0.15

The model itself usually takes about 80~100ms, so having MIN_ENDPOINTING_DELAY set to 150ms is getting too tight as this is the timeout for model inference too.

We do have a 300ms/2.5s defaults for the audio turn detector models if they are not given by the users. Docs will be updated to reflect those defaults/recommendations soon, including the silence requirement. Having a lower max delay is usually fine though.

The memory usage should be comparable with previous plugins. To better understand the extra memory consumption, can you share your previous setup and how you are measuring the memory usage?

[kaushal-aubie]

Thanks for opening an issue!

The warning suggests that EOU detection is completing after the turn has already been flushed, possibly because the endpointing delay is too aggressive or the cloud turn detector response arrives late.

The audio turn detector has decided that it is an end of turn, and has committed the turn. But late STT transcript triggered another EOU detection call. It becomes possible exactly because the audio turn detector doesn't need to wait for transcripts to make a decision while previous text-based eot models need to run after STT transcripts.

I will rephrase the warning message to avoid the confusion.

MIN_ENDPOINTING_DELAY=0.15

The model itself usually takes about 80~100ms, so having MIN_ENDPOINTING_DELAY set to 150ms is getting too tight as this is the timeout for model inference too.

We do have a 300ms/2.5s defaults for the audio turn detector models if they are not given by the users. Docs will be updated to reflect those defaults/recommendations soon, including the silence requirement. Having a lower max delay is usually fine though.

The memory usage should be comparable with previous plugins. To better understand the extra memory consumption, can you share your previous setup and how you are measuring the memory usage?

Update on this:

1. The warning still comes if we increase min endpointing delay to MIN_ENDPOINTING_DELAY=0.5
2. The memory usage is also logged in sentry when it cross job memory warn set in AgentServer, earlier we had 750 and we never get any warnings, and now we had to increase upto 1024 to remove warnings.

[kaushal-aubie]

This is how we use it

# We have two more such agent which uses diff endpointing, vad etc

class TestAgent(Agent):
    """agent for the voice."""

    def __init__(self, userdata: SessionData) -> None:
        """Initialize the agent."""
        vad = userdata.ctx.proc.userdata[ProcessKey.VAD]
        super().__init__(
            stt=stt,
            tts=tts,
            llm=llm,
            vad=vad,
            turn_handling=TurnHandlingOptions(
                turn_detection=build_turn_detection(),
                endpointing={
                    "mode": settings.VAD_ENDPOINTING_MODE,
                    "min_delay": settings.VAD_MIN_ENDPOINTING_DELAY,
                    "max_delay": settings.VAD_MAX_ENDPOINTING_DELAY,
                },
                interruption={
                    "enabled": True,
                    "resume_false_interruption": True,
                    "mode": "adaptive",
                    "min_words": 2,
                },
                preemptive_generation={"enabled": settingsPREEMPTIVE_GENERATION},
            ),
            instructions=_build_instructions(userdata.meta),
        )

def prewarm(proc: JobProcess) -> None:
    """Prewarm the userdata for the agent."""
    proc.userdata[ProcessKey.VAD_STT] = inference.VAD(
        model="silero",
        min_silence_duration=settings.VAD_STT_MIN_SILENCE_DURATION,
        min_speech_duration=settings.VAD_STT_MIN_SPEECH_DURATION,
    )
    proc.userdata[ProcessKey.VAD] = inference.VAD(
        model="silero",
        min_silence_duration=settings.VAD_MIN_SILENCE_DURATION,
    )
    # we have two more vad initialization for different agent
    if settings.LIVEKIT_NOISE_CANCELLATION_ENABLED:
        proc.userdata[ProcessKey.ENHANCER] = noise_cancellation.BVC()


server = AgentServer(
    port=settings.LIVEKIT_WORKER_HTTP_PORT,
    log_level=settings.LIVEKIT_LOG_LEVEL,
    num_idle_processes=2,
    job_memory_warn_mb=1024,  # Earlier it was 750
)

 await session.start(
            agent=TestAgent, # we switch start agent based on metdata
            room=ctx.room,
            room_options=room_io.RoomOptions(
                close_on_disconnect=False,
                delete_room_on_close=True,
            ),
            capture_run=True,
            ),
        )

[chenghao-mou]

The warning still comes if we increase min endpointing delay to MIN_ENDPOINTING_DELAY=0.5

As this is related to slow STT, you can adjust the value to the typical transcript delay value (plus some buffer time) from the STT you are using, which also means whatever STT you are using is now likely a bottleneck for your agent's turn-taking.

The memory usage is also logged in sentry when it cross job memory warn set in AgentServer, earlier we had 750 and we never get any warnings, and now we had to increase upto 1024 to remove warnings.

I believe this is going to be addressed by #6082. On Linux/forkserver, the memory usage reported double counts parent process memory usage, even if it is from copy-on-write pages. The actual usage should stay the same. Previous turn-detector plugin doesn't have this issue because it uses it's own process to isolate the memory, and in the new version, we leverage copy-on-write to save the memory and drop the process requirement.

[kaushal-aubie]

The warning still comes if we increase min endpointing delay to MIN_ENDPOINTING_DELAY=0.5

As this is related to slow STT, you can adjust the value to the typical transcript delay value (plus some buffer time) from the STT you are using, which also means whatever STT you are using is now likely a bottleneck for your agent's turn-taking.

The memory usage is also logged in sentry when it cross job memory warn set in AgentServer, earlier we had 750 and we never get any warnings, and now we had to increase upto 1024 to remove warnings.

I believe this is going to be addressed by #6082. On Linux/forkserver, the memory usage reported double counts parent process memory usage, even if it is from copy-on-write pages. The actual usage should stay the same. Previous turn-detector plugin doesn't have this issue because it uses it's own process to isolate the memory, and in the new version, we leverage copy-on-write to save the memory and drop the process requirement.

Thanks makes sense, to add about STT inference time is on an avg around ~400ms, its deepgram so usually below 328.54ms as we had recorded in most of sessions

[kaushal-aubie]

Sorry about cofusion for memory usage issue, its not linked with livekit turn detecor audio or silero vad, it was because of version upgrade from 1.6.0 to 1.6.2

1. So on 1.6.0, ram usage is mostly under 700mb, as i cannot see any warning log
2. For 1.6.1 its around 800mb.

{"message": "job process memory usage is above the warning threshold (advisory only, the process will not be terminated)", "level": "WARNING", "name": "livekit.agents", "memory_usage_mb": 787.0, "memory_warn_mb": 750, "memory_limit_mb": 0, "uptime": 36.0, "has_running_job": true, "baseline_memory_mb": 655.8, "growth_memory_mb": 131.2, "pid": 333, "job_id": "AJ_q48stDNZ5vmV", "room": "osce-773670-330", "timestamp": "2026-06-22T11:31:24.769759+00:00"}

3. For 1.6.2 its more than 950

[chenghao-mou]

The double counting emerges because of the introduction of the audio turn detector (livekit-local-inference to be precise) since 1.6.1. It should return to normal values once the patch is released.

In 1.6.2, we have not changed anything noticeable about the core package. Most, if not all, changes are related to vendors/plugins, you can see the change logs here