Skip to content

(phonic): Use stream_ahead_of_real_time mode for Phonic WebSocket and sample rate 24000 #1885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tinalenguyen merged 3 commits into from
Jun 26, 2026
+51 −43
Merged

(phonic): Use stream_ahead_of_real_time mode for Phonic WebSocket and sample rate 24000 #1885

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
02e782a
feat(phonic): stream ahead of real time at native pcm_24000
qionghuang6 Jun 25, 2026
78a0497
chore: add changeset for phonic stream ahead
qionghuang6 Jun 25, 2026
33febb0
refactor(phonic): build session config from a single source
qionghuang6 Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/phonic-stream-ahead.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@livekit/agents-plugin-phonic': patch
---

Enable stream_ahead_of_real_time mode for phonic
2 changes: 1 addition & 1 deletion plugins/phonic/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
"typescript": "^5.0.0"
},
"dependencies": {
"phonic": "^0.31.10"
"phonic": "^0.32.5"
},
"peerDependencies": {
"@livekit/agents": "workspace:*",
Expand Down
55 changes: 37 additions & 18 deletions plugins/phonic/src/realtime/realtime_model.ts
View file
Open in desktop

@devin-ai-integration devin-ai-integration Bot Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Config deduplication removes redundant overrides from sendConfig

The old connect() method spread buildConfigOptions(...) into the sendConfig payload and then re-specified many of the same properties (additional_languages, multilingual_mode, audio_speed, tools, boosted_keywords, generate_no_input_poke_text, no_input_poke_sec, no_input_poke_text, no_input_end_conversation_sec) as direct overrides. Since these had identical values to what buildConfigOptions already returned, the overrides were purely redundant. The one exception was min_words_to_interrupt, which was commented out in buildConfigOptions but included directly in sendConfig. The PR un-comments it in buildConfigOptions (line 950-952) to maintain equivalence. This also means min_words_to_interrupt is now sent during mid-session resets via _updateSession → sendReset, which previously did not include it. This is likely a bug fix rather than a regression.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,20 @@ import {
DEFAULT_API_CONNECT_OPTIONS,
Future,
asError,
createTimedString,
llm,
log,
shortuuid,
stream,
type TimedString,
} from '@livekit/agents';
import { AudioFrame, AudioResampler } from '@livekit/rtc-node';
import type { Phonic } from 'phonic';
import { PhonicClient } from 'phonic';
import type { ServerEvent, Voice } from './api_proto.js';

const PHONIC_INPUT_SAMPLE_RATE = 44100;
const PHONIC_OUTPUT_SAMPLE_RATE = 44100;
const PHONIC_INPUT_SAMPLE_RATE = 24000;
const PHONIC_OUTPUT_SAMPLE_RATE = 24000;
const PHONIC_NUM_CHANNELS = 1;
const PHONIC_INPUT_FRAME_MS = 20;
const DEFAULT_MODEL = 'merritt';
Expand Down Expand Up @@ -183,7 +185,7 @@ export class RealtimeModel extends llm.RealtimeModel {
midSessionInstructionsUpdate: true,
midSessionToolsUpdate: true,
perResponseToolChoice: false,
nativeTranscriptSync: true,
nativeTranscriptSync: false,
});

const apiKey = options.apiKey || process.env.PHONIC_API_KEY;
Expand Down Expand Up @@ -245,9 +247,10 @@ interface GenerationState {
responseId: string;
messageChannel: stream.StreamChannel<llm.MessageGeneration>;
functionChannel: stream.StreamChannel<llm.FunctionCall>;
textChannel: stream.StreamChannel<string>;
textChannel: stream.StreamChannel<string | TimedString>;
audioChannel: stream.StreamChannel<AudioFrame>;
outputText: string;
audioCursorSec: number;
}

/**
Expand All @@ -274,7 +277,7 @@ export class RealtimeSession extends llm.RealtimeSession {
private toolsReady = new Future<void>();
private closedFuture = new Future<void, never>();
private connectTask: Promise<void>;
private toolDefinitions: Record<string, unknown>[] = [];
private toolDefinitions: Phonic.InlineWebSocketTool[] = [];
private forbidSpeechAfterToolCall = new Set<string>();
private pendingToolCallIds = new Set<string>();
private readyToStart = new Future<void>();
Expand Down Expand Up @@ -407,7 +410,7 @@ export class RealtimeSession extends llm.RealtimeSession {
this.toolsReady.resolve();
}

private buildToolDefinitions(tools: llm.ToolContext): Record<string, unknown>[] {
private buildToolDefinitions(tools: llm.ToolContext): Phonic.InlineWebSocketTool[] {
this.forbidSpeechAfterToolCall = new Set(this.options.forbidSpeechAfterToolCall ?? []);
return Object.entries(tools)
.filter(([, tool]) => llm.isFunctionTool(tool))
Expand All @@ -418,7 +421,7 @@ export class RealtimeSession extends llm.RealtimeSession {
function: {
name,
description: tool.description,
parameters: llm.toJsonSchema(tool.parameters),
parameters: llm.toJsonSchema(tool.parameters) as Phonic.OpenAiFunctionParameters,
strict: true,
},
},
Expand Down Expand Up @@ -466,7 +469,7 @@ export class RealtimeSession extends llm.RealtimeSession {
this.closeCurrentGeneration({ interrupted: true });
this.pendingUserText = undefined;

const toolsPayload: Phonic.ConfigOptions.Tools.Item[] = [
const toolsPayload: Phonic.ToolDefinition[] = [
...(this.options.phonicTools ?? []),
...this.toolDefinitions,
];
Expand Down Expand Up @@ -749,10 +752,8 @@ export class RealtimeSession extends llm.RealtimeSession {
const gen = this.currentGeneration;
if (gen === undefined) return;

if (message.text) {
gen.outputText += message.text;
gen.textChannel.write(message.text);
}
let audioFrame: AudioFrame | undefined;
let audioDurationSec = 0;

if (message.audio) {
const bytes = Buffer.from(message.audio, 'base64');
Expand All @@ -764,15 +765,31 @@ export class RealtimeSession extends llm.RealtimeSession {
bytes.byteOffset + sampleCount * Int16Array.BYTES_PER_ELEMENT,
),
);
const frame = new AudioFrame(
audioFrame = new AudioFrame(
pcm,
PHONIC_OUTPUT_SAMPLE_RATE,
PHONIC_NUM_CHANNELS,
sampleCount / PHONIC_NUM_CHANNELS,
);
gen.audioChannel.write(frame);
audioDurationSec = audioFrame.samplesPerChannel / PHONIC_OUTPUT_SAMPLE_RATE;
}
}

if (message.text) {
gen.outputText += message.text;
gen.textChannel.write(
createTimedString({
text: message.text,
startTime: gen.audioCursorSec,
endTime: gen.audioCursorSec + audioDurationSec,
}),
);
}

if (audioFrame) {
gen.audioChannel.write(audioFrame);
gen.audioCursorSec += audioDurationSec;
}
}

private handleInputText(message: Phonic.InputTextPayload): void {
Expand Down Expand Up @@ -838,7 +855,7 @@ export class RealtimeSession extends llm.RealtimeSession {

const responseId = shortuuid('PS_');

const textChannel = stream.createStreamChannel<string>();
const textChannel = stream.createStreamChannel<string | TimedString>();
const audioChannel = stream.createStreamChannel<AudioFrame>();
const functionChannel = stream.createStreamChannel<llm.FunctionCall>();
const messageChannel = stream.createStreamChannel<llm.MessageGeneration>();
Expand All @@ -857,6 +874,7 @@ export class RealtimeSession extends llm.RealtimeSession {
textChannel,
audioChannel,
outputText: '',
audioCursorSec: 0,
};

const generationEvent: llm.GenerationCreatedEvent = {
Expand Down Expand Up @@ -922,7 +940,7 @@ export class RealtimeSession extends llm.RealtimeSession {
toolsPayload,
}: {
systemPrompt: string;
toolsPayload: Phonic.ConfigOptions.Tools.Item[];
toolsPayload: Phonic.ToolDefinition[];
}): Phonic.ConfigOptions {
return {
agent: this.options.phonicAgent,
Expand All @@ -931,8 +949,8 @@ export class RealtimeSession extends llm.RealtimeSession {
generate_welcome_message: this.options.generateWelcomeMessage,
system_prompt: systemPrompt,
voice_id: this.options.voice,
input_format: 'pcm_44100',
output_format: 'pcm_44100',
input_format: 'pcm_24000',
output_format: 'pcm_24000',
...(this.options.defaultLanguage !== undefined && {
default_language: this.options.defaultLanguage,
}),
Expand All @@ -952,6 +970,7 @@ export class RealtimeSession extends llm.RealtimeSession {
no_input_poke_sec: this.options.noInputPokeSec,
no_input_poke_text: this.options.noInputPokeText,
no_input_end_conversation_sec: this.options.noInputEndConversationSec,
stream_ahead_of_real_time: true,

@devin-ai-integration devin-ai-integration Bot Jun 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 stream_ahead_of_real_time is hardcoded with no user opt-out

The stream_ahead_of_real_time: true option at line 957 is hardcoded in buildConfigOptions with no corresponding constructor option to disable it. While this aligns with the PR's stated intent, users who experience issues with ahead-of-real-time streaming have no way to fall back to the previous real-time behavior without modifying source code. This may be intentional if the feature is considered stable, but it's worth confirming this is the desired default for all use cases.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@qionghuang6 qionghuang6 Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that's intentional!

};
}

Expand Down
10 changes: 5 additions & 5 deletions pnpm-lock.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.