feat(realtime): add Meta Muse Voice transcription by seyeong-han · Pull Request #39395 · BerriAI/litellm

seyeong-han · 2026-09-02T20:24:13Z

TLDR

Problem this solves:

LiteLLM cannot stream Meta Muse Voice transcription
Generic realtime providers cannot forward raw binary PCM

How it solves it:

Adds meta/muse-voice-transcribe-1.0 realtime routing
Maps OpenAI transcription events to Muse binary streaming
Closes every Muse turn on its own terminal signal, no cross-turn queue
Bills duration usage without logging transcript text

User Flow

Before: a developer cannot use Muse Voice through LiteLLM's realtime endpoint

The proxy admin adds meta/muse-voice-transcribe-1.0 to the model list with api_key: os.environ/META_API_KEY
The developer connects to WSS /v1/realtime?model=meta/muse-voice-transcribe-1.0&intent=transcription
The socket returns an error event and closes with code 1011 and reason Unsupported model: muse-voice-transcribe-1.0
Their application must integrate Meta's provider protocol directly

After: the same developer receives OpenAI-compatible transcription events from Muse

The proxy admin adds meta/muse-voice-transcribe-1.0 to the model list with api_key: os.environ/META_API_KEY
The developer connects to WSS /v1/realtime?model=meta/muse-voice-transcribe-1.0&intent=transcription
They receive session.created, send a GA session.update with audio/pcm at 16 or 24 kHz, and receive session.updated
They stream input_audio_buffer.append PCM16 chunks, then input_audio_buffer.commit (push to talk) or input_audio_buffer.end (server VAD)
They receive speech_started, transcription deltas, speech_stopped, and a completed event carrying usage: {"type": "duration", "seconds": ...} per turn
With server VAD, each detected utterance gets its own item_id, and a silent stretch never stalls the next utterance
The socket closes with code 1000 once Muse has no more segments, and the spend row shows the billed duration

Relevant issues

Docs: BerriAI/litellm-docs#1138

Linear ticket

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally (tests/test_litellm/llms/meta/realtime, tests/test_litellm/realtime_api/test_main.py, tests/test_litellm/litellm_core_utils/test_realtime_streaming.py, tests/test_litellm/litellm_core_utils/test_realtime_errors.py: 255 passed)
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Shared setup:

Proxy: python litellm/proxy/proxy_cli.py --config muse_config.yaml --port 4041 --detailed_debug --use_v2_migration_resolver with LITELLM_LOCAL_MODEL_COST_MAP=True and META_API_KEY in the environment
Config:

model_list:
  - model_name: meta/muse-voice-transcribe-1.0
    litellm_params:
      model: meta/muse-voice-transcribe-1.0
      api_key: os.environ/META_API_KEY

general_settings:
  master_key: sk-1234

Route: ws://localhost:4041/v1/realtime?model=meta/muse-voice-transcribe-1.0&intent=transcription with Authorization: Bearer sk-1234
Audio: repository tests/e2e/llm_translation/realtime/fixtures/weather_question_24k.wav, mono PCM16 at 24 kHz, 1.35 s
Upstream: live wss://api.meta.ai/v1/asr/realtime, real Meta spend
Both cases send this session.update first, then base64 input_audio_buffer.append frames of 100 ms each (turn_detection is null for push to talk, {"type": "server_vad"} for server VAD):

{"type": "session.update", "session": {"type": "transcription", "audio": {"input": {
  "format": {"type": "audio/pcm", "rate": 24000},
  "transcription": {"model": "meta/muse-voice-transcribe-1.0"},
  "turn_detection": null}}}}

Before (`7672399`)

Push to talk, fixture once, then commit

Connected to the route and sent the session.update above
Observed:

<- error
websockets.exceptions.ConnectionClosedError: received 1011 (internal error) Unsupported model: muse-voice-transcribe-1.0

Server VAD, 2 s silence, fixture, 2 s silence, fixture, 2 s silence, then end

Connected to the route and sent the session.update above with "turn_detection": {"type": "server_vad"}
Observed:

<- error
websockets.exceptions.ConnectionClosedError: received 1011 (internal error) Unsupported model: muse-voice-transcribe-1.0

After (`0e435e4`)

Push to talk, fixture once, then commit

Connected to the route, sent the session.update above, streamed the fixture as 14 append frames (64,766 bytes), then sent input_audio_buffer.commit
Observed:

<- session.created
<- session.updated
-> sent 14 input_audio_buffer.append chunks (64766 bytes PCM16 @ 24000 Hz)
<- input_audio_buffer.speech_started
<- conversation.item.input_audio_transcription.delta {"delta": "What"}
<- conversation.item.input_audio_transcription.delta {"delta": " is the"}
<- conversation.item.input_audio_transcription.delta {"delta": " weather"}
<- conversation.item.input_audio_transcription.delta {"delta": " in"}
<- conversation.item.input_audio_transcription.delta {"delta": " Paris?."}
<- input_audio_buffer.speech_stopped
<- conversation.item.input_audio_transcription.completed {"transcript": "What is the weather in Paris?.", "usage": {"type": "duration", "seconds": 1.36}}
closed code=1000 reason='No more transcript segments'

Proxy log: response_cost: 6.8e-05, which is 1.36 s at the recorded input_cost_per_second of 0.00005

Server VAD, 2 s silence, fixture, 2 s silence, fixture, 2 s silence, then end

Connected to the route, sent the session.update above with "turn_detection": {"type": "server_vad"}, streamed 87 append frames (417,532 bytes, 8.7 s) at real time, then sent input_audio_buffer.end
Observed:

<- session.created
<- session.updated
<- input_audio_buffer.speech_started item_id=0
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": "What is"}
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": " the"}
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": " weather in"}
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": " Paris?"}
<- input_audio_buffer.speech_stopped item_id=0
<- conversation.item.input_audio_transcription.completed item_id=0 {"transcript": "What is the weather in Paris?", "usage": {"type": "duration", "seconds": 3.6}}
<- input_audio_buffer.speech_started item_id=1
<- conversation.item.input_audio_transcription.delta item_id=1 {"delta": "What"}
<- conversation.item.input_audio_transcription.delta item_id=1 {"delta": " is the"}
<- conversation.item.input_audio_transcription.delta item_id=1 {"delta": " weather in"}
<- conversation.item.input_audio_transcription.delta item_id=1 {"delta": " Paris?"}
<- input_audio_buffer.speech_stopped item_id=1
<- conversation.item.input_audio_transcription.completed item_id=1 {"transcript": "What is the weather in Paris?", "usage": {"type": "duration", "seconds": 3.2}}
-> sent 87 input_audio_buffer.append chunks (417532 bytes PCM16 @ 24000 Hz)
-> input_audio_buffer.end
closed code=1000 reason='No more transcript segments'

Proxy log: response_cost: 0.000436, which is 8.72 s at 0.00005 per second: 6.8 s rode on the two completed events and the trailing 1.92 s of silence was billed on session close
The proxy debug log contains no transcript text and no raw Muse frames

Guardrail on a transcription session

Restarted the proxy with this guardrail added to the config, then ran the server VAD flow above once

guardrails:
  - guardrail_name: block-paris
    litellm_params:
      guardrail: litellm_content_filter
      mode: realtime_input_transcription
      default_on: true
      on_violation: end_session
      realtime_violation_message: Transcript blocked by policy
      blocked_words:
        - keyword: paris
          action: BLOCK

Observed:

<- session.created
<- session.updated
<- input_audio_buffer.speech_started item_id=0
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": "What"}
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": " is the"}
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": " weather"}
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": " in"}
<- conversation.item.input_audio_transcription.delta item_id=0 {"delta": " Paris?"}
<- input_audio_buffer.speech_stopped item_id=0
<- conversation.item.input_audio_transcription.completed item_id=0 {"transcript": "What is the weather in Paris?", "usage": {"type": "duration", "seconds": 3.52}}
<- error {"error": {"type": "guardrail_violation", "message": "Transcript blocked by policy", "code": "content_policy_violation"}}
closed code=1000 reason='upstream websocket closed with code 1000'

Proxy log: [realtime guardrail] BLOCKED transcript (violation 1) then ending session after violation 1; the backend never received response.create. Before 0e435e4 the transcription branch skipped the guardrail, so no error was sent and the session stayed open

Type

🆕 New Feature

Caveats (if any)

Medium

Supports realtime transcription only; batch transcription is excluded
Supports mono PCM16 at 16 kHz or 24 kHz
Muse speechComplete closes its turn immediately in both modes; a later speechEnd for that turn is a no-op
Overlapping Muse turns emit independently as their frames arrive, correlated by item_id, not in first-started order

Low

Conversational realtime response events are rejected
A late speechComplete with text for an already stopped turn emits a bare completed for that item_id
Finished turns are remembered for 128 turns; a duplicate frame for an older turn re-emits speech_stopped once
A duplicate speechStart for a finished turn is ignored and never re-captures turnless transcripts
A turn stays active for turnless partials after speechEnd until its speechComplete, so post-processing partials land on the right item_id
With a realtime_input_transcription guardrail on, the shared realtime guardrail code rewrites a client's turn_detection: null into {"create_response": false} before Muse sees it, so a push to talk client runs in ENDPOINTING and Meta closes the socket with 1008 Ingress below real-time once it stops streaming. Server VAD clients are unaffected and the injection predates this PR
End-to-end tests under tests/e2e/ land in a follow-up PR

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

codspeed · 2026-09-02T20:27:04Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing seyeong-han:litellm_meta_muse_voice_realtime (0e435e4) with litellm_internal_staging (9276317)¹}

No successful run was found on litellm_internal_staging (fe5ff9d) during the generation of this report, so 9276317 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

greptile-apps · 2026-09-02T20:27:25Z

Greptile Summary

Adds Meta Muse Voice realtime transcription through LiteLLM’s shared realtime transport.

Adds Muse session configuration, binary PCM packetization and pacing, event transformation, and duration-based usage accounting.
Extends shared realtime streaming to forward binary provider payloads and keeps transcription guardrails active while suppressing conversational response.create.
Adds model metadata, realtime types, provider registration, and focused tests.

Confidence Score: 5/5

The PR appears safe to merge because the previously reported transport concern has been resolved and no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/llms/meta/realtime/transformation.py	Implements Muse protocol configuration, PCM streaming, event normalization, turn tracking, and duration usage accounting through BaseRealtimeConfig.
litellm/litellm_core_utils/realtime_streaming.py	Adds raw binary forwarding and prevents transcription sessions from sending conversational response.create events without bypassing transcript guardrails.
litellm/llms/base_llm/realtime/transformation.py	Broadens provider transformation contracts to support binary messages and optional send pacing.
litellm/utils.py	Registers Meta’s realtime provider configuration with the existing provider manager.
tests/test_litellm/llms/meta/realtime/test_meta_realtime_transformation.py	Covers Muse configuration parsing, binary audio handling, pacing, provider events, turns, and usage behavior.
tests/test_litellm/litellm_core_utils/test_realtime_streaming.py	Verifies shared binary forwarding, transcription behavior, guardrail execution, and usage flushing.

_{Reviews (7): Last reviewed commit: "fix(realtime): run transcription guardra..." | Re-trigger Greptile}

codecov · 2026-09-02T20:27:52Z

Codecov Report

❌ Patch coverage is 95.55985% with 23 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/llms/meta/realtime/transformation.py	95.16%	22 Missing ⚠️
litellm/llms/base_llm/realtime/transformation.py	66.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

veria-ai · 2026-09-02T20:29:18Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

seyeong-han · 2026-09-02T20:32:08Z

Companion documentation PR: BerriAI/litellm-docs#1138

It documents MODEL_API_KEY, the Muse Voice model, /v1/realtime, PCM requirements, and the realtime-only scope. The exact tests/documentation_tests/test_env_keys.py gate passes when that docs branch is checked out. The current code-quality failure reads the docs repository default branch, so it will remain expected until the companion docs PR lands.

seyeong-han · 2026-09-02T20:46:40Z

@greptileai Please re-review the latest commit. The oversized base64 allocation finding is fixed in 298a729; the direct connector rationale is documented in the inline reply.

ryan-crabbe-berri · 2026-09-12T03:04:25Z

@greptileai re review

ryan-crabbe-berri · 2026-09-12T03:04:26Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

Replace the hand-rolled Meta realtime handler with a MetaRealtimeConfig that plugs into the shared realtime handler and RealTimeStreaming relay. Clients keep speaking the OpenAI realtime wire: session.update, input_audio_buffer.append/commit and the OpenAI transcription events. Unsupported transcription settings are logged and dropped, matching the Gemini realtime precedent, and the Meta-specific session.mode, keywords, language_bias, DIARIZATION and speaker extensions are removed. Drop the MODEL_API_KEY env var in favor of the standard META_API_KEY, remove the private-logging flag so spend logs record the transcript the same way other realtime models do, and add per-second pricing for muse-voice-transcribe-1.0. The relay now sends raw bytes from transform_realtime_request straight to the backend after pace_backend_send, and transcription sessions never trigger response.create.

ryan-crabbe-berri · 2026-09-12T03:22:59Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

Turns no longer wait behind each other in a FIFO queue, so an empty server_vad turn (speechStart then speechEnd with no transcript) cannot stall every later turn, and a PUSH_TO_TALK speechComplete now closes its turn without waiting for a speechEnd that never arrives. Each turn keeps its own idempotent emit state, so late or duplicate speechEnd, speechComplete and transcript frames are no-ops, and finished turns are remembered in a bounded map instead of a separate tombstone deque. The session.created ack and the sanitized error frame are now typed as members of OpenAIRealtimeEvents, which removes the typing.cast calls that the strict ruff budget flagged.

ryan-crabbe-berri · 2026-09-12T19:40:20Z

@greptileai re review

ryan-crabbe-berri · 2026-09-12T19:40:21Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

A duplicate speechStart for a turn that already stopped used to make that closed turn active again, so the next turnless PUSH_TO_TALK transcript was routed to the finished item and dropped.

ryan-crabbe-berri · 2026-09-12T19:54:21Z

@greptileai re review

ryan-crabbe-berri · 2026-09-12T19:54:22Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…eechEnd Muse partials carry no turnId and belong to the most recent speechStart, and the docs say the model may keep post processing a turn after speechEnd until speechComplete. Releasing the active turn on speechEnd made any partial arriving in that window raise and get dropped in ENDPOINTING mode. The turn now stays active until its speechComplete or final transcript.

ryan-crabbe-berri · 2026-09-12T22:14:24Z

@greptileai re review. Pushed 4647cd1: a Muse turn now stays active for turnless partials after speechEnd until speechComplete.

ryan-crabbe-berri · 2026-09-12T22:14:25Z

bugbot run

ryan-crabbe-berri

lgtm, thanks!

…sions The provider_config path skipped run_realtime_guardrails for transcription sessions to avoid sending response.create, which also dropped every realtime_input_transcription guardrail: no violation error reached the client and on_violation / end_session_after_n_fails never fired. Run the guardrail for every completed transcript and only suppress response.create when the session has no assistant turn.

ryan-crabbe-berri · 2026-09-12T22:35:11Z

@greptileai re review. Pushed 0e435e4: transcription sessions now run realtime_input_transcription guardrails and only skip response.create.

ryan-crabbe-berri · 2026-09-12T22:35:12Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0e435e4. Configure here.}

mateo-berri

LGTM. Thanks for the contribution and to Ryan for the review!

seyeong-han requested a review from mateo-berri as a code owner September 2, 2026 20:24

greptile-apps Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread litellm/llms/meta/realtime/handler.py Outdated

veria-ai Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread litellm/llms/meta/realtime/handler.py Outdated

seyeong-han mentioned this pull request Sep 2, 2026

docs(meta): add Muse Voice realtime setup BerriAI/litellm-docs#1138

Merged

devin-ai-integration Bot mentioned this pull request Sep 3, 2026

fix(model_prices): verified registry audit, Databricks Sep-2026 catalog, realtime image pricing, deprecation dates #39388

Merged

7 tasks

ryan-crabbe-berri force-pushed the litellm_meta_muse_voice_realtime branch from 298a729 to 51105c0 Compare September 12, 2026 03:03

ryan-crabbe-berri requested a review from a team September 12, 2026 03:03

seyeong-han added 2 commits September 11, 2026 20:10

feat(realtime): add Meta Muse Voice transcription

b82b31a

fix(realtime): bound Muse audio before decoding

1acb994