fix(stt): define and validate the direct STT audio contract (OSS-30) - #38
Merged
corasan merged 2 commits intoAug 11, 2026
Conversation
STT.transcribe/transcribeStream previously treated every nonempty ArrayBuffer as native-endian mono Float32 at 16 kHz and forced English. Encoded containers, misaligned buffers, and non-16-kHz PCM (including the library's own 24 kHz TTS output) silently produced wrong text. The buffer contract is now explicit and enforced on both sides of the bridge: recognizable encoded containers (WAV, ID3-tagged MP3, FLAC, Ogg, AIFF, CAF, MP4/M4A) and non-Float32-aligned byte lengths reject with actionable errors. A new STTTranscribeOptions makes sampleRate explicit: rates in 8-48 kHz are linearly resampled to 16 kHz natively before inference, others reject. Language is now selectable per call (and per listening session via STTListeningOptions); when omitted, Qwen3ASR auto-detects instead of forcing English. Contract rules live in a pure Foundation-only STTAudioContract covered by swiftc-run unit tests; TS-side guards mirror them before crossing the bridge. Sample copying now happens synchronously in the bridge call, so the JS ArrayBuffer is no longer read from a detached Task. README documents the audio format, language behavior, and the NSMicrophoneUsageDescription requirement for live transcription.
Section comments that restated adjacent expect messages are gone; bare validate calls in tests get named values instead. Remaining comments state only what code cannot show: cross-language mirroring, the deliberate MP3 frame-sync omission, the ArrayBuffer lifetime rule, and NaN equality semantics.
corasan
deleted the
feature/oss-30-define-and-validate-the-mlx-direct-stt-audio-contract
branch
August 11, 2026 23:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Direct STT accepted any nonempty
ArrayBufferas 16 kHz mono Float32 and forced English. Encoded files, Int16 buffers, and the library's own 24 kHz TTS output silently produced incorrect transcriptions.This PR makes the audio contract explicit and enforces it:
transcribe/transcribeStreamreject encoded containers and misaligned buffers with errors that say what to fix.STTTranscribeOptions.sampleRateresamples 8–48 kHz PCM to the model rate natively; rates outside that range reject.languageis now selectable per call and per listening session; when omitted, the model auto-detects instead of forcing English.NSMicrophoneUsageDescriptionrequirement.Review notes
STTAudioContract(no MLX import) withswiftc-run tests; TS guards mirror them. Design notes:docs/superpowers/specs/2026-08-11-oss-30-stt-audio-contract-design.md.ArrayBufferwas read inside a detachedTask, after its guaranteed lifetime.Closes OSS-30.