Skip to content

Add SpeechTranscriber as STT option to Assist#4384

Draft
Copilot wants to merge 7 commits intomainfrom
copilot/add-local-audio-transcription
Draft

Add SpeechTranscriber as STT option to Assist#4384
Copilot wants to merge 7 commits intomainfrom
copilot/add-local-audio-transcription

Conversation

Copy link
Contributor

Copilot AI commented Feb 23, 2026

Summary

Adds opt-in local audio transcription using the SpeechTranscriber SPM package as an alternative to the assist websocket API for speech-to-text.

SPM Integration:

  • Added bgoncal/SpeechTranscriber v1.0.0 as an Xcode SPM package dependency
  • SpeechTranscriberAdapter wraps the SPM class, conforming to SpeechTranscriberProtocol for DI/testability
  • The SPM package manages its own audio engine—no AudioRecorder or buffer forwarding needed for on-device STT

iOS 17+ Availability:

  • Protocols (SpeechTranscriberProtocol, SpeechTranscriberDelegate) are in a separate SpeechTranscriberProtocol.swift file with no SPM import, available on all iOS versions
  • SpeechTranscriberAdapter is gated with @available(iOS 17, *) since the SPM package requires iOS 17+
  • NoOpSpeechTranscriber provides a fallback for pre-iOS 17 devices
  • Construction site in AssistView+Build.swift uses #available(iOS 17, *) to conditionally select the implementation
  • All new files are registered in the Xcode project (pbxproj) for proper compilation

Settings & Configuration:

  • AssistConfiguration gains sttLanguage field (persisted in GRDB with auto-migration)
  • AssistSettingsView adds on-device STT toggle + language picker from SFSpeechRecognizer.supportedLocales(), with unsupported-language warning
  • Feature is off by default (opt-in)

AssistViewModel Flow:

  • assistWithAudio() branches: on-device STT bypasses AudioRecorder entirely, server STT unchanged
  • On transcription finish, text is sent via .text() source to the assist pipeline
  • On failure, graceful fallback to server STT with info message in chat
// When on-device STT is enabled, audio capture is handled by the SPM package
if configuration.enableOnDeviceSTT {
    isRecording = true
    startOnDeviceTranscription() // SpeechTranscriberAdapter → SPM SpeechTranscriber
} else {
    audioRecorder.startRecording() // existing websocket STT path
}

Tests: 10 test cases covering enable/disable, language routing, text completion, empty input, failure fallback, and stop streaming.

Screenshots

Link to pull request in Documentation repository

Documentation: home-assistant/companion.home-assistant#

Any other notes

The adapter uses nonisolated protocol conformance with Task { @MainActor in } bridging to satisfy the SPM package's @MainActor requirement. Active tasks are tracked and cancelled to prevent race conditions on rapid start/stop.


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits February 23, 2026 11:03
…nguage selection, and fallback mechanism

Co-authored-by: bgoncal <5808343+bgoncal@users.noreply.github.com>
…chTranscriber

Co-authored-by: bgoncal <5808343+bgoncal@users.noreply.github.com>
Copilot AI changed the title [WIP] Add local audio transcription feature with opt-in mechanism Add opt-in local audio transcription via SFSpeechRecognizer Feb 23, 2026
Copilot AI requested a review from bgoncal February 23, 2026 11:09
Copilot AI and others added 2 commits February 23, 2026 11:18
Co-authored-by: bgoncal <5808343+bgoncal@users.noreply.github.com>
Co-authored-by: bgoncal <5808343+bgoncal@users.noreply.github.com>
Copilot AI changed the title Add opt-in local audio transcription via SFSpeechRecognizer Use SpeechTranscriber as SPM dependency instead of copied implementation Feb 23, 2026
@bgoncal bgoncal changed the title Use SpeechTranscriber as SPM dependency instead of copied implementation Add SpeechTranscriber as STT option to Assist Feb 23, 2026
@bgoncal
Copy link
Member

bgoncal commented Feb 23, 2026

@copilot Build is failing, code needs to also be restricted to iOS17+ which is what SPM supports

…ailable, add to Xcode project

Co-authored-by: bgoncal <5808343+bgoncal@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Feb 23, 2026

@copilot Build is failing, code needs to also be restricted to iOS17+ which is what SPM supports

Fixed in 0d69f31. Split the protocols into a separate SpeechTranscriberProtocol.swift (no SPM import, always available), gated SpeechTranscriberAdapter with @available(iOS 17, *), added NoOpSpeechTranscriber fallback for pre-iOS 17, and use #available(iOS 17, *) at the construction site in AssistView+Build.swift. Also registered all new files in the Xcode project (they were missing from the pbxproj which caused the "cannot find type" errors).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants