Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# FUTO Voice Input Moonshine

This personal fork keeps the FUTO voice keyboard experience and uses Moonshine v2 Small Streaming as its default offline recognizer. NVIDIA Parakeet TDT 0.6B V3 and the legacy FUTO Whisper/GGML backend remain selectable from Model Options.
This personal fork keeps the FUTO voice keyboard experience and uses Moonshine v2 Streaming as its default offline recognizer. Choose the Small model for balanced speed and accuracy or Medium for higher accuracy. NVIDIA Parakeet TDT 0.6B V3 and the legacy FUTO Whisper/GGML backend remain selectable from Model Options.

This fork's Parakeet integration and repository changes were built with AI assistance using Codex (GPT-5).

The goal is straightforward: keep the FUTO UI and recording flow while adding responsive streaming transcription and personal vocabulary corrections.

## What Changed

- Moonshine v2 Small Streaming is the default backend and emits live partial transcripts.
- Moonshine v2 Small Streaming is the default balanced option and emits live partial transcripts.
- Moonshine v2 Medium Streaming is available as a higher-accuracy option.
- Parakeet and legacy Whisper/GGML remain selectable backends.
- Batch and streaming recognizers share backend-neutral Kotlin contracts.
- Personal vocabulary entries correct partial and final transcripts; use `heard => preferred` for explicit aliases.
Expand All @@ -25,16 +26,17 @@ Moonshine is selected by default, with Parakeet and Whisper/GGML available as fa

## Active Model

The default backend is:
The default backend and model are:

```text
Moonshine v2 Small Streaming English
```

The app downloads the quantized model assets from:
The app downloads the selected quantized model assets from:

```text
https://download.moonshine.ai/model/small-streaming-en/quantized/
https://download.moonshine.ai/model/medium-streaming-en/quantized/
```

Model files are downloaded on first use rather than packaged into the APK.
Expand All @@ -43,7 +45,7 @@ After installing the APK, the model is downloaded by the app:

1. Open FUTO Voice Input Moonshine Settings.
2. Tap **Model**.
3. Select **Moonshine v2 Small Streaming**.
3. Select **Balanced** (Small) or **Higher accuracy** (Medium).
4. Confirm the download.

If you try voice input before downloading the model, the app prompts for the download. Transcription runs offline after installation.
Expand All @@ -52,6 +54,7 @@ Downloaded model files are stored in app-private storage:

```text
filesDir/moonshine-small-streaming-en/
filesDir/moonshine-medium-streaming-en/
```

Parakeet and Whisper/GGML models are also stored in app-private storage and run offline after download.
Expand Down
7 changes: 5 additions & 2 deletions app/src/main/java/org/futo/voiceinput/AudioRecognizer.kt
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import kotlinx.coroutines.withContext
import kotlinx.coroutines.yield
import org.futo.voiceinput.ml.RunState
import org.futo.voiceinput.moonshine.MoonshineBackend
import org.futo.voiceinput.moonshine.getSelectedMoonshineModelVariant
import org.futo.voiceinput.moonshine.isMoonshineModelDownloaded
import org.futo.voiceinput.settings.ENABLE_30S_LIMIT
import org.futo.voiceinput.settings.IS_VAD_ENABLED
Expand Down Expand Up @@ -252,7 +253,8 @@ abstract class AudioRecognizer {
SpeechBackendType.Parakeet -> ParakeetEngineManager.acquire(context)
SpeechBackendType.Moonshine -> {
ParakeetEngineManager.forceClose()
MoonshineBackend().also { it.load(context) }
val variant = context.getSelectedMoonshineModelVariant()
MoonshineBackend(variant).also { it.load(context) }
}
SpeechBackendType.WhisperGGML -> {
ParakeetEngineManager.forceClose()
Expand Down Expand Up @@ -318,7 +320,8 @@ abstract class AudioRecognizer {
}
}
SpeechBackendType.Moonshine -> {
if (!context.isMoonshineModelDownloaded()) {
val variant = context.getSelectedMoonshineModelVariant()
if (!context.isMoonshineModelDownloaded(variant)) {
needMoonshineModelDownload()
return@launch
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ import kotlinx.coroutines.launch
import kotlinx.coroutines.withContext
import org.futo.voiceinput.backend.StreamingSpeechBackend

class MoonshineBackend : StreamingSpeechBackend {
class MoonshineBackend(
private val variant: MoonshineModelVariant
) : StreamingSpeechBackend {

private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Default)
private var transcriber: Transcriber? = null
Expand All @@ -28,8 +30,11 @@ class MoonshineBackend : StreamingSpeechBackend {
override suspend fun load(context: Context) = withContext(Dispatchers.IO) {
transcriber = Transcriber().apply {
loadFromFiles(
context.applicationContext.moonshineModelDir().absolutePath,
JNI.MOONSHINE_MODEL_ARCH_SMALL_STREAMING
context.applicationContext.moonshineModelDir(variant).absolutePath,
when (variant) {
MoonshineModelVariant.Small -> JNI.MOONSHINE_MODEL_ARCH_SMALL_STREAMING
MoonshineModelVariant.Medium -> JNI.MOONSHINE_MODEL_ARCH_MEDIUM_STREAMING
}
)
}
}
Expand Down
38 changes: 26 additions & 12 deletions app/src/main/java/org/futo/voiceinput/moonshine/MoonshineModel.kt
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ import org.futo.voiceinput.downloader.EXTRA_DOWNLOAD_FILE_HASHES
import org.futo.voiceinput.downloader.EXTRA_DOWNLOAD_FILE_NAMES
import org.futo.voiceinput.downloader.EXTRA_DOWNLOAD_FILE_URLS
import org.futo.voiceinput.downloader.EXTRA_TARGET_SUBDIR
import org.futo.voiceinput.settings.MOONSHINE_MODEL_VARIANT
import org.futo.voiceinput.settings.getSetting
import org.futo.voiceinput.settings.getSettingBlocking
import java.io.File

object MoonshineModel {
const val directoryName = "moonshine-small-streaming-en"
const val completionMarker = ".download_complete"
private const val baseUrl =
"https://download.moonshine.ai/model/small-streaming-en/quantized"

val files = listOf(
"adapter.ort",
Expand All @@ -27,35 +27,49 @@ object MoonshineModel {
"tokenizer.bin"
)

fun url(file: String) = "$baseUrl/$file"
}

fun Context.moonshineModelDir(): File = File(filesDir, MoonshineModel.directoryName)
fun Context.moonshineModelDir(variant: MoonshineModelVariant): File =
File(filesDir, variant.directoryName)

fun Context.isMoonshineModelDownloaded(): Boolean {
val directory = moonshineModelDir()
fun Context.isMoonshineModelDownloaded(variant: MoonshineModelVariant): Boolean {
val directory = moonshineModelDir(variant)
return File(directory, MoonshineModel.completionMarker).exists() &&
MoonshineModel.files.all { File(directory, it).exists() }
}

fun Context.moonshineModelDownloadIntent(): Intent =
private fun Context.selectedMoonshineModelVariant() =
getSettingBlocking(MOONSHINE_MODEL_VARIANT.key, MOONSHINE_MODEL_VARIANT.default)
.toMoonshineModelVariant()

suspend fun Context.getSelectedMoonshineModelVariant() =
getSetting(MOONSHINE_MODEL_VARIANT).toMoonshineModelVariant()

fun Context.isMoonshineModelDownloaded(): Boolean =
isMoonshineModelDownloaded(selectedMoonshineModelVariant())

fun Context.moonshineModelDownloadIntent(
variant: MoonshineModelVariant = selectedMoonshineModelVariant()
): Intent =
Intent(this, DownloadActivity::class.java).apply {
putExtra(EXTRA_TARGET_SUBDIR, MoonshineModel.directoryName)
putExtra(EXTRA_TARGET_SUBDIR, variant.directoryName)
putExtra(EXTRA_COMPLETION_MARKER, MoonshineModel.completionMarker)
putStringArrayListExtra(
EXTRA_DOWNLOAD_FILE_NAMES,
ArrayList(MoonshineModel.files)
)
putStringArrayListExtra(
EXTRA_DOWNLOAD_FILE_URLS,
ArrayList(MoonshineModel.files.map(MoonshineModel::url))
ArrayList(MoonshineModel.files.map { "${variant.baseUrl}/$it" })
)
putStringArrayListExtra(
EXTRA_DOWNLOAD_FILE_HASHES,
ArrayList(List(MoonshineModel.files.size) { "" })
)
}

fun Context.startMoonshineModelDownloadActivity() {
startActivity(moonshineModelDownloadIntent())
fun Context.startMoonshineModelDownloadActivity(
variant: MoonshineModelVariant = selectedMoonshineModelVariant()
) {
startActivity(moonshineModelDownloadIntent(variant))
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
package org.futo.voiceinput.moonshine

enum class MoonshineModelVariant(
val id: String,
val directoryName: String,
val baseUrl: String
) {
Small(
id = "small",
directoryName = "moonshine-small-streaming-en",
baseUrl = "https://download.moonshine.ai/model/small-streaming-en/quantized"
),
Medium(
id = "medium",
directoryName = "moonshine-medium-streaming-en",
baseUrl = "https://download.moonshine.ai/model/medium-streaming-en/quantized"
)
}

fun String.toMoonshineModelVariant(): MoonshineModelVariant =
MoonshineModelVariant.entries.firstOrNull { it.id == this } ?: MoonshineModelVariant.Small
3 changes: 3 additions & 0 deletions app/src/main/java/org/futo/voiceinput/settings/Settings.kt
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withContext
import org.futo.voiceinput.BuildConfig
import org.futo.voiceinput.moonshine.MoonshineModelVariant
import org.futo.voiceinput.theme.presets.DevThemeYellow
import org.futo.voiceinput.theme.presets.VoiceInputTheme

Expand Down Expand Up @@ -126,6 +127,8 @@ fun String.toSpeechBackendType(): SpeechBackendType {
}

val SPEECH_BACKEND = SettingsKey(stringPreferencesKey("speech_backend"), SpeechBackendType.Moonshine.id)
val MOONSHINE_MODEL_VARIANT =
SettingsKey(stringPreferencesKey("moonshine_model_variant"), MoonshineModelVariant.Small.id)
val PARAKEET_KEEP_WARM = SettingsKey(booleanPreferencesKey("parakeet_keep_warm"), true)
val PARAKEET_KEEP_WARM_TIMEOUT_MS =
SettingsKey(longPreferencesKey("parakeet_keep_warm_timeout_ms"), 5 * 60 * 1000L)
Expand Down
64 changes: 45 additions & 19 deletions app/src/main/java/org/futo/voiceinput/settings/pages/Models.kt
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,16 @@ import org.futo.voiceinput.migration.NeedsMigration
import org.futo.voiceinput.parakeet.isParakeetModelDownloaded
import org.futo.voiceinput.parakeet.startParakeetModelDownloadActivity
import org.futo.voiceinput.moonshine.isMoonshineModelDownloaded
import org.futo.voiceinput.moonshine.MoonshineModelVariant
import org.futo.voiceinput.moonshine.startMoonshineModelDownloadActivity
import org.futo.voiceinput.moonshine.toMoonshineModelVariant
import org.futo.voiceinput.settings.DISMISS_MIGRATION_TIP
import org.futo.voiceinput.settings.ENABLE_MULTILINGUAL
import org.futo.voiceinput.settings.ENGLISH_MODEL_INDEX
import org.futo.voiceinput.settings.LANGUAGE_TOGGLES
import org.futo.voiceinput.settings.MANUALLY_SELECT_LANGUAGE
import org.futo.voiceinput.settings.MODELS_MIGRATED
import org.futo.voiceinput.settings.MOONSHINE_MODEL_VARIANT
import org.futo.voiceinput.settings.MULTILINGUAL_MODEL_INDEX
import org.futo.voiceinput.settings.PERSONAL_DICTIONARY
import org.futo.voiceinput.settings.SPEECH_BACKEND
Expand All @@ -63,6 +66,7 @@ import org.futo.voiceinput.startModelDownloadActivity
fun modelsSubtitle(): String? {
val context = LocalContext.current
val (backend, _) = useDataStore(SPEECH_BACKEND)
val (moonshineVariantId, _) = useDataStore(MOONSHINE_MODEL_VARIANT)
return when (backend.toSpeechBackendType()) {
SpeechBackendType.Parakeet -> {
if (context.isParakeetModelDownloaded(verifyHashes = true)) {
Expand All @@ -72,7 +76,7 @@ fun modelsSubtitle(): String? {
}
}
SpeechBackendType.Moonshine -> {
if (context.isMoonshineModelDownloaded()) {
if (context.isMoonshineModelDownloaded(moonshineVariantId.toMoonshineModelVariant())) {
stringResource(R.string.moonshine_model_active_subtitle)
} else {
stringResource(R.string.moonshine_model_download_required)
Expand Down Expand Up @@ -155,34 +159,56 @@ fun ParakeetModelStatus() {
fun MoonshineModelStatus() {
val context = LocalContext.current
val lifecycleOwner = LocalLifecycleOwner.current
val isDownloaded = remember { mutableStateOf(context.isMoonshineModelDownloaded()) }
val modelVariant = useDataStore(MOONSHINE_MODEL_VARIANT)
val downloadedVariants = remember(context) {
mutableStateOf(
MoonshineModelVariant.entries.filter { context.isMoonshineModelDownloaded(it) }.toSet()
)
}

DisposableEffect(lifecycleOwner) {
DisposableEffect(lifecycleOwner, context) {
val observer = LifecycleEventObserver { _, event ->
if (event == Lifecycle.Event.ON_RESUME) {
isDownloaded.value = context.isMoonshineModelDownloaded()
downloadedVariants.value = MoonshineModelVariant.entries
.filter { context.isMoonshineModelDownloaded(it) }
.toSet()
}
}
lifecycleOwner.lifecycle.addObserver(observer)
onDispose { lifecycleOwner.lifecycle.removeObserver(observer) }
}

ScreenTitle(stringResource(R.string.moonshine_model))
SettingItem(
title = stringResource(R.string.moonshine_streaming_model_name),
subtitle = if (isDownloaded.value) {
stringResource(R.string.moonshine_model_downloaded)
} else {
stringResource(R.string.moonshine_model_download_required)
},
onClick = {
if (!isDownloaded.value) context.startMoonshineModelDownloadActivity()
},
icon = {
RadioButton(selected = isDownloaded.value, onClick = null, enabled = false)
},
disabled = false
) { }
MoonshineModelVariant.entries.forEach { variant ->
val selected = modelVariant.value.toMoonshineModelVariant() == variant
val downloaded = variant in downloadedVariants.value
val (titleResource, descriptionResource) = when (variant) {
MoonshineModelVariant.Small ->
R.string.moonshine_balanced to R.string.moonshine_small_description
MoonshineModelVariant.Medium ->
R.string.moonshine_higher_accuracy to R.string.moonshine_medium_description
}
val title = stringResource(titleResource)
val description = stringResource(descriptionResource)
val status = stringResource(
if (downloaded) R.string.moonshine_model_downloaded
else R.string.moonshine_model_download_required
)
val selectOrDownload = {
modelVariant.setValue(variant.id)
if (!downloaded) context.startMoonshineModelDownloadActivity(variant)
}

SettingItem(
title = title,
subtitle = stringResource(R.string.moonshine_model_option_subtitle, description, status),
onClick = selectOrDownload,
icon = {
RadioButton(selected = selected, onClick = selectOrDownload)
},
disabled = false
) { }
}
Tip(stringResource(R.string.moonshine_download_model_tip))
}

Expand Down
10 changes: 7 additions & 3 deletions app/src/main/res/values/strings.xml
Original file line number Diff line number Diff line change
Expand Up @@ -115,10 +115,14 @@
<string name="whisper_download_required_body">The selected Whisper model must be downloaded before voice input can run. This may incur data fees on mobile data.</string>
<string name="whisper_ggml_model_active_subtitle">Whisper/GGML model active</string>
<string name="moonshine_model">Moonshine Model</string>
<string name="moonshine_model_active_subtitle">Moonshine v2 Small Streaming active</string>
<string name="moonshine_streaming_model_name">Moonshine v2 Small Streaming English</string>
<string name="moonshine_model_active_subtitle">Moonshine v2 Streaming active</string>
<string name="moonshine_balanced">Balanced</string>
<string name="moonshine_higher_accuracy">Higher accuracy</string>
<string name="moonshine_small_description">Moonshine v2 Small Streaming English (~245 MB)</string>
<string name="moonshine_medium_description">Moonshine v2 Medium Streaming English (~429 MB)</string>
<string name="moonshine_model_option_subtitle">%1$s — %2$s</string>
<string name="moonshine_model_download_required">Download required</string>
<string name="moonshine_model_downloaded">Downloaded and active</string>
<string name="moonshine_model_downloaded">Downloaded</string>
<string name="moonshine_download_model_tip">Low-latency English transcription that updates while you speak. The model remains offline after download.</string>
<string name="moonshine_download_required_body">The Moonshine streaming model must be downloaded before voice input can run. This may incur data fees on mobile data.</string>
<string name="payment_text_1">You\'ve been using FUTO Voice Input Moonshine for <xliff:g name="days" example="30">%d</xliff:g> days. If you find this app useful, please consider paying to support future development of FUTO software.</string>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
package org.futo.voiceinput.moonshine

import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotEquals
import org.junit.Test

class MoonshineModelVariantTest {
@Test
fun persistedIdsSelectTheExpectedQuality() {
assertEquals(MoonshineModelVariant.Small, "small".toMoonshineModelVariant())
assertEquals(MoonshineModelVariant.Medium, "medium".toMoonshineModelVariant())
}

@Test
fun unknownIdsFallBackToBalanced() {
assertEquals(MoonshineModelVariant.Small, "future-model".toMoonshineModelVariant())
}

@Test
fun variantsUseSeparateDownloads() {
assertNotEquals(
MoonshineModelVariant.Small.directoryName,
MoonshineModelVariant.Medium.directoryName
)
assertNotEquals(
MoonshineModelVariant.Small.baseUrl,
MoonshineModelVariant.Medium.baseUrl
)
}
}
Loading
Loading