Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion docs/features/light-ocr-integration/plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,12 +129,36 @@ entries. Corruption discards the derived cache and rebuilds it without affecting
- Pending-input migration adds nullable blocking data and does not rewrite existing payloads.
- Cache is not a fact source and can be cleared or lost without affecting sent messages.

## Composer Progressive Disclosure

Keep node commands and reactive environment state behind separate injection keys.
`INPUT_NODE_ACTIONS` continues to own mutations and the existing vision-model picker action.
`ATTACHMENT_NODE_CONTEXT` carries the ACP flag, the advisory selected-model vision capability and
an on-demand OCR availability snapshot.

The chat page and new-thread page derive model capability from their effective provider/model
selection and pass `null` until the renderer can resolve the selected model. `ChatInputBox` provides
that state to TipTap attachment nodes, owns one request-versioned OCR status read shared by its
nodes, coalesces concurrent reads and reuses a successful snapshot for a short TTL. Refresh does not
replace known availability with an indistinguishable loading state; a failed read resets to
`unknown` rather than retaining stale availability. Nodes treat both unknown model capability and
unknown OCR availability as selectable; the main router is the only enforcement boundary.

The composer hides representation controls entirely for ACP without mutating stored preferences.
For DeepChat, an `auto` attachment renders only a focusable attachment-options affordance that is
visually disclosed on hover or focus. Explicit preferences render a compact intent badge. A known
non-vision selection disables creation of a new `image` preference and routes the existing
switch-model action to the same picker used by attachment-preparation recovery. No composer code
rewrites an explicit preference in response to a model or Agent change.

## Validation Strategy

- Unit test routing, preprocessing, cache keys/GC, process protocol and failure recovery.
- Test direct/new-thread/queue/steer/remote semantics and provider non-invocation when blocked.
- Test persistence, restart, retry, compaction, delete, export, sync-compatible JSON and search.
- Test renderer draft preservation, action dialogs, pending blocked controls and settings states.
- Test renderer draft preservation, action dialogs, pending blocked controls, settings states,
ACP control suppression, nullable model capability, on-demand OCR status failure and explicit
preference preservation.
- Run real packaged OCR on the current macOS target; configure but do not claim remote target results
until their workflows run.
- Record cold/warm latency, peak/idle RSS and packaged size. Stop for review above 768 MiB peak RSS
Expand Down
40 changes: 39 additions & 1 deletion docs/features/light-ocr-integration/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,21 @@ returns an actionable explanation instead of synthesizing a generic caption or c

## Settings And UX

Add Tools -> File processing -> OCR with:
Expose OCR as a built-in capability in the main-window Plugins Hub. Keep the legacy
`settings-ocr` route registered for direct navigation and persisted settings activity, but hide it
from the Settings sidebar. The route continues to render the same management component as a
compatibility surface; the Plugins Hub remains the canonical entry. When an ACP agent is selected,
Spotlight falls back to that compatibility route because the whole Plugins Hub is gated. The Plugins
Hub entry remains visible when the runtime is unavailable so users can inspect the reason. Both
surfaces may coexist in separate windows, but only the visible, focused instance polls runtime
status.

Runtime-status polling runs only while the owning renderer document is visible and its window is
focused. Returning to an active surface refreshes status immediately. Catalog refresh failure is
shown as an unknown status even when a previous snapshot exists; it must not keep presenting a stale
Available badge.

The OCR management page provides:

- automatic OCR for non-vision models, enabled by default;
- Auto/CPU execution backend;
Expand All @@ -138,6 +152,27 @@ Image attachment actions are named `Auto`, `Send image` and `Use OCR text`. Do n
optimized provider payload an "original" image. Sent attachments show their effective
representation and allow the OCR snapshot to be inspected.

Composer representation controls use progressive disclosure:

- `auto` is the implicit default and does not render a persistent label on image or PDF chips;
- an accessible attachment-options trigger is discoverable on hover and keyboard focus, and remains
visible for coarse-pointer devices that cannot hover;
- explicit `image`, `embedded_text` and `ocr_text` preferences remain visible as compact intent
badges until the user restores `auto`;
- menu actions keep action-oriented labels, while badges keep the existing short state labels;
- renderer capability checks are advisory only. Unknown model capability and failed OCR status
reads fail open, while the main-process attachment router remains authoritative at dispatch;
- a known non-vision model cannot create a new `image` preference from the menu. An existing
`image` preference remains intact, and the UI offers the existing vision-model picker and an
`auto` reset instead of silently changing user intent;
- ACP composers hide representation controls and intent badges because ACP does not consume the
DeepChat attachment representation contract. Stored preferences remain unchanged and become
visible again when the draft returns to a DeepChat Agent;
- OCR availability is read on demand when the attachment menu opens. A short-lived successful
snapshot is shared across nodes, and an expired snapshot remains authoritative while its refresh
is in flight so a known unavailable capability cannot flicker back to selectable. Composer nodes
never poll the runtime or reuse a potentially stale plugin-catalog snapshot.

## Acceptance Criteria

- A non-vision model receives useful, marked OCR text from a supported image without network access.
Expand All @@ -149,6 +184,9 @@ representation and allow the OCR snapshot to be inspected.
- Helper crashes, hangs, cancellation and app shutdown leave no orphan process or stale private temp
files.
- Unsupported platforms clearly report why OCR is unavailable.
- Composer attachment chips keep the default path free of representation labels, expose advanced
choices to pointer and keyboard users, suppress no-op representation controls for ACP, and never
destroy an explicit preference merely because the selected model or Agent changes.
- Packaged smoke verifies the bundled Node version, helper, native package, model identity, real OCR
and offline execution on each supported target before that target is considered enabled.
- Release and package-regression packaging compare every selected installer role against the
Expand Down
10 changes: 10 additions & 0 deletions docs/features/light-ocr-integration/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,16 @@ Status: implemented; reusable packaging workflow remote validation pending.
- [ ] Run the refactored six-target reusable packaging workflows after this branch is pushed by a
maintainer.

## Composer Progressive Disclosure

- [x] Add a typed attachment-node context without mixing reactive state into node actions.
- [x] Hide representation controls and badges for ACP while preserving stored draft preferences.
- [x] Make `auto` implicit and expose advanced choices through a hover- and focus-accessible menu.
- [x] Fail open for unresolved capability reads while preserving known OCR state during refresh.
- [x] Prevent new `image` overrides for known non-vision models without rewriting existing intent.
- [x] Reuse the existing vision-model picker action from the attachment menu.
- [x] Add focused renderer regression coverage and run the repository validation gate.

## Merge-blocking Review Hardening

- [x] Make legacy attachment detection tolerate missing and malformed metadata.
Expand Down
23 changes: 17 additions & 6 deletions docs/features/plugins-hub/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,14 @@ Status: implemented and maintained.

## Product surface

`/plugins` is the primary extension hub. It exposes official plugin detail routes plus the existing MCP,
Skills and Remote management surfaces. ACP availability is shown as a boundary state rather than pretending
ACP is an installable Plugin.
`/plugins` is the primary extension hub. It exposes official plugin detail routes, built-in OCR
management, and the existing MCP, Skills and Remote management surfaces. Built-in capabilities
route to their owning modules and do not adopt plugin installation or enablement semantics. ACP
availability is shown as a boundary state rather than pretending ACP is an installable Plugin.

The hub remains unavailable while an ACP agent is selected. OCR does not run for ACP sessions, but
its configuration still affects DeepChat agents, so the Spotlight OCR action opens the retained
`settings-ocr` compatibility route in that state instead of navigating into the blocked hub.

The hub must preserve direct navigation, back/refresh behavior, loading/empty/error states and platform
availability. Plugin-specific settings render in a Desktop-owned settings window; Plugin code does not create
Expand All @@ -33,9 +38,15 @@ launch. Persistent installation records cannot make a missing or untrusted packa

## Renderer contract

Renderer routes use typed Plugin clients. Official detail pages display install/enable/runtime/MCP/settings
state and surface action errors without exposing filesystem secrets. Virtual cards for MCP, Skills and Remote
route to their real owners instead of duplicating configuration inside Plugin state.
Renderer routes use typed clients. Official detail pages display install/enable/runtime/MCP/settings
state and surface action errors without exposing filesystem secrets. Virtual cards and pages for
built-in OCR, MCP, Skills and Remote route to their real owners instead of duplicating
configuration inside Plugin state. Built-in OCR stays first in the catalog and remains visible when
its runtime is unavailable. Its catalog snapshot and refresh-error state live in the shared catalog
store; a failed refresh must not leave a stale Available badge visible or fail the rest of the
catalog. The compact catalog card therefore suppresses stale status entirely, while the diagnostic
management page may retain its last snapshot only when it also renders an explicit stale warning;
the surfaces intentionally optimize for different levels of diagnostic context.

## Validation

Expand Down
31 changes: 25 additions & 6 deletions src/renderer/settings/components/OcrSettings.vue
Original file line number Diff line number Diff line change
Expand Up @@ -283,8 +283,8 @@
</template>

<script setup lang="ts">
import { computed, onBeforeUnmount, onMounted, ref } from 'vue'
import { useIntervalFn } from '@vueuse/core'
import { computed, onBeforeUnmount, onMounted, ref, watch } from 'vue'
import { useDocumentVisibility, useIntervalFn, useWindowFocus } from '@vueuse/core'
import { Icon } from '@iconify/vue'
import { nanoid } from 'nanoid'
import type { AcceptableValue } from 'reka-ui'
Expand Down Expand Up @@ -346,7 +346,10 @@ const clearDialogOpen = ref(false)
const advancedOpen = ref(false)
const diagnosticsOpen = ref(false)
const statusHasError = ref(false)
const documentVisibility = useDocumentVisibility()
const windowFocused = useWindowFocus()
const statusStale = computed(() => statusHasError.value && status.value !== null)
const pollingAllowed = computed(() => documentVisibility.value === 'visible' && windowFocused.value)

const availabilityLabel = computed(() =>
status.value?.availability.status === 'available'
Expand Down Expand Up @@ -410,24 +413,35 @@ const canClearCache = computed(() => {
)
})

const { pause: pausePolling, resume: resumePolling } = useIntervalFn(refreshStatus, 5_000, {
const { pause: pausePolling, resume: resumePolling } = useIntervalFn(pollRuntimeStatus, 5_000, {
immediate: false,
immediateCallback: false
})
let mounted = false
let disposed = false

const activatePolling = () => {
if (!mounted || disposed || !pollingAllowed.value) return
void refreshStatus()
resumePolling()
}

onMounted(() => {
mounted = true
void loadSettings()
void refreshStatus().finally(() => {
if (!disposed) resumePolling()
})
activatePolling()
})

onBeforeUnmount(() => {
disposed = true
pausePolling()
})

watch(pollingAllowed, (allowed) => {
pausePolling()
if (allowed) activatePolling()
})

async function loadSettings(): Promise<void> {
if (settingsOperationPending.value) return

Expand Down Expand Up @@ -506,6 +520,11 @@ async function refreshStatus(): Promise<void> {
}
}

async function pollRuntimeStatus(): Promise<void> {
if (!pollingAllowed.value) return
await refreshStatus()
}

async function clearCache(): Promise<void> {
if (!canClearCache.value) return
cacheClearFailed.value = false
Expand Down
75 changes: 74 additions & 1 deletion src/renderer/src/components/chat/ChatInputBox.vue
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ import { TextSelection } from '@tiptap/pm/state'
import type { MessageFile, UserMessageInlineItem } from '@shared/types/agent-interface'
import { useI18n } from 'vue-i18n'
import { Spinner } from '@shadcn/components/ui/spinner'
import { createOcrClient, type OcrClient } from '@api/OcrClient'
import {
buildChatInputWorkspaceReferenceText,
getChatInputWorkspaceItemDragData
Expand All @@ -70,7 +71,12 @@ import { useSkillsData } from '@/components/chat-input/composables/useSkillsData
import { SkillChip } from './nodes/skillChip'
import { FileAttachment } from './nodes/fileAttachment'
import { CommandForm } from './nodes/commandForm'
import { INPUT_NODE_ACTIONS, type InputNodeActions } from './nodes/symbols'
import {
ATTACHMENT_NODE_CONTEXT,
INPUT_NODE_ACTIONS,
type AttachmentOcrAvailability,
type InputNodeActions
} from './nodes/symbols'

const SlashMention = Mention.extend({
name: 'slashMention'
Expand All @@ -84,6 +90,7 @@ const props = withDefaults(
agentId?: string | null
workspacePath?: string | null
isAcpSession?: boolean
supportsVision?: boolean | null
isGenerating?: boolean
editable?: boolean
submitDisabled?: boolean
Expand All @@ -100,6 +107,7 @@ const props = withDefaults(
agentId: 'deepchat',
workspacePath: null,
isAcpSession: false,
supportsVision: null,
isGenerating: false,
editable: true,
submitDisabled: false,
Expand All @@ -118,6 +126,7 @@ const emit = defineEmits<{
'update:files': [files: MessageFile[]]
'command-submit': [command: string]
'pending-skills-change': [skills: string[]]
'switch-vision-model': []
'draft-change': []
'toggle-voice-input': []
}>()
Expand Down Expand Up @@ -186,6 +195,10 @@ const actions: InputNodeActions = {
files.updateFile(idx, { requestedRepresentation: preference })
}
},
switchToVisionModel: () => {
if (!props.editable) return
emit('switch-vision-model')
},
submitCommandForm: (values) => {
if (!props.editable) return
mentions.submitDialog(values)
Expand All @@ -197,6 +210,65 @@ const actions: InputNodeActions = {

provide(INPUT_NODE_ACTIONS, actions)

const attachmentOcrAvailability = ref<AttachmentOcrAvailability>({ status: 'unknown' })
const attachmentIsAcpSession = computed(() => props.isAcpSession)
const attachmentSupportsVision = computed(() => props.supportsVision)
const ATTACHMENT_OCR_AVAILABILITY_TTL_MS = 30_000
let attachmentOcrClient: OcrClient | null = null
let attachmentOcrStatusLoadedAt = 0
let attachmentOcrStatusRequest: Promise<void> | null = null
let attachmentOcrStatusRequestId = 0

async function refreshAttachmentOcrAvailability(): Promise<void> {
const cacheAge = Date.now() - attachmentOcrStatusLoadedAt
if (
attachmentOcrAvailability.value.status !== 'unknown' &&
cacheAge >= 0 &&
cacheAge < ATTACHMENT_OCR_AVAILABILITY_TTL_MS
) {
return
}
if (attachmentOcrStatusRequest) {
await attachmentOcrStatusRequest
return
}

const requestId = ++attachmentOcrStatusRequestId
const request = (async () => {
try {
attachmentOcrClient ??= createOcrClient()
const status = await attachmentOcrClient.getRuntimeStatus()
if (requestId !== attachmentOcrStatusRequestId) {
return
}
attachmentOcrAvailability.value = status.availability
attachmentOcrStatusLoadedAt = Date.now()
} catch (error) {
if (requestId !== attachmentOcrStatusRequestId) {
return
}
attachmentOcrAvailability.value = { status: 'unknown' }
attachmentOcrStatusLoadedAt = 0
console.warn('[ChatInputBox] Failed to load OCR availability:', error)
}
})()
attachmentOcrStatusRequest = request
try {
await request
} finally {
if (attachmentOcrStatusRequest === request) {
attachmentOcrStatusRequest = null
}
}
}

provide(ATTACHMENT_NODE_CONTEXT, {
isAcpSession: attachmentIsAcpSession,
supportsVision: attachmentSupportsVision,
ocrAvailability: attachmentOcrAvailability,
refreshOcrAvailability: refreshAttachmentOcrAvailability
})

// ── Editor helpers ─────────────────────────────────────────────

const sameFiles = (a: MessageFile[], b: MessageFile[]) => {
Expand Down Expand Up @@ -546,6 +618,7 @@ watch(
)

onUnmounted(() => {
attachmentOcrStatusRequestId += 1
editor.destroy()
})

Expand Down
25 changes: 25 additions & 0 deletions src/renderer/src/components/chat/attachmentModelPicker.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import { nextTick, type Ref } from 'vue'

export interface ChatStatusBarModelPicker {
openModelPicker?: () => boolean
}

export function openChatStatusBarModelPicker(
statusBarRef: Readonly<Ref<ChatStatusBarModelPicker | null>>,
logScope: string
): void {
void nextTick(() => {
const opened = statusBarRef.value?.openModelPicker?.() ?? false
if (!opened) {
console.warn(`[${logScope}] Model picker is unavailable`)
}
})
}

export function switchAttachmentToVisionModel(
cancelPreparation: () => void,
openModelPicker: () => void
): void {
cancelPreparation()
openModelPicker()
}
Loading