fix: activate the canonical corpus indexer (agents.entries + startup sync) - #982
fix: activate the canonical corpus indexer (agents.entries + startup sync)#982benjaml4 wants to merge 1 commit into
Conversation
Two independent faults left the canonical corpus permanently empty when canonicalCorpus.enabled was set: 1. canonicalCorpusIndexer.sync() was never invoked. Its only call sites are inside the memory capability (search()/sync()), reached via getActiveMemorySearchManager — but agent searches resolve to the plugin's own memory_search/memory_get tools and bypass the capability entirely. Add a fire-and-forget sync at service start; sync() already checks enabled, rate-limits via syncIntervalMs, and de-duplicates concurrent runs. 2. resolveCanonicalCorpusWorkspaces read agents.list, but OpenClaw stores agents as an object keyed by id under agents.entries. The loop matched nothing and the function fell back to the single default workspace, silently skipping every per-agent workspace. Support both shapes. Verified on a 6-agent install: workspaces resolved went 1 -> 6, and the startup sync indexed 2505/2505 chunks from 796 documents.
rwmjhb
left a comment
There was a problem hiding this comment.
Reviewed head 00550dc. Activating canonical corpus sync and supporting agents.entries addresses issue #980, but startup activation currently introduces a cost/storage blocker.
Every process start calls sync(). runSync() then unconditionally calls embedPassage() and store.upsert() for every chunk; lastSyncAt is process-local and resets on restart, while upsert() deletes and re-adds each row. An unchanged corpus is therefore fully re-embedded and rewritten on every restart. Issue #981 reports roughly 1 GB of LanceDB growth per restart for 2,505 chunks, in addition to repeated embedding cost.
Please land content-hash change detection with startup activation: bulk-load existing corpus refs, compare deterministic IDs and corpus_content_sha256, preserve unchanged IDs in expectedIds, and embed/upsert only changed chunks. Add regressions for unchanged corpora across restarts, agents.entries workspace discovery, enabled/disabled startup sync, and startup failure handling. The one-time startup call also leaves normal memory_search stale after later file changes because that path still bypasses syncOnSearch; please either wire the rate-limited sync into the primary search path or track that explicitly as a follow-up. Revert or explain the unrelated LanceDB platform lockfile removal.
Requesting changes.
Fixes #980.
Two independent faults left the canonical corpus permanently empty when
canonicalCorpus.enabledwas set — details and reproduction in the issue.canonicalCorpusIndexer.sync()was never invoked. Its only call sites are inside the memory capability, which agent searches bypass because the plugin registers its ownmemory_search/memory_gettools. This adds a fire-and-forget sync at service start, mirroring the existingrunStartupCheckspattern so gateway startup is never blocked.sync()already checksenabled, rate-limits viasyncIntervalMs, and de-duplicates concurrent runs.resolveCanonicalCorpusWorkspacesreadagents.list, but OpenClaw stores agents keyed by id underagents.entries— the loop matched nothing and fell back to the single default workspace. Both shapes are now supported;agents.listhandling is untouched.Verified on a 6-agent install (OpenClaw 2026.7.2-beta.6): workspaces resolved went 1 → 6, and the startup sync indexed 2505/2505 chunks from 796 documents.
Note: the startup log line intentionally omits the
unchangedcounter so this branch stands alone; #981's PR adds that counter.tsc --noEmitclean.