Skip to content

fix: activate the canonical corpus indexer (agents.entries + startup sync) - #982

Open
benjaml4 wants to merge 1 commit into
CortexReach:masterfrom
benjaml4:fix/canonical-corpus-activation
Open

fix: activate the canonical corpus indexer (agents.entries + startup sync)#982
benjaml4 wants to merge 1 commit into
CortexReach:masterfrom
benjaml4:fix/canonical-corpus-activation

Conversation

@benjaml4

@benjaml4 benjaml4 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Fixes #980.

Two independent faults left the canonical corpus permanently empty when canonicalCorpus.enabled was set — details and reproduction in the issue.

  1. canonicalCorpusIndexer.sync() was never invoked. Its only call sites are inside the memory capability, which agent searches bypass because the plugin registers its own memory_search/memory_get tools. This adds a fire-and-forget sync at service start, mirroring the existing runStartupChecks pattern so gateway startup is never blocked. sync() already checks enabled, rate-limits via syncIntervalMs, and de-duplicates concurrent runs.
  2. resolveCanonicalCorpusWorkspaces read agents.list, but OpenClaw stores agents keyed by id under agents.entries — the loop matched nothing and fell back to the single default workspace. Both shapes are now supported; agents.list handling is untouched.

Verified on a 6-agent install (OpenClaw 2026.7.2-beta.6): workspaces resolved went 1 → 6, and the startup sync indexed 2505/2505 chunks from 796 documents.

Note: the startup log line intentionally omits the unchanged counter so this branch stands alone; #981's PR adds that counter. tsc --noEmit clean.

Two independent faults left the canonical corpus permanently empty when
canonicalCorpus.enabled was set:

1. canonicalCorpusIndexer.sync() was never invoked. Its only call sites
   are inside the memory capability (search()/sync()), reached via
   getActiveMemorySearchManager — but agent searches resolve to the
   plugin's own memory_search/memory_get tools and bypass the capability
   entirely. Add a fire-and-forget sync at service start; sync() already
   checks enabled, rate-limits via syncIntervalMs, and de-duplicates
   concurrent runs.

2. resolveCanonicalCorpusWorkspaces read agents.list, but OpenClaw stores
   agents as an object keyed by id under agents.entries. The loop matched
   nothing and the function fell back to the single default workspace,
   silently skipping every per-agent workspace. Support both shapes.

Verified on a 6-agent install: workspaces resolved went 1 -> 6, and the
startup sync indexed 2505/2505 chunks from 796 documents.

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 00550dc. Activating canonical corpus sync and supporting agents.entries addresses issue #980, but startup activation currently introduces a cost/storage blocker.

Every process start calls sync(). runSync() then unconditionally calls embedPassage() and store.upsert() for every chunk; lastSyncAt is process-local and resets on restart, while upsert() deletes and re-adds each row. An unchanged corpus is therefore fully re-embedded and rewritten on every restart. Issue #981 reports roughly 1 GB of LanceDB growth per restart for 2,505 chunks, in addition to repeated embedding cost.

Please land content-hash change detection with startup activation: bulk-load existing corpus refs, compare deterministic IDs and corpus_content_sha256, preserve unchanged IDs in expectedIds, and embed/upsert only changed chunks. Add regressions for unchanged corpora across restarts, agents.entries workspace discovery, enabled/disabled startup sync, and startup failure handling. The one-time startup call also leaves normal memory_search stale after later file changes because that path still bypasses syncOnSearch; please either wire the rate-limited sync into the primary search path or track that explicitly as a follow-up. Revert or explain the unrelated LanceDB platform lockfile removal.

Requesting changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

canonicalCorpus never indexes: sync() is never invoked, and workspace resolution reads agents.list instead of agents.entries

2 participants