Skip to content

fix: skip unchanged canonical corpus chunks via stored content hashes - #983

Merged
rwmjhb merged 3 commits into
CortexReach:masterfrom
benjaml4:fix/canonical-corpus-change-detection
Aug 9, 2026
Merged

fix: skip unchanged canonical corpus chunks via stored content hashes#983
rwmjhb merged 3 commits into
CortexReach:masterfrom
benjaml4:fix/canonical-corpus-change-detection

Conversation

@benjaml4

@benjaml4 benjaml4 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Fixes #981.

runSync() re-embedded and re-upserted every chunk on every sync — stats.skipped only counts errors, so there was no unchanged path. Each pass creates a new LanceDB version set, and runStorageMaintenance() clamps retention to ≥ 1 day, so on restart-heavy installs the store grew ~1GB per boot (76M → 9.2G here) with no same-day way to reclaim it.

The fix reads back what toMemoryEntry() already writes: corpus_content_sha256, bulk-loaded once per sync via the existing store.listCorpusEntryRefs(). Chunks whose hash matches are skipped (no embed, no upsert) but still land in expectedIds, so cleanupStaleCorpusEntries() behaviour is unchanged. Adds an unchanged counter to CorpusIndexStats and the completion log.

Measured across consecutive restarts on a real 2,505-chunk corpus:

before: indexed 2505/2505 every boot — store 76M -> 9.2G over a few restarts
after:  indexed 1, unchanged 2504   (one file genuinely edited)
after:  indexed 0, unchanged 2505   — store flat at 78M

Also saves 2,505 embedding calls per restart. tsc --noEmit clean. Independent of the #980 PR; both touch corpus-indexer.ts in non-overlapping regions.

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 781f344. Changes requested:

  1. The full local suite is red at test/corpus-indexer.test.mjs:163: sync() now returns unchanged: 0, but the existing deepStrictEqual expectation still uses the old result shape. Please update that assertion and add a real two-sync regression that forces a second completed sync and proves there are no additional embedding/upsert calls, unchanged chunks remain in stale cleanup's expected set, and stats.unchanged is correct.

  2. src/corpus-indexer.ts:611 skips solely on chunk ID plus text hash, while the stored row also carries corpus_start_line, corpus_end_line, corpus_chunk_count, document hash, mtime, and timestamp. An edit elsewhere in a document can preserve a chunk's index/text while shifting its line range, leaving citations built from the stored metadata stale. Please include retrieval-relevant metadata in the comparison, reindex all chunks when the document fingerprint changes, or refresh metadata while retaining the existing vector.

This head is also behind master and has no GitHub check runs, so please rebase and rerun the full suite with the regression coverage above.

benjaml4 and others added 2 commits August 8, 2026 03:54
…verything (CortexReach#981)

runSync() had no change detection — every sync re-embedded and re-upserted
every chunk (stats.skipped only counts errors). Because lastSyncAt is
instance state, every gateway restart rewrote the whole corpus, and each
upsert pass creates a new LanceDB version set: ~1GB of version growth per
restart for a 2,505-chunk corpus whose vectors total ~23MB. The growth is
unreclaimable same-day because runStorageMaintenance() clamps retention
to >= 1 day.

The data to fix it was already stored: toMemoryEntry() writes
corpus_content_sha256 on every chunk, and store.listCorpusEntryRefs()
bulk-returns id + metadata for all corpus rows. Load those hashes once
per sync and skip chunks whose content is unchanged — no embed, no
upsert. Skipped chunks still land in expectedIds so stale cleanup does
not delete them. A new 'unchanged' counter is reported in stats and the
completion log.

Measured across consecutive restarts on a real install:
  before: indexed 2505/2505 every boot, store 76M -> 9.2G
  after:  indexed 1 (one file edited), unchanged 2504
  after:  indexed 0, unchanged 2505 — store flat at 78M
- Skip logic now checks corpus_document_sha256 in addition to
  corpus_content_sha256, so edits elsewhere in a document trigger
  reindex of its chunks even when a chunk's own text is unchanged
  (prevents stale line-range citations)
- Fix test assertion at line 163: add unchanged to expected shape
- Add two-sync regression: force sync skips all unchanged chunks
  (zero embed calls, zero upsert calls, unchanged=4, staleDeleted=0)
- Add document-fingerprint-change regression: editing a file triggers
  reindex of its chunks (embedding calls > 0)
@benjaml4
benjaml4 force-pushed the fix/canonical-corpus-change-detection branch from 781f344 to 1899fbb Compare August 8, 2026 02:54

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 1be4c49. The previous blockers are resolved: unchanged detection now requires both the chunk content hash and document hash, the result-shape assertions are updated, the forced second-sync regression covers the no-embed/no-upsert path, and the generated runtime artifact is committed. The branch is mergeable and all required CI checks are green.

Non-blocking follow-up: the document-edit fixture still changes both hashes at once, so a multi-chunk test that edits one chunk and asserts exact reindex/unchanged counts would more directly protect the document-hash-only path.

@rwmjhb
rwmjhb merged commit 2bbe8d2 into CortexReach:master Aug 9, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Canonical corpus re-embeds and re-upserts every chunk on every sync (no change detection) — ~1GB store growth per restart

2 participants