fix: skip unchanged canonical corpus chunks via stored content hashes - #983
Conversation
rwmjhb
left a comment
There was a problem hiding this comment.
Reviewed head 781f344. Changes requested:
-
The full local suite is red at
test/corpus-indexer.test.mjs:163:sync()now returnsunchanged: 0, but the existingdeepStrictEqualexpectation still uses the old result shape. Please update that assertion and add a real two-sync regression that forces a second completed sync and proves there are no additional embedding/upsert calls, unchanged chunks remain in stale cleanup's expected set, andstats.unchangedis correct. -
src/corpus-indexer.ts:611skips solely on chunk ID plus text hash, while the stored row also carriescorpus_start_line,corpus_end_line,corpus_chunk_count, document hash, mtime, and timestamp. An edit elsewhere in a document can preserve a chunk's index/text while shifting its line range, leaving citations built from the stored metadata stale. Please include retrieval-relevant metadata in the comparison, reindex all chunks when the document fingerprint changes, or refresh metadata while retaining the existing vector.
This head is also behind master and has no GitHub check runs, so please rebase and rerun the full suite with the regression coverage above.
…verything (CortexReach#981) runSync() had no change detection — every sync re-embedded and re-upserted every chunk (stats.skipped only counts errors). Because lastSyncAt is instance state, every gateway restart rewrote the whole corpus, and each upsert pass creates a new LanceDB version set: ~1GB of version growth per restart for a 2,505-chunk corpus whose vectors total ~23MB. The growth is unreclaimable same-day because runStorageMaintenance() clamps retention to >= 1 day. The data to fix it was already stored: toMemoryEntry() writes corpus_content_sha256 on every chunk, and store.listCorpusEntryRefs() bulk-returns id + metadata for all corpus rows. Load those hashes once per sync and skip chunks whose content is unchanged — no embed, no upsert. Skipped chunks still land in expectedIds so stale cleanup does not delete them. A new 'unchanged' counter is reported in stats and the completion log. Measured across consecutive restarts on a real install: before: indexed 2505/2505 every boot, store 76M -> 9.2G after: indexed 1 (one file edited), unchanged 2504 after: indexed 0, unchanged 2505 — store flat at 78M
- Skip logic now checks corpus_document_sha256 in addition to corpus_content_sha256, so edits elsewhere in a document trigger reindex of its chunks even when a chunk's own text is unchanged (prevents stale line-range citations) - Fix test assertion at line 163: add unchanged to expected shape - Add two-sync regression: force sync skips all unchanged chunks (zero embed calls, zero upsert calls, unchanged=4, staleDeleted=0) - Add document-fingerprint-change regression: editing a file triggers reindex of its chunks (embedding calls > 0)
781f344 to
1899fbb
Compare
rwmjhb
left a comment
There was a problem hiding this comment.
Re-reviewed at 1be4c49. The previous blockers are resolved: unchanged detection now requires both the chunk content hash and document hash, the result-shape assertions are updated, the forced second-sync regression covers the no-embed/no-upsert path, and the generated runtime artifact is committed. The branch is mergeable and all required CI checks are green.
Non-blocking follow-up: the document-edit fixture still changes both hashes at once, so a multi-chunk test that edits one chunk and asserts exact reindex/unchanged counts would more directly protect the document-hash-only path.
Fixes #981.
runSync()re-embedded and re-upserted every chunk on every sync —stats.skippedonly counts errors, so there was no unchanged path. Each pass creates a new LanceDB version set, andrunStorageMaintenance()clamps retention to ≥ 1 day, so on restart-heavy installs the store grew ~1GB per boot (76M → 9.2G here) with no same-day way to reclaim it.The fix reads back what
toMemoryEntry()already writes:corpus_content_sha256, bulk-loaded once per sync via the existingstore.listCorpusEntryRefs(). Chunks whose hash matches are skipped (no embed, no upsert) but still land inexpectedIds, socleanupStaleCorpusEntries()behaviour is unchanged. Adds anunchangedcounter toCorpusIndexStatsand the completion log.Measured across consecutive restarts on a real 2,505-chunk corpus:
Also saves 2,505 embedding calls per restart.
tsc --noEmitclean. Independent of the #980 PR; both touchcorpus-indexer.tsin non-overlapping regions.