fix: exempt canonical corpus chunks from length normalization and decay - #986
Merged
rwmjhb merged 1 commit intoAug 4, 2026
Merged
Conversation
Canonical corpus recall was near-impossible: after fusion, results pass through applyLengthNormalization and applyDecayBoost/applyTimeDecay, both calibrated for conversation memories. - Corpus chunks are line-span document chunks (~3,900 chars against the 500-char anchor -> factor ~0.40). Length is a property of the chunker, not entry quality, and chunk size is already bounded by the indexer. - Corpus rows carry the source file's mtime as their timestamp (toMemoryEntry: timestamp: doc.mtimeMs), so weeks-old reference docs take the decay multiplier down to ~boostMin. Reference truth does not age like chat memory. Measured: a corpus chunk that fused at 0.917 (top FTS hit, exact-keyword floor) finished ~0.15 after both stages — below hardMinScore/minScore — so memory_search and memory_recall returned 0 results for content that is the store's CortexReach#1 match on both retrieval sides. With the exemption the same query returns it correctly. Corpus ids are always 'corpus:'-prefixed (buildCorpusId), so the guard is a cheap string check with no metadata parsing.
rwmjhb
approved these changes
Aug 4, 2026
rwmjhb
left a comment
rwmjhb
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed head 8ce9371. Exempting canonical corpus chunks from conversation-oriented length normalization and decay directly addresses the reported retrieval failure. The targeted checks and full local suite pass.
Please follow up with direct retrieval regressions covering all three exemptions and a shared corpus-entry predicate; these are non-blocking for this focused change. GitHub currently reports no check runs for this head.
Approved.
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #985.
Corpus chunks are line-span document chunks whose length is set by the indexer, and whose timestamps are source-file mtimes — so conversation-memory length normalisation and decay compound to ~0.15 on a chunk that fused at 0.917, dropping the store's #1 match below
minScore. Details and measurements in the issue.The guard is
id.startsWith("corpus:")(always true forbuildCorpusIdoutput), applied inapplyLengthNormalization,applyDecayBoost, andapplyTimeDecay. Conversation-memory scoring is untouched.tsc --noEmitclean; verified before/after on a live 2,505-chunk store.Complements #982/#983 — with all three, a canonical-corpus-only store (no import-markdown copies) has working recall.