Skip to content

Canonical corpus chunks are crushed by length normalization + decay — top-match content returns 0 results #985

Description

@benjaml4

Plugin version: 1.1.0-beta.11 (master) + the fixes from #982/#983 applied locally
OpenClaw: 2026.7.2-beta.6

Symptom

With the canonical corpus indexed and healthy (2,505 chunks, correct rows verified present), memory_search and memory_recall returned 0 results for a query whose target chunk is the store's #1 match on both retrieval sides — FTS raw 27.5 (top hit) and vector rank 6 within the agent's scope.

Root cause

Post-fusion lifecycle scoring is calibrated for conversation memories and structurally crushes canonical corpus chunks:

  1. applyLengthNormalization anchors at ~500 chars. Corpus chunks are line-span document chunks — ~3,900 chars here — giving 1/(1 + 0.5·log2(7.8)) ≈ 0.40. Length is a property of the chunker (already bounded by the indexer), not of entry quality.
  2. applyDecayBoost/applyTimeDecay score by entry timestamp, and corpus rows carry the source file's mtime (toMemoryEntry: timestamp: doc.mtimeMs). A six-week-old reference document lands near boostMin.

Compound: fused 0.917 (exact-keyword floor) × ~0.40 (length) × ~0.4 (decay) ≈ 0.15 → below hardMinScore/minScore → dropped. Every sufficiently old or long corpus chunk is unreachable regardless of how well it matches.

This interacts with #981's dedup direction: while memory-pro import-markdown copies exist (small, freshly-timestamped), they mask the problem by serving recall; a store relying on the canonical corpus alone loses recall almost entirely.

Fix (PR attached)

Exempt corpus entries from both stages — corpus ids are always corpus:-prefixed (buildCorpusId), so the guard is a cheap string check. Verified before/after on a live store: the 0-result query returns its target correctly with the exemption, and conversation-memory scoring is untouched.

An alternative, if you'd rather keep some ageing on corpus content, would be a separate (much gentler) decay profile for openclaw_corpus entries — happy to rework the PR that way.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions