Skip to content

fix: apply fusion weighting uniformly and recalibrate the exact-keyword floor - #984

Open
benjaml4 wants to merge 1 commit into
CortexReach:masterfrom
benjaml4:fix/hybrid-fusion-weighting
Open

fix: apply fusion weighting uniformly and recalibrate the exact-keyword floor#984
benjaml4 wants to merge 1 commit into
CortexReach:masterfrom
benjaml4:fix/hybrid-fusion-weighting

Conversation

@benjaml4

@benjaml4 benjaml4 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Addresses the confirmed remainder of #978 (post-correction).

As the reporter's follow-up established, BM25 scores are sigmoid-normalised before fusion — but the structural asymmetry stands: the BM25-only branch used the normalised score directly, bypassing bm25Weight, so keyword-only candidates (sigmoid floor 0.5) outranked semantic candidates capped at vectorWeight no matter how the weights were configured.

Two changes:

  1. Uniform scoring — every candidate gets clamp01(max(v·vectorWeight + b·bm25Weight, keywordFloor), 0.1).
  2. Floor recalibration — the exact-keyword floor's 0.75 threshold predates the sigmoid: 1/(1+e^(-raw/5)) reaches 0.75 at raw ≈ 5.5, which almost every FTS hit clears, so the floor (b·0.92 ≥ 0.69) quietly re-created the bypass on both branches. 0.95 corresponds to raw ≈ 14.7 — genuinely exceptional matches (identifiers, exact strings), preserving the floor's documented purpose of keeping those above minScore.

Observed effect on a 7,253-row store: queries with no relevant corpus content used to return unrelated keyword-only hits at floored confidence 0.85; they now score ~b·bm25Weight (≈0.28, below default minScore) and are suppressed, while on-topic queries are unaffected (verified before/after on the same store).

If you'd prefer the floor threshold configurable rather than a constant, happy to add a retrieval.keywordFloorThreshold option instead. An alternative worth considering longer-term is rank-based fusion (RRF) — which the comment above this code already names — as it sidesteps score-scale calibration entirely. tsc --noEmit clean.

…or (CortexReach#978)

fuseResults() scored candidates differently by branch: candidates with a
vector hit got clamp01(max(v*vectorWeight + b*bm25Weight, keywordFloor)),
while BM25-only candidates got their normalised BM25 score directly —
bypassing bm25Weight entirely. Since store.bm25Search sigmoid-normalises
raw BM25 (floor 0.5 by construction), any keyword-only hit outranked
semantic candidates capped at vectorWeight, regardless of configured
weights.

Score every candidate with the same weighted formula, and keep the
exact-keyword floor on all branches — recalibrated to the sigmoid: the
old 0.75 threshold corresponds to raw BM25 ~5.5, which nearly every FTS
hit clears (the floor value b*0.92 >= 0.69 then outranks any vector hit
capped at vectorWeight, quietly re-creating the bypass). 0.95 maps to
raw ~14.7 — genuinely exceptional matches like identifiers and exact
strings, preserving the floor's documented purpose of keeping those
above minScore.

Observed on a 7,253-row store: for queries with no relevant corpus
content, unrelated keyword-only hits (normalised 0.92) were returned at
floored confidence 0.85, above every semantic candidate. With this
change they score b*bm25Weight (~0.28, below the default minScore) and
are correctly suppressed, while on-topic queries are unaffected.

@rwmjhb rwmjhb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 2043eb9. Applying configured fusion weights uniformly is the right direction, but the new 0.95 keyword-floor policy introduces two retrieval blockers.

  1. test/retriever-rerank-regression.mjs fails because a strong lexical/semantic candidate (vector 0.5006586, BM25 0.78) now fuses to about 0.5845, falls below minScore=0.6, and is removed before reranking.
  2. BM25 graceful degradation is broken. With default bm25Weight=0.3, every BM25-only score below 0.95 is weighted below 0.285 and cannot pass the default minScore=0.3; the existing graceful-degradation test reproduces this with a 0.9 BM25 hit when vector search fails.

Please preserve configured weighting when both backends participate while renormalizing or using the surviving backend's score when one backend fails, and replace the universal floor with a calibrated/configurable exact-match signal that keeps existing strong lexical matches. Add boundary tests around the floor and keep both existing regressions green.

Requesting changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants