fix: apply fusion weighting uniformly and recalibrate the exact-keyword floor - #984
Open
benjaml4 wants to merge 1 commit into
Open
fix: apply fusion weighting uniformly and recalibrate the exact-keyword floor#984benjaml4 wants to merge 1 commit into
benjaml4 wants to merge 1 commit into
Conversation
…or (CortexReach#978) fuseResults() scored candidates differently by branch: candidates with a vector hit got clamp01(max(v*vectorWeight + b*bm25Weight, keywordFloor)), while BM25-only candidates got their normalised BM25 score directly — bypassing bm25Weight entirely. Since store.bm25Search sigmoid-normalises raw BM25 (floor 0.5 by construction), any keyword-only hit outranked semantic candidates capped at vectorWeight, regardless of configured weights. Score every candidate with the same weighted formula, and keep the exact-keyword floor on all branches — recalibrated to the sigmoid: the old 0.75 threshold corresponds to raw BM25 ~5.5, which nearly every FTS hit clears (the floor value b*0.92 >= 0.69 then outranks any vector hit capped at vectorWeight, quietly re-creating the bypass). 0.95 maps to raw ~14.7 — genuinely exceptional matches like identifiers and exact strings, preserving the floor's documented purpose of keeping those above minScore. Observed on a 7,253-row store: for queries with no relevant corpus content, unrelated keyword-only hits (normalised 0.92) were returned at floored confidence 0.85, above every semantic candidate. With this change they score b*bm25Weight (~0.28, below the default minScore) and are correctly suppressed, while on-topic queries are unaffected.
rwmjhb
requested changes
Aug 4, 2026
rwmjhb
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed head 2043eb9. Applying configured fusion weights uniformly is the right direction, but the new 0.95 keyword-floor policy introduces two retrieval blockers.
test/retriever-rerank-regression.mjsfails because a strong lexical/semantic candidate (vector0.5006586, BM250.78) now fuses to about0.5845, falls belowminScore=0.6, and is removed before reranking.- BM25 graceful degradation is broken. With default
bm25Weight=0.3, every BM25-only score below0.95is weighted below0.285and cannot pass the defaultminScore=0.3; the existing graceful-degradation test reproduces this with a0.9BM25 hit when vector search fails.
Please preserve configured weighting when both backends participate while renormalizing or using the surviving backend's score when one backend fails, and replace the universal floor with a calibrated/configurable exact-match signal that keeps existing strong lexical matches. Add boundary tests around the floor and keep both existing regressions green.
Requesting changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the confirmed remainder of #978 (post-correction).
As the reporter's follow-up established, BM25 scores are sigmoid-normalised before fusion — but the structural asymmetry stands: the BM25-only branch used the normalised score directly, bypassing
bm25Weight, so keyword-only candidates (sigmoid floor 0.5) outranked semantic candidates capped atvectorWeightno matter how the weights were configured.Two changes:
clamp01(max(v·vectorWeight + b·bm25Weight, keywordFloor), 0.1).1/(1+e^(-raw/5))reaches 0.75 at raw ≈ 5.5, which almost every FTS hit clears, so the floor (b·0.92 ≥ 0.69) quietly re-created the bypass on both branches. 0.95 corresponds to raw ≈ 14.7 — genuinely exceptional matches (identifiers, exact strings), preserving the floor's documented purpose of keeping those aboveminScore.Observed effect on a 7,253-row store: queries with no relevant corpus content used to return unrelated keyword-only hits at floored confidence 0.85; they now score ~b·bm25Weight (≈0.28, below default
minScore) and are suppressed, while on-topic queries are unaffected (verified before/after on the same store).If you'd prefer the floor threshold configurable rather than a constant, happy to add a
retrieval.keywordFloorThresholdoption instead. An alternative worth considering longer-term is rank-based fusion (RRF) — which the comment above this code already names — as it sidesteps score-scale calibration entirely.tsc --noEmitclean.