research(nightly): hybrid-sparse-dense-fusion — coherence-adaptive BM25+vector search in Rust#507
Draft
ruvnet wants to merge 3 commits into
Draft
Conversation
Pass 1-3 SOTA research loop completed. Selected topic: Hybrid Sparse-Dense Fusion with Coherence-Adaptive Weighting (DAT arXiv:2503.23013 principle in pure Rust). Covers ADR-194, corpus generator, BM25+cosine legs, three fusion variants.
Pure-Rust hybrid BM25+cosine retrieval crate with three fusion strategies: - rrf_fuse: Reciprocal Rank Fusion (k=60), standard baseline - linear_fuse: min-max normalised, fixed alpha=0.5 - coherence_fuse: per-query alpha from score concentration ratio Corpus: 10 topics x 300 docs, D=128, TextDominant/VectorDominant split. All 28 unit tests pass. 9/9 acceptance tests pass.
Hardware: x86-64 Linux 6.18.5, rustc 1.94.1, release profile, seed=42 N=3000 docs, D=128, K=10, 200 queries SparseOnly recall=0.372, DenseOnly recall=0.500 HybridRRF recall=0.738, HybridCoherence recall=0.717 Coherence wins KwHeavy: +4.2 pp over RRF (0.784 vs 0.742)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nightly RuVector Research — 2026-05-25
Adds nightly research for hybrid sparse-dense fusion with coherence-adaptive per-query weighting.
What this adds
crates/ruvector-hybrid-fusion— pure-Rust hybrid retrieval crate (zero external service deps):Bm25Index: Okapi BM25, k1=1.2, b=0.75, Robertson-Sparck Jones IDFDenseIndex: unit-normalised flat cosine scanrrf_fuse: Reciprocal Rank Fusion (k=60), the standard baselinelinear_fuse: min-max normalised linear combination, α=0.5coherence_fuse: per-query alpha from score concentration ratio (novel)docs/adr/ADR-194-hybrid-sparse-dense-fusion.md— full ADRdocs/research/nightly/2026-05-25-hybrid-sparse-dense-fusion/README.md— research documentdocs/research/nightly/2026-05-25-hybrid-sparse-dense-fusion/gist.md— SEO-optimised public technical articleReal benchmark results (x86-64 Linux 6.18.5, rustc 1.94.1, release, seed=42)
Per-query-type coherence vs RRF:
Hybrid retrieval achieves +98% relative recall gain over the best single leg (0.372 → 0.738).
Research loop passes completed: 3
Top alternatives rejected: Streaming HNSW delete-repair (0.657), PQ+ADC (0.647), ColBERT MaxSim (0.627)
Build status: PASS
cargo build --release -p ruvector-hybrid-fusionTest status: PASS
cargo test -p ruvector-hybrid-fusion(28/28)Acceptance: PASS
cargo run --release -p ruvector-hybrid-fusion(9/9)Research doc:
docs/research/nightly/2026-05-25-hybrid-sparse-dense-fusion/README.mdADR:
docs/adr/ADR-194-hybrid-sparse-dense-fusion.mdGenerated by Claude Code