Hybrid retrieval (BM25 + vector RRF fusion) for document-RAG#1030
Open
sunnyadn wants to merge 1 commit into
Open
Hybrid retrieval (BM25 + vector RRF fusion) for document-RAG#1030sunnyadn wants to merge 1 commit into
sunnyadn wants to merge 1 commit into
Conversation
Contributor License Agreement ✅All contributors have signed the CLA. Thank you! |
…rustgraph-ai#875) Adds a sparse keyword retrieval path beside the existing vector path in document-RAG, fused by weighted Reciprocal Rank Fusion on chunk_id, behind --retrieval-mode (vector | keyword | hybrid, default vector). The keyword index is a new pluggable service (KeywordIndexService / KeywordIndexClientSpec); the first backend is SQLite FTS5, consuming Chunk messages off the ingestion stream and answering BM25 queries from one process, since the index is a single local file. Query text is sanitized into per-term quoted phrases (raw text is not valid FTS5 syntax), which also makes dotted clause numbers and error codes exact-match without a trigram index. Indexes are scoped per (workspace, collection) and dropped on collection deletion. The keyword-index client spec is only registered when the sparse path is enabled, so existing flow definitions without keyword-index queues are untouched; with retrieval_mode=vector the retrieval path is unchanged. In hybrid mode a keyword-path failure degrades to vector-only.
sunnyadn
force-pushed
the
feat/hybrid-retrieval
branch
from
43cc7fd to
a93e91d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #875.
Adds a sparse keyword retrieval path beside the existing vector path in document-RAG, fused by weighted Reciprocal Rank Fusion, behind
--retrieval-mode(vector|keyword|hybrid, defaultvector).Targets
release/v2.7per the issue's "latest release branch" note — happy to retarget to 2.6 if you'd like it there too.Design
KeywordIndexServicebase +KeywordIndexClientSpec, request/response shaped like the doc-embeddings service. Matches shareChunkMatch, so both paths key onchunk_idand the existing librarian text-hydration inQuery.get_docs()is reused unchanged.storage/kw_index/fts5), consumingChunkmessages off the ingestion stream and answering BM25 queries from one process — ingest and query aren't split like the vector stores because the FTS5 index is a single local file; a server-backed backend (Elasticsearch/OpenSearch) can be split later behind the same schema. One FTS table per (workspace, collection), dropped on collection deletion; re-ingested chunks replace their previous row.7.3.2is a syntax error; the-inAURA-7parses as a column filter), so each token is quoted as a phrase and OR-joined. A quoted phrase of sub-tokens also makes dotted clause numbers exact-match —"7.3.2"finds clause 7.3.2 without matching 7.3.1 — with the default tokenizer, no trigram index.retrieval_mode=vectorleaves the retrieval path unchanged. In hybrid mode a keyword-path failure degrades to vector-only with a warning.Testing
chunk-load;keywordandhybridmodes answered clause-number and error-code questions exactly;vectormode makes no keyword-index connections; killing the kw-index service in hybrid mode logged the degradation warning and answered from the vector path.Companion deployment wiring: trustgraph-ai/trustgraph-templates#297 (keyword-index-fts5 component).