semantics.md

Codexa semantic toolkit

codexa.semantic.* — pure-Python, offline-first primitives for entity normalization, keyphrase extraction, near-duplicate detection, query expansion, summarization, and cluster labeling. Used by the indexer's analysis pass and by the search panel's "🧠 Passage insights" expander.

← README

Table of contents

1. Statistical primitives — no embedder needed
2. Embedding-based primitives — reuse the index embedder
3. Where these surface in the UI

Statistical primitives

Pure-python, no embedder, no network:

• normalize.canonicalize(text) / apply_aliases(...) — fold surface variants under a single canonical key.
• linking.KnowledgeBase — load a YAML/JSON KB and look up entities through alias-aware exact match. metadata.entities.extract_entities(..., cfg=cfg) resolves ranked spans through cfg.metadata.knowledge_base_path when configured, collapsing aliases to canonical records before storage (WIRE-23).
• phrases.extract_keyphrases_rake(text) — RAKE keyphrase extraction.
• pmi_collocations(text) — PMI-ranked token pairs.
• phrases.tokenize_for_indexing(text, *, cfg) — cfg.semantic.tokenizer enum (unicode / url-aware / cjk-aware) routes both indexing.bm25_index._tokenize (write side) AND search.fusion._tokenize (in-memory rerank) so the sparse pipeline honours operator-set tokeniser shape (WIRE-18, Phase 86).
• lang_detect.detect_language(text) — fast script-aware language detection; powers per-locale extractive follow-up stubs, the LLM follow-up prompt locale, and the prompt builder's locale directive resolver.
• cooccurrence.CooccurrenceGraph.from_buckets(...) — entity co-occurrence graph; optional to_networkx() export.
• cluster_labels.label_cluster_tfidf(members, corpus) — distinctive terms relative to the wider corpus, with an LLM-assisted variant (label_cluster_llm) that falls back to the statistical label when the LLM is disabled or the call fails.

Embedding-based primitives

Still offline — these reuse the indexer's already-loaded embedder, no extra weights:

• dedup.dedupe_indices(vectors) / dedupe_items(items, vectors) — greedy order-preserving near-duplicate filter on embedding vectors. Catches paraphrase duplicates the lexical MMR pass can't see. Vector dedup is wired behind search.semantic_dedup.enabled; lexical MMR is independently wired behind search.semantic_dedup.mmr_enabled before and after PRF.
• phrases.extract_keyphrases_embed(text, embed_fn) — KeyBERT-style keyphrases: rank candidate phrases by cosine to the document embedding, MMR-diversified. Higher quality than statistical RAKE at one embed pass.
• expansion.rocchio_expand(query_vec, feedback_vecs) — pseudo-relevance-feedback query expansion (Rocchio). Wired into search behind search.prf.enabled; only ever adds recall (results are merged, never replaced).
• summarize.summarize_extractive(text, embed_fn) — LexRank: PageRank over a sentence-similarity graph, returns the most central sentences in original order. Offline per-document abstract.

embed_fn is any list[str] → np.ndarray callable — LocalEmbeddingModel.embed from the index, or a stub in tests.

Prompt context budgets

search.context_budget_unit controls how retrieved passages consume search.context_max_chars. The default chars mode preserves the character budget. tokens mode reuses the configured embedding model's loaded tokenizer, with truncation disabled, before admitting each passage. Blocking, streaming, cached-answer, FLARE, and ungrounded-regeneration paths use the same counter. Tokenizer-load failures log degraded mode; embedders without this optional capability retain the legacy whitespace proxy.

Embedding input recipes

Catalog entries can own a versioned asymmetric input recipe. The multilingual E5 entries use e5-prefix-v1: corpus chunks are encoded as passage: …, while primary and expanded retrieval queries are encoded as query: …, matching the model's training contract. Other catalog and custom models keep their existing raw inputs.

The recipe id is persisted beside model id, effective dimension, and immutable model revision in manifest.meta.embedding. All four axes participate in the search fingerprint, query-cache identity, persistent embedding-cache namespace, failed-input ledger, and generation-resume contract. A populated legacy manifest or a known model/dimension/recipe/revision mismatch promotes an incremental index run to a full rebuild before change detection; a fresh empty manifest remains valid. Legacy E5 manifests also rebuild once because their vectors were created without required prefixes.

Embedding revisions

The shipped sentence-transformers/all-MiniLM-L6-v2 id defaults to Hugging Face commit 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. cfg.embeddings.revision is passed to both cache-only and explicitly authorized online loads, including the device-to-CPU fallback and codexa fetch embedding. The implicit pin applies only to that shipped id: changing model_path clears it unless the operator supplies a model-specific revision. Remote Hugging Face revisions must be full 40-hex commits; mutable tags such as main are rejected. Existing local model paths may instead use an operator-owned snapshot label, which must change whenever that snapshot's weights change. The deprecated raw model_revision key remains a read fallback under the same validation.

If search detects any four-axis mismatch, it excludes every stale dense row and serves only hydrated persistent-BM25 hits. Missing sparse state or failed hydration returns no results and emits a one-shot warning with force-reindex remediation; dense fusion, semantic dedup, PRF, and dense rerank stages do not run in this mode. Rebuild with the configured revision to restore dense retrieval.

Embedding dimensionality (matryoshka)

The default embedder is CPU-only all-MiniLM-L6-v2. For matryoshka-capable catalog models (for example BAAI/bge-m3), cfg.embeddings.matryoshka_target_dim truncates + re-normalizes vectors to a smaller nested dimension on both the index-write and query paths, trading a little recall for a faster first-stage ANN. Changing matryoshka_target_dim requires codexa-index --force-reindex — the target is baked into the persisted vectors, so querying an index built at one dimension with a different dimension corrupts retrieval (it compares vectors of different widths). The embedding fingerprint records the effective stored dim (ML-30) and trips an embedding_fingerprint_mismatch warning + BM25 fallback if they diverge, but the fix is always a force-reindex. See docs/performance.md for the runtime detail.

Hierarchical chunk identity

When indexer.hierarchical.enabled is on, every stored L0/L1/L2 row carries the hierarchy node's stable hash id, plus parent_id and doc_id. ExpandToParentStage joins an L2 child to the matching L1 node by that metadata ID; numeric ID 0 is valid and is not treated as missing.

The producer contract is persisted separately as manifest.meta.hierarchy = {enabled, version} without changing MANIFEST_VERSION or CHUNKER_VERSION. Flat indexes use (false, 0), so future active-hierarchy metadata versions do not rebuild them. A populated manifest with missing, malformed, toggled, or incompatible hierarchy identity is promoted to a full rebuild before change detection; a fresh empty manifest is repaired and stamped in place. The same identity participates in failed-input and generation-resume contracts. indexer.only_ocr rejects a required hierarchy migration because OCR-only work cannot safely replace every chunk.

Optional semantic extras

Language detection uses a pure-stdlib stopword heuristic by default. Install pip install -e ".[lang]" to add the optional fastlangid accelerator probed by semantic.lang_detect. The unmaintained pycld3 binding is excluded because it does not support Codexa's Python 3.12 baseline.

Entity extraction defaults to the NLTK proper-noun path. Install pip install -e ".[entities]" when selecting metadata.entities.backend: spacy / stanza or enabling metadata.entities.coref; spaCy/stanza language models still need to exist locally.

Rerank stages

The search pipeline runs two optional rerank stages between dense retrieval and the prompt builder (their position and cfg-gating follow the stage rules of ADR 0003):

• Cross-encoder rerank (search.rerank) — cfg.search.reranker.model_id loads a sentence-transformers CrossEncoder lazily. Scores every (query, passage) pair and reorders. The scorer cache (Phase 88-91) caps at the documented size + ships a hit/miss stats accessor.
• ColBERT rerank (search.colbert_rerank) — IR-4 late-interaction scorer behind cfg.search.colbert.enabled. Install pip install -e ".[colbert]" for the default pylate backend or the legacy colbert-ai shim. reset_colbert_scorer_cache wires into AppContext.reset, and the scorer cache is bounded with LazyKeyedCache(maxsize=2) plus close-on-evict stats. IR-22/23 can cache or persist corpus-side token vectors so repeated queries avoid re-encoding documents.

Sparse retrieval (corpus BM25)

indexing.bm25_index.BM25Index writes per-corpus postings to bm25.db at index time. search.fusion.corpus_bm25_search queries it at search time; _BM25_SEARCHER_CACHE is a LazyKeyedCache("bm25_corpus", maxsize=8, on_evict=close_searcher) so A/B sweeps across corpora don't leak mmaps (SCAL-15 / CACHE-8).

The healthy hybrid dense + sparse fusion runs through search.fusion.hybrid_fuse (RRF default, weighted optional via cfg.search.hybrid.mode). Weighted mode is alpha*dense + (1-alpha)*bm25 by default; cfg.search.hybrid.quality_weight_enabled: true additionally discounts rows by their quality_score. During an embedding-identity mismatch, persistent BM25 becomes the complete retrieval path rather than being fused with incompatible dense rows. cfg.semantic.tokenizer threads through both write + query sides so the postings tokenise the same way as the queries (WIRE-18).

History compression

search.history_compress.HistoryCompressor (DDD-1 lift from the UI layer — part of keeping turn orchestration UI-free, ADR 0003) owns the BOT-5..7 / BOT-10 / BOT-15 / RAG-6 axes:

• Bounded plain history — _turns_to_transcript renders the last K turns with stable [T{n}] prefixes + SEC-1 fencing per turn + smart-truncation on assistant bodies.
• LLM summary of dropped prefix — when cfg.search.history_compress is "auto" (budget-triggered) or "always", the dropped prefix turns get summarised into a [Recap] synthetic turn at the head of the prompt-history block. The recap budget caps at cfg.search.history_summary_budget_chars (default 1500, clamped [200, 20000], COH-10).
• Recap prompt — _build_history_summary_prompt fences the transcript + prepends _language_directive(current_locale()) so a pt_BR thread gets a pt_BR recap (PROMPT-5).
• Recap cache identity — saved extra.history_summary rows carry identity schema 2: active locale plus a SHA-256 fingerprint of the prompt recipe, provider/model chain, and resolved token/character policies. Any identity change regenerates the recap; legacy checksum-only rows miss safely (COH-39).
• Saved-session compress audit — when compaction fires, turn.extra["compress_audit"] carries (compress_mode, dropped_prefix_n, summary_checksum) so a dashboard query "which turns auto-compressed?" has a source + a saved-session reload reconstructs the BOT-7 / BOT-11 audit trail (COH-11).

Where these surface in the UI

• 🧠 Passage insights expander — set search.insights.enabled: true and the search panel renders an offline insights block under the results: an extractive LexRank TL;DR of the retrieved passages plus the top KeyBERT key phrases, computed with the search embedder already in memory. No LLM provider, no network — works on a fully offline box. Off by default; bounded to the lead ~8 passages.
• Co-occurrence graph — the Semantic Structure tab renders the entity co-occurrence pairs as an interactive Graphviz node-link graph (edge weight = shared-bucket count) above the clickable list. Always on; falls back to the list when the Streamlit build lacks graphviz_chart.
• Query-expansion provenance — passages the PRF pass surfaced (on-topic but few shared surface terms) carry an "↗ surfaced via query expansion" caption so the operator sees why a hit with no obvious term overlap is there.
• Inline entity highlight — search.highlight_entities: true bolds recognised entities inside each passage (reuses the NER already powering the entity chips). Off by default.
• Saved-session grounding replay — with search.faith.replay_score: true, a legacy assistant turn without grounding metadata is scored once on reload and its hallucination_flags are atomically merged into that turn. JSON and SQLite preserve unrelated turn metadata, and later reloads reuse the durable flags (COH-34).