Skip to content

wikipedia.md

Latest commit

 

History

History
84 lines (58 loc) · 6.06 KB

wikipedia.md

File metadata and controls

84 lines (58 loc) · 6.06 KB
Raw

Codexa Wikipedia integration

Two distinct surfaces: the search-time enrichment tier (cfg.search.wikipedia.*) and ZIM-as-corpus ingestion (drop a .zim under data_dirs). Independent — use either, both, or neither.

← README

Table of contents

  1. Wikipedia background (RAG enrichment)
  2. Indexing ZIM archives directly

Wikipedia background (RAG enrichment)

cfg.search.wikipedia.* controls the offline encyclopedic background the dispatcher folds into RAG answers. Three tiers, walked in order:

  1. ZIM archive — operator-supplied Kiwix ZIM file(s). When configured AND libzim>=3 is installed, full-text searches run against the archive offline. Walked first so a configured ZIM always shadows the small bundled seed when both have a relevant hit.
  2. Bundled seed — codexa/data/wikipedia_seed.json ships 25 common English topics, with 25-topic overlays for pt, es, and fr. Always available, no setup. Cheap fallback when the ZIM misses (or no ZIM is configured).
  3. Online fallback — opt-in (online: true); breaks the offline-first guarantee (ADR 0004 is why this tier can never default on). Off by default. Cascades <locale>.wikipedia.org → en.wikipedia.org.

The Diagnostics tab's Wikipedia panel reports which tiers are reachable via seed: ✔ ZIM: ✔ online: ✘ glyphs. A configured-but-unusable ZIM (path set, libzim missing or file gone) shows ✘ with a codexa-fetch-zim pointer — silent fallthrough has bitten too many operators.

Search adds a Wikipedia fallback row only when the corpus has no usable match. For cosine indexes, search.wikipedia.fallback_max_distance (default 0.6) remains the distance ceiling. Stores reporting l2 or ip through the VectorStore.doc_space contract instead compare their bounded distance_to_similarity value with search.wikipedia.fallback_min_similarity (default 0.6). This prevents a valid legacy L2 hit from being rejected by a cosine-only raw-distance threshold.

To configure the offline ZIM tier:

# Picks the latest dated archive from Kiwix's listing, downloads it
# next to your chroma_db, and rewrites `cfg.search.wikipedia.zim_path`
# in place.
codexa-fetch-zim --lang en_simple        # ~1 GB
codexa-fetch-zim --lang pt               # ~7 GB
codexa-fetch-zim --lang en               # ~55 GB (no pictures)
codexa-fetch-zim --lang en_maxi          # ~95 GB (full text + pictures, biggest)

# Bulk-fetch every supported language and let the dispatcher blend
# hits across them. Wires `zim_dir` into config.yml automatically.
codexa-fetch-zim --all-langs                   # 7 nopic ZIMs (~150-200 GB)
codexa-fetch-zim --all-langs --include-maxi    # also pull en_maxi (+95 GB)

--dest overrides the destination directory; --no-config-patch skips the in-place cfg rewrite; --install-libzim explicitly opts into pip install libzim>=3 after download. The safer default is to leave Python packages untouched; install the [zim] extra yourself when you want the offline ZIM tier active.

Indexing ZIM archives directly

ZIM files dropped under data_dirs are now ingested like any other corpus file. The .zim extractor walks every C-namespace HTML article via libzim, strips tags, and feeds the article text to the chunker. Each article carries its # title heading so chunked passages keep provenance.

This is independent of the cfg.search.wikipedia.zim_path knob — the two surfaces serve different intents and can be wired together, separately, or not at all:

where the ZIM lives what happens cost
cfg.search.wikipedia.zim_path / zim_dir On-demand full-text lookup at search time; result renders as 📚 Wikipedia: <title> in the RAG prompt as cheap background grounding. Zero indexing cost.
Under cfg.data_dirs The articles become regular indexed passages, queryable like any other corpus document. Full chunk + embed pass — hours on a 100 GB ZIM.

Use the wikipedia knob when you want cheap encyclopedic grounding on every query; use data_dirs when the ZIM IS (part of) the corpus.

pip install -e ".[zim]"   # pulls in libzim>=3
# Then drop a .zim under one of cfg.data_dirs and rerun:
codexa index --config config.yml

Heads up: a full English Wikipedia ZIM (~100 GB unpacked, millions of articles) takes minutes per file on chunking + hours on embedding. Use cfg.indexer.max_files / the CLI's --max-files flag for ramp-up testing. The [wikipedia] extra is a kept-for-compat alias of [zim] — both pull the same libzim>=3 binding.

Licence / attribution

Wikipedia and most Kiwix Wikipedia ZIM archives are distributed under CC BY-SA. Codexa stamps Wikipedia/ZIM search rows with license, license_url, and attribution metadata when they are rendered or exported, and search-time Wikipedia background includes the attribution sentence in the RAG prompt. Saved-session and result JSON exports include those fields so redistributed passages keep their provenance.

If you index a non-Wikipedia ZIM, verify that archive's own licence before sharing exported passages. The current corpus-ingestion fallback assumes Wikipedia-style CC BY-SA metadata for .zim sources when no more specific archive metadata is available.

The dependency doctor exposes one-shot variants of both fetchers:

codexa-check-deps --fetch-zims     # bulk-pull every language, wire zim_dir
codexa-check-deps --fetch-model    # download `--largest` GGUF + patch llm.local.model_path

Both report which archives or model weights could not be fetched (network failure, sha mismatch, config-patch error) and leave config.yml untouched on failure rather than committing a partial update.