Skip to content

llm.md

Latest commit

 

History

History
106 lines (75 loc) · 7.08 KB

llm.md

File metadata and controls

106 lines (75 loc) · 7.08 KB
Raw

Codexa LLM providers

Three values of cfg.llm.provider (none / ollama / local), a curated GGUF catalog with host-probe / largest-fit helpers, and the cross-provider fallback chain.

← README

Table of contents

  1. Providers
  2. Cross-provider fallback
  3. Local provider — install + fetch-model
  4. Curated GGUF catalog
  5. Multi-part GGUFs + storage layout
  6. Minimal YAML + determinism

Providers

Codexa supports three values of cfg.llm.provider:

  • "none" — no LLM. RAG search returns retrieved passages only.
  • "ollama" — external Ollama server (cfg.llm.host + cfg.llm.model).
  • "local" — embedded llama.cpp via llama-cpp-python. Fully offline, deterministic (greedy decoding with temperature=0, top_k=1, fixed seed), no separate server.

Cross-provider fallback

Set cfg.llm.fallback to one of the other values to retry the call against the fallback when the primary is unreachable. Default is "local", so an Ollama setup keeps working when the server is down.

Local provider — install + fetch-model

pip install -e ".[local-llm]"                       # installs llama-cpp-python

# Easiest: have Codexa fetch a curated GGUF for you and patch config.yml.
codexa-fetch-model --update-config config.yml       # default: Qwen2.5-0.5B
codexa-fetch-model --list                           # full catalog with sizes/licenses
codexa-fetch-model --recommend                      # probe RAM/CPU/GPU, suggest best fit
codexa-fetch-model --largest --update-config config.yml   # biggest entry that fits
codexa-fetch-model --url <hf-link> --out ~/m.gguf   # custom mirror

--recommend prints a host probe (RAM / CPU cores / GPU VRAM) and a ✓/✗ table of every catalog entry annotated with expected throughput on this host. --largest is the one-shot equivalent: it resolves to the biggest entry that fits the host's RAM budget (size × 1.2 ≤ RAM) and downloads it.

Curated GGUF catalog

Catalog entry Size License Notes
qwen2.5-0.5b (default) 0.4 GB Apache-2.0 Smallest, fastest first run
llama-3.2-1b 0.8 GB Llama 3.2 Community 131 K context, runs anywhere
qwen2.5-1.5b 1.0 GB Apache-2.0 Balanced 32 K context
qwen3-30b-a3b 19 GB Apache-2.0 MoE (3 B active) — fastest of the big-class on CPU
qwen3-32b 20 GB Apache-2.0 Dense, hybrid thinking/non-thinking modes
llama-3.3-70b 42 GB Llama 3.3 Community Highest-quality dense for ≥64 GB RAM hosts
deepseek-r1-distill-70b 42 GB MIT Reasoning-tuned (chain-of-thought baked in)
qwen2.5-72b 47 GB Apache-2.0 Multi-part download (split + concatenated locally)
mixtral-8x22b 85 GB Apache-2.0 Multi-part; MoE (~39 B active); RAM-hungry

Multi-part GGUFs + storage layout

Multi-part GGUFs (qwen2.5-72b, mixtral-8x22b) exceed Hugging Face's ~50 GB single-file cap; the downloader fetches each part in order and concatenates them into a single .gguf so model_path: stays one string.

Models land in {store.persist_dir}/models/ by default — co-located with the chroma index so a single backup root covers both. Override with $CODEXA_MODEL_DIR (env) or --out (per-call).

Minimal YAML + determinism

llm:
  provider: "local"
  local:
    model_path: "/models/qwen2.5-0.5b-instruct.gguf"
    n_ctx: 4096
    seed: 1337
    max_tokens: 512

Greedy decoding with a fixed seed makes a given (model, prompt) pair reproduce the same answer turn after turn — useful when grounding answers in indexed corpus passages and you want results to be stable.

Prompt builders

Where these prompts sit in the turn flow — and why the LLM is called only at fixed, named points instead of driving an agent loop — is recorded in ADR 0003.

Every LLM prompt the dispatcher constructs runs through a builder that pins five contracts per AGENTS.md prompt-template-integrity rule:

  1. SEC-1 fence — user-supplied text wraps in <<<USER_INPUT>>> / <<<END_USER_INPUT>>> via utils.text.fence_user_segment before interpolation. The system prompt names the fence so the model treats the block as inert.
  2. NO_ANSWER_PHRASE — the constant lives in search.prompt_builder and the answer template + detector reference the same string so a refusal phrase emitted by the LLM is detected programmatically and routed to the BOT-25 escalation packet.
  3. Length rule — every template carries an explicit length rule (e.g. "2-3 short sentences" for the history-summary recap).
  4. Locale directive — _language_directive(current_locale()) + _language_header(...) prepend the prompt so a pt_BR session generates pt_BR sub-queries / rewrites / HyDE paragraphs / recaps / follow-ups / answers. Wired in HyDE, query_decomp, query_rewrite (PROMPT-10), history_summary (PROMPT-5), follow-ups (COH-6), and the answer template.
  5. [N] citation contract — each retrieved passage's row carries _citation_id matching its [N] order in the prompt. Orphan markers whose index points past the surviving passage set get stripped post-generate via drop_orphan_citations (FAITH-5) AND on saved-session reload (FAITH-7).

Each prompt builder ships a regression test under tests/test_phase87_prompt_integrity_* pinning the fence + locale + NO_ANSWER + length rule + [N] contract.

Answer-cache identity

The process-local answer cache keys the SHA-256 digest of the exact rendered RAG prompt together with canonical fingerprints of the full llm section and the complete search.citations, search.regenerate_when_ungrounded, and search.reranker policies. Mapping order is irrelevant; any value change that can alter generation or grounding causes a miss. _ANSWER_CACHE_KEY_SCHEMA names the recipe. Bump it whenever the key's field meaning or normalization changes; prompt wording changes invalidate themselves through the prompt digest.

RAG-7 regenerate-when-ungrounded

cfg.search.regenerate_when_ungrounded.enabled: true opts the dispatcher into a second LLM pass when the first answer's per-sentence grounding score drops below cfg.search.regenerate_when_ungrounded.threshold (default 0.3). The revision prompt cites the unsupported sentences + instructs the model to drop or revise them.

The post-regen answer re-scores against the same passage pool + emits ungrounded_regenerate_outcome with (before_unsupported_n, after_unsupported_n, score_delta_mean, before_score_mean, after_score_mean, threshold) (FAITH-9) so operators can audit whether RAG-7 actually reduced unsupported claims across the eval set. eval_harness.run_case_with_regen_comparison (RETQ-8) runs each case twice (regen off / on) and stamps the delta on EvalResult.answer_faithfulness_after_regen.