Skip to content

performance.md

Latest commit

 

History

History
177 lines (135 loc) · 20.9 KB

performance.md

File metadata and controls

177 lines (135 loc) · 20.9 KB
Raw

Codexa performance

GPU acceleration, indexer throughput knobs, and the ramp-up benchmark used to tune a new host.

← README · Decision records: ADR 0007 (the concurrency shape these knobs tune), ADR 0019 (measured worker admission and rejected scheduler options), ADR 0010 (the cache contract behind every cache listed here)

Table of contents

  1. GPU acceleration
  2. Performance knobs
  3. Benchmarking

GPU acceleration

cfg.embeddings.device: "auto" (the default) probes cuda → mps → xpu → cpu in priority order and picks the first backend that's available. Override with an explicit value when needed:

Hardware device value install hint
NVIDIA "cuda" (or "cuda:0" etc) pip install torch --index-url https://download.pytorch.org/whl/cu121
AMD (ROCm) "cuda" (yes — ROCm masquerades under the CUDA API) pip install torch --index-url https://download.pytorch.org/whl/rocm6.0
Apple Silicon "mps" default pip install torch already includes MPS
Intel Arc / iGPU "xpu" pip install torch --index-url https://download.pytorch.org/whl/xpu
Anything else "cpu" n/a

The Diagnostics tab reports which backend is currently active. For the local LLM provider, llama-cpp-python needs to be rebuilt with the matching backend flag (LLAMA_CUDA=1, LLAMA_HIPBLAS=1, LLAMA_METAL=1, LLAMA_VULKAN=1); the default wheel is CPU-only.

Performance knobs

Indexer-side

These knobs tune the two-lane producer/consumer design recorded in ADR 0007 — parallel chunk workers feeding one serial main-process embed consumer.

  • Auto-batch — embeddings.auto_batch: false to disable. Halves the encoding batch at ≥60 % RAM (or ≥80 % VRAM on CUDA), doubles back to the configured ceiling at ≤40 % RAM (and ≤55 % VRAM), hard floor of 1. Hysteresis keeps the batch from oscillating turn-to-turn; RAM is re-polled every 8 batches so the per-batch cost stays negligible.
  • Cache stats — every run logs embedding_cache_summary (hits, misses, hit rate, total entries/bytes); codexa-cache-stats prints it on demand.
  • Cross-file chunk dedup — within a single embed() call, identical strings (boilerplate / license headers / EPUB nav chunks) are encoded once and the result is reused across every position. Layered on top of the on-disk shard cache.
  • mtime-first change detection — the manifest records each file's (mtime_ns, size). On the next run, files whose stat is unchanged skip both the rehash and the comparison entirely; the change_detection log event reports rehash_skipped so you can see the win. For restored backups, rsync -a / cp -p, network/shared-FS corpora, or cross-host workflows where content can change while mtime and size are preserved, set indexer.trust_mtime: false to rehash known files every run.
  • SQLite manifest — new configs use a .db path, so checkpoints apply only changed rows instead of rewriting the full manifest. Existing explicit .json paths remain supported. To migrate an existing JSON install, stop Codexa, enable a pre-reindex backup, change the suffix to .db, and run codexa index --force-reindex; do not rename JSON bytes to a .db file. The connection pool keeps WAL-mode handles warm across partial UPSERTs.
  • Guarded worker admission — OCR remains capped at 2. Chunking defaults to the measured-safe ceiling of 8. Counts up to 16 are an explicit trial: set indexer.chunk_worker_ceiling and a positive measured indexer.chunk_worker_rss_mb. Admission never exceeds half the available logical CPUs, physical cores, or logical cores left after reserving embeddings.main_thread_cap plus one OS core. It also projects current host use plus the requested worker residents under 70% of total RAM. Missing RSS or RAM telemetry automatically retreats to 8, and every clamp emits workers_downscaled plus its reasons.
  • Scheduler verdict — ADR 0019 keeps the existing PDF/non-PDF process lanes and the eight-worker default. A matched 8/12/16-worker PDF matrix changed wall time from 424.21 s to 420.56/421.66 s while peak tree RSS rose from 8,689.5 MiB to 12,264.8/15,389.5 MiB. Global shortest-job-first, a third PDF pool, and default result spilling are rejected: none moved the ordered embed/store cost centre. Future candidates must clear the ADR's throughput, memory, fairness, output-parity, and recovery gates.
  • CPU-only chunkers — the chunking workers are spawned with CUDA_VISIBLE_DEVICES="" and 1-thread BLAS caps so they don't fight the main process embedder for the GPU or saturate every core. Linux preloads those caps in the forkserver before worker imports; the embedder parent keeps and independently respects embeddings.main_thread_cap.
  • OOM backoff (CUDA) — model.encode() is wrapped in a single-retry path: on a CUDA RuntimeError whose message looks OOM-shaped, the embedder halves its batch (embed_oom_backoff WARNING), calls torch.cuda.empty_cache(), and retries the same texts split in halves. Floors at size 1 so a single oversized chunk still surfaces the failure instead of infinite-halving.
  • Effective-budget pre-flight log — every run emits an effective_budget event with CPU worker counts, embedder limits, chroma_runtime_threads_per_client, chroma_max_open_clients, and the measured steady-state chroma_native_tasks_estimate. store.chroma_runtime_threads defaults to one Tokio worker per embedded client plus a process-wide Rayon cap because the Tokio cost multiplies by store.max_open_shards; valid TOKIO_WORKER_THREADS / RAYON_NUM_THREADS overrides in the range 1–64 win. The estimate is clients * (threads + 3) + threads, measured on Chroma 1.5.x across 2–16 open clients at 1–3 runtime threads. It splits into a tokio-rt-worker half that chroma_runtime_threads caps and a flat 2 * clients sqlx-sqlite-worker half that it does not — reported separately as chroma_native_tasks_uncapped (MT-32). At the default of one runtime thread the uncapped half is roughly half the total, so store.max_open_shards is the only lever for it. The figure is steady-state: a client evicted from the LRU pool keeps its threads until each pool's idle timer reaps them, and Tokio may create temporary blocking helpers, so a live count can sit above the estimate. Pair the event with the optional ramp-up benchmark in scripts/bench_indexer.py when tuning a new host.

Search-side

Search concurrency stays at named orchestration and pipeline-stage seams (ADR 0003); the entries below are those bounded parallel points plus per-surface memos.

  • A/B rewrite parallel retrieval — when cfg.search.query_rewrite.ab_fuse is enabled, original and rewritten queries run concurrently through separate pipeline instances on a dedicated two-worker pool before their deterministic RRF merge (PERF-56). The pool is deliberately separate from nested sub-query retrieval so two outer branches cannot consume the workers their own expansion futures need.
  • HyDE parallel fan-out — cfg.search.hyde.n controls how many hypothetical paragraphs the dispatcher draws for the dense-embedding RRF merge. The draws fire concurrently via ThreadPoolExecutor (Phase 88 PERF-31) so n=3 no longer pays 3× cold-Ollama round-trip on the latency-critical path. n<=1 skips the executor (no pool-spawn overhead).
  • ChatbotObjective memo — ChatbotObjective.from_cfg is memoised on cfg_fingerprint(cfg.chatbot) (PERF-33). The chat-thread render + escalation render + sidebar all hit this per-paint; the memo keeps the cost to one walk per distinct cfg snapshot. Cap=8 FIFO + reset_chatbot_objective_cache wired into AppContext.reset.
  • Jaccard scorer query-hoist — HistoryCompressor.relevance_compressed (BOT-9) hoists the query tokenisation outside the per-turn loop when the default Jaccard scorer is used (PERF-34). Custom scorers keep the legacy (query, body) signature so their contract is unchanged.
  • _last_user_queries — walks reversed(history) directly instead of reversed(list(history)) (PERF-35). Native Turn lists carry __reversed__ so no copy is needed; the early-break bound (len(prior) >= k) caps the walk.
  • Pilot store tombstone overlay — PilotRunStore.update (PERF-32, Phase 96) appends an _op="update" overlay line instead of re-reading + re-writing the whole JSONL. Lazy compaction (drop overlays + rewrite consolidated base records via utils.io.atomic_write_text) fires once the overlay-to-base ratio crosses 2.0 AND base_n ≥ 4. Operator UI status toggles drop from O(n) read + O(n) write to O(1) append.
  • Sub-query parallel — search.semantic_retrieve._retrieve_candidates batches HyDE / query-decomp / multilingual / synonym sub-query embeddings into one embedder.embed([...]) call, then dispatches each store.query concurrently with order-preserved results before the RRF merge (PERF-30).
  • Matryoshka view — cfg.embeddings.matryoshka_target_dim truncates and re-normalizes supported catalog models (for example BAAI/bge-m3) on both the index-write path and the query path, so stored vectors and query vectors keep the same reduced dimension (WIRE-21 / ML-7 / VEC-1). Changing matryoshka_target_dim requires a codexa-index --force-reindex: the value is baked into the persisted vectors, so querying an index built at one target with a different target compares mismatched dimensionalities and corrupts retrieval. The embedding fingerprint records the effective stored dim (ML-30), so a mismatched query now trips an embedding_fingerprint_mismatch warning + BM25 fallback rather than returning silent garbage — but the correct fix is always a force-reindex.

Cache contract

The contract and the AppContext lifecycle behind it are recorded in ADR 0010. Wired caches track the six-point AGENTS.md contract (key shape · size cap · TTL or invalidation · lock · reset hook · hit/miss counters + <name>_cache_stats()). For migrated ONNX vision, entity NLP, and learned-sparse consumers, LazyKeyedCache.borrow() returns a CacheBorrow that pins one exact entry. LRU/TTL/replacement/reset detaches a retired entry immediately so a new caller can load fresh state; an active borrower gets deferred close after its final pin, and on_evict runs outside the cache lock. A retained timeout consumer releases on actual completion, not merely at its deadline.

An in-flight stale loader cannot republish after clear or replacement. On the borrowed path it returns an owned retired value; after a timeout takeover, the old and current loader results keep separate ownership and each closes exactly once. Raw get_or_load consumers have no managed lifetime guarantee. A fork child quarantines inherited caches and borrows, rejects their use, and the at-fork handler invokes no vendor callback or destructor; an inherited loader frame may continue, but its child value cannot publish and is strongly quarantined. The cache's reported size counts only entries visible to new lookups, not detached values still held by borrowers. Relevant instances include:

  • search._query_llm_cache — TtlCache(maxsize=256, ttl_s=600) shared by rewrite, decomposition, and HyDE. Keys include query/history/locale plus the effective category-specific transform settings and full LLM provider chain; a model or transform edit misses immediately, while unrelated search tuning preserves the warm entry.
  • search.fusion._BM25_SEARCHER_CACHE — LazyKeyedCache("bm25_corpus", maxsize=8, on_evict=close_searcher). This remains a raw get_or_load compatibility consumer: the cache bounds mappings, but it does not provide a retained active-query lifetime (SCAL-15 / CACHE-8).
  • metadata.entities._BACKEND_CACHE — LazyKeyedCache("entities_backend", maxsize=2, on_evict=close_pipeline). The extraction call holds a borrow through NLP result materialization; hot-swapping cfg.metadata.entities.backend / model detaches the old pipeline without interrupting that call (SCAL-16 / CACHE-9).
  • ingestion.onnx_vision_adapter._session_cache and search.sparse_rerank._SPARSE_ENCODER_CACHE — bounded close-bearing caches whose inference/scoring operations hold lexical borrows, so reset or model replacement opens fresh state while an active operation drains safely (CONC-55).
  • search.multilingual._CORPUS_LANGUAGE_CACHE — BoundedLRU(maxsize=16) keyed by (resolved_manifest_path, mtime_ns). Manifest replacement or generation publication changes the key; a load lock provides one parse per identity across concurrent queries, AppContext.reset() clears it, and registry stats expose hits, misses, size, cap, and zero TTL (PERF-59).
  • ingestion.ocr_cache — one recipe-aware OcrCache per OCR worker/process, shared by image and whole-PDF calls. Keys combine the streamed content hash with output-changing OCR parameters; ocr.cache.max_entries defaults to 100,000, oldest rows from the largest shard are pruned every prune_every_writes writes (default 256) and at deterministic teardown, and the cache registry exposes aggregate process hits, misses, size, cap, prune count, and reset (CACHE-23).
  • ui.sessions_base._RETENTION_PURGE_CACHE — TtlCache(maxsize=128, ttl_s=300) keyed by (session_store_path, retention_days). A module lock makes lookup + claim atomic across Streamlit reruns; TTL expiry, capacity eviction, or AppContext.reset() invalidates claims, and registry stats expose hits, misses, size, cap, and TTL (CACHE-22).

Every cache hooks into AppContext.reset() so the Diagnostics tab's Drain caches button + every test fixture retires them through one entry point. A successful reset means old handles are unavailable to new work; active borrows can drain afterward.

Benchmarking

scripts/bench_scan_handoff.py exercises the startup scan/change handoff with 200,000 synthetic 100-byte paths and no filesystem fixture. It reports incremental peak RSS and exits non-zero above the 80 MiB budget; the test suite also caps growth from 100,000 to 200,000 paths at 45 MiB. Production keeps one path/stat/scope inventory, skips duplicate full-corpus stat, seen-path, and unchanged-hash structures, and releases the inventory before normal queue construction:

python scripts/bench_scan_handoff.py
python scripts/bench_scan_handoff.py --files 100000 --max-peak-mib 80

scripts/bench_run_ledger.py extends that probe through physical file/directory identity tracking, change outcomes, and lane/format routing. These current-run structures live in one temporary SQLite ledger with a 2 MiB page cache; query-backed sequences feed the chunk pools and the ledger is closed and unlinked on every normal/error exit. The authoritative manifest remains the sole in-memory O(N) corpus map:

python scripts/bench_run_ledger.py --mode fresh
python scripts/bench_run_ledger.py --mode unchanged

The CI gate runs fresh and 200,000-row-manifest overlap probes at 100,000 and 200,000 files. Each mode permits at most 25 MiB incremental handoff RSS and 12 MiB growth across that doubling; ledger bytes are reported separately because bounded memory deliberately trades RAM for temporary disk.

scripts/bench_generation_clone.py reproduces the SD-12 copy/hardlink bookkeeping evidence without an operator corpus. It reports logical, copied, hardlinked, and fallback bytes plus median classic clone and publish/GC time. It also runs ROB-116's durable progressive allocation clone and proof-bound CURRENT publication, reporting their medians and enforcing zero hardlink fallback. File/byte accounting is deterministic; timing is host-specific and informational:

uv run python scripts/bench_generation_clone.py --runs 5 --buckets 32

The 2026-08-13 reference run on the development host (3 runs, 32 buckets, 51.4 MB logical tree) measured 4.55 ms classic clone, 40.27 ms classic publish/GC, 15.49 ms progressive clone, and 23.60 ms progressive pointer publication. The progressive clone hardlinked 64 files, copied four metadata files, and used no fallback copies.

scripts/bench_progressive_manifest.py compares legacy cumulative-row admission with the proof-bound incremental-auth path over the same closed SQLite manifest:

uv run python scripts/bench_progressive_manifest.py --files 200000 --runs 5

The 2026-08-14 development-host run measured a 24.5 ms authenticated median versus 1.6571 s for legacy admission: 67.68x faster and 98.5% less wall time for that base-admission step. Setup, including construction of all 200,000 rows and their auth leaves, took 5.258 s and is excluded from the paired medians. This does not claim the whole checkpoint is 67x faster: READY still performs two ordered corpus passes to converge append-order auth leaves with path-order manifest proof evidence.

ROB-116's existing 200,000-source bounded-memory probes measured 6.03 s for terminal completion evidence, 10.65 s for the driver fingerprint, 40.22 s for the held checkpoint snapshot, and 318.70 s for the full authenticated compaction-trigger admission on the same host. Each corpus-scale scan retains its existing sub-5-MiB Python allocation assertion; the five-minute trigger is a one-time terminal operation, not a per-checkpoint or query cost.

The backend decision and reopening thresholds are recorded in ADR 0001.

scripts/bench_indexer.py ramps the indexer through a series of (workers, main_thread_cap, batch_size, sample) rungs against a small regular-file fixture made with hardlinks (or copies across filesystems). It reuses the operator's existing Hugging Face cache without copying its private path into benchmark artifacts. Each rung runs codexa-index as a subprocess inside its own watchdog:

  • kill if available RAM drops below --floor-pct × total (or --floor-min-mb, whichever is higher) — default 10 % / 2 GB,
  • kill if system pressure stays above --pressure-max for 3 s — default 0.92,
  • kill if 1-min loadavg exceeds ncpu × --loadavg-mult for --loadavg-dwell-s — default 1.2 / 10 s, the "host responsiveness" gate that fires before the desktop UI gets laggy,
  • kill on --rung-timeout-s (default 600 s).

Per-rung metrics land in logs/bench/<utc>/results.json (peak RSS, peak worker RSS, peak system pressure, mean / peak CPU %, child-tree CPU %, process-pool slot idleness, error/timeout counts, peak 1-min loadavg, files-and-chunks/sec, run duration, kill reason). The default ladder bumps one knob at a time so when a rung is killed you know exactly which lever crossed the line.

--worker-trial-16 instead runs matched 8- and 16-worker windows over the same deterministic sample. The baseline's measured peak child-worker RSS is fed into candidate admission. worker_trial_decision.json recommends 16 only when the candidate is admitted at 16, improves chunk throughput by at least 15%, stays at or below 70% RAM, adds no error/timeout events, and keeps mean process-pool slot idleness at or below 50%; every failed gate recommends retreating to 8. Use a representative corpus mix—the bundled demo files are too small to establish production throughput or memory behavior.

The accepted 2026-08-14 scheduler matrix used the same deterministic 100-PDF sample at 8, 12, and 16 admitted workers. All arms produced 98 files and 17,124 chunks with zero errors/timeouts; twelve and sixteen improved wall time by less than 1% while raising peak RSS by 41% and 77%. The default therefore remains eight. The complete decision, rejected alternatives, and mandatory gates for another scheduler proposal are recorded in ADR 0019.

Benchmark directories also contain per-rung configs, logs, and sample_paths.txt. Treat logs/bench/<utc>/ as operator-private output when the source corpus path is private; the current writer records sampled absolute paths for reproducibility.

python scripts/bench_indexer.py --help            # all knobs
python scripts/bench_indexer.py                   # default 14-rung text ramp
python scripts/bench_indexer.py --ocr-mode        # gentler 8-rung OCR ramp
python scripts/bench_indexer.py --worker-trial-16 # matched guarded 8/16 trial
python scripts/bench_indexer.py --rungs my.json   # custom ladder

--ocr-mode swaps in an OCR-specific ladder (sweeps engine, pdf_dpi, ocr_workers, sample size), restricts the fixture to image extensions (.png/.jpg/.tiff/…), and bumps the per-rung timeout to 20 min since OCR is seconds-per-page. The default engine is tesseract; paddle / both rungs need paddlepaddle>=3.0 plus a working paddleocr install before they can be exercised.