Common error modes (first-run model cache misses, Chroma sqlite lock, segment corruption) and the structured-log surface used to diagnose them.
← README · Most entries here are the operator face of two decisions: soft-fail degradation (ADR 0011) and the two-process/advisory-lock model (ADR 0006)
- Troubleshooting
- All searches return zero hits
- Embedding model is not in the local cache
- Chroma sqlite store locked
- Database is locked (chromadb traceback)
- Progressive first checkpoint is blocked
- HNSW segment corrupt
- Weak relevance scores on every query
- Indexer appears stalled
- UI hangs on a search, nothing logs
- Clean process state without restarting
- Half-written metadata after power loss
- Logging
The dashboard is search-only: it reads the persisted Chroma store under
cfg.store.persist_dir and never crawls data_dirs itself. On a fresh
checkout or a new persist_dir, build the index first, then reload the UI:
codexa index --config config.yml
codexa uiIf you are running from a checkout without console scripts, use:
python -m codexa.cli.indexer_cli --config config.yml
python -m codexa.uiIf the index already exists, verify that the UI and indexer are reading the
same config.yml and store.persist_dir.
Codexa's default path is offline-first (ADR 0004). If embeddings.model_path points at a
Hugging Face model ID and the weights are not already under
<persist_dir>/hf_cache, the indexer raises a ModelNotCachedError instead of
silently contacting the network.
For the default embedder, pre-populate the cache once:
codexa fetch embedding --config config.yml \
--model-path sentence-transformers/all-MiniLM-L6-v2 \
--revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41You only see this when allow_model_download: false is set (air-gapped
installs). To let this run fetch the model instead, restore the default:
allow_model_download: trueor:
CODEXA_ALLOW_MODEL_DOWNLOAD=1 codexa index --config config.ymlStrict-offline deployments should set embeddings.model_path to an absolute
local model directory that already contains the SentenceTransformer files.
embeddings.backend: "llama_cpp_vulkan" deliberately has no CPU fallback.
Common failures and remedies:
llama-cpp-python is unavailable— install thevulkan-embeddingsextra, then install/rebuild the binding withGGML_VULKAN=onas shown in the performance guide.GPU offload unavailableornot a verified Vulkan build— a CPU-only or different accelerator build is installed. Force-reinstall the Vulkan wheel or rebuild withCMAKE_ARGS="-DGGML_VULKAN=on".GGUF model not found/must be a .gguf file— pointembeddings.model_pathat an existing local embedding GGUF, not a Hugging Face id or SentenceTransformer directory.- No Vulkan device appears — verify the vendor ICD with
vulkaninfo --summary. IfGGML_VK_VISIBLE_DEVICESis set, confirm its host-specific index and set it before starting the indexer or Streamlit. rank,shape, orembedding dimensionerrors — the GGUF is not exposing a pooled sentence embedding compatible with Codexa. Use a qualified embedding conversion/model; do not use a generative/instruction LLM GGUF.
After changing backend or model, run codexa index --config config.yml --force-reindex. Search rejects incompatible dense vectors and falls back to
BM25 until the new generation publishes.
The persisted index and live embedder disagree on model id, effective dimension,
input recipe, or immutable revision. Codexa excludes the incompatible dense rows
and serves only hydrated hits from the persistent bm25.db. If that sparse index
is missing, empty, filtered out, or cannot hydrate its documents, search returns
no rows and logs embedding_mismatch_sparse_unavailable once with remediation.
Confirm embeddings.model_path and embeddings.revision, then rebuild all vectors
and sparse state:
codexa index --config config.yml --force-reindexFor a custom Hugging Face id, use that model's full 40-hex immutable commit;
mutable tags such as main are rejected. Omitting revision leaves a custom
remote model unpinned and does not inherit the shipped MiniLM commit, so pin it
before building a persisted production index. Existing local model paths may
use an operator-owned snapshot label and must advance it when weights change.
ChromaDB's sqlite leaves sidecars such as chroma.sqlite3-shm / chroma.sqlite3-wal (or, on sharded_chroma, named shard sidecars like pdf_a_0000000000.sqlite3-wal) while a writer process holds the database. A clean shutdown removes them; a kill -9, OOM, or power loss leaves them on disk and the next PersistentClient open fails with the lock error.
The store layer retries the open with exponential backoff to cover the transient case (an indexer is genuinely running). When the retries exhaust, choose one:
-
An indexer is running — wait for it to finish (
codexa-indexprints a finish summary; the dashboard's Index tab shows the holder's pid). Then re-open the dashboard. -
No indexer is running — the sidecars are stale. Clear them:
codexa clear-stale-lock --config config.yml # removes them codexa clear-stale-lock --config config.yml --dry-run # preview
The same action exists as a
Clear stale chroma lockbutton in the dashboard's Index tab (only visible when sidecars actually exist + no indexer is running).
The cleanup helper refuses to delete the sidecars when the IndexerLock reports a live holder, so a click-time race between "start indexer" and "clear lock" can't corrupt a fresh run's journal.
Same root cause; this is the underlying chromadb traceback before our retry layer wraps it. There are two surfaces:
- During
PersistentClientstartup, Codexa retries for~15 stotal wall time before raisingChromaInitError. - During index writes (
add_chunks, metadata upserts, deletes), the native Chroma error is surfaced. Codexa deliberately does not probe the database through a second raw SQLite connection because that races Chroma's Rust log compactor. Normal index runs are serialized by the indexer lease; stop any separate process writing directly to the same store.
Codexa excludes ChromaDB 1.5.9 because that release can lose embedded-store writes across a rapid close/reopen and can report log-compaction failures. Run uv sync --frozen to restore the tested version from uv.lock if an existing environment already installed 1.5.9.
If the error keeps repeating, either another Chroma writer is genuinely active or a previous process left the sqlite sidecars stale. Stop the competing writer, wait for the active indexer to finish, or use codexa clear-stale-lock --config config.yml --dry-run to verify stale sidecars before removing them.
Before allocating a staging generation, a clean full-corpus build logs
progressive_first_checkpoint_verdict. An eligible verdict selects progressive
indexing, whose first successful checkpoint publishes an immutable CURRENT
generation that searches can use while later checkpoints continue.
An ineligible verdict names every blocker. In particular, inactive classic
gen_NNNNNN directories and legacy root store files are never allowed to cause
a silent fallback to an all-or-nothing build. A normal run stops without changing
them and asks for an explicit codexa index --force-reindex. On that forced run,
Codexa atomically moves only recognized inactive residue into
.codexa-progressive-migration/, records the move in MIGRATION.json, rechecks
eligibility, and starts progressive indexing. The migration does not delete data.
Codexa refuses automatic migration if CURRENT or PUBLISHED exists, or if a
generation marker cannot be inspected safely. Recover or inspect the named state
instead of deleting it blindly. Keep a completed quarantine until a new CURRENT
has published and a real search has passed; then remove only the selected migration
directory with normal filesystem tooling. Generation GC does not manage this
pre-publication quarantine.
Manifest schema v2 replaces the sampled large-file fingerprint with a full
streamed digest. Versionless, malformed, and explicit-v1 JSON or SQLite
manifests cannot safely prove that every indexed source still has the same
content, so a normal index run rejects them before reading file rows and
performs a full rebuild only when store.publish: generations can stage and
validate the replacement before switching readers. An in-place flat store
refuses automatic recovery before touching its live dense or sparse state;
restore a valid manifest backup, or back up the flat store and explicitly run
codexa index --force-reindex. The safer option is rebuilding in a separate
generation-published persist_dir before switching readers. An
indexer.only_ocr run stops with an actionable error; run a full index first.
Manifest writers always stamp v2. SQLite partial checkpoints also clear
incompatible rows before applying a v2 diff, so they cannot relabel untouched
v1 records. If a generation build stops after the strict version check but
before its first checkpoint, the old manifest remains v1 and the next run
safely requests the staged rebuild again. For a large or important corpus, set
indexer.backup_before_reindex: true before upgrading.
OCR cache keys use namespace v2. Existing v1 cache rows stay on disk and miss lazily until normal retention prunes them; newly written v2 rows round-trip as usual. The failed-input cooldown ledger is likewise v2: v1 records are ignored and the next timeout write replaces the ledger under the new contract.
Populated manifests without meta.hierarchy rebuild once so every hierarchical
chunk receives its stable node ID and flat indexes receive the explicit
(enabled=false, version=0) identity. Codexa performs this gate before change
detection and stamps JSON or SQLite metadata without changing manifest schema
v2 or chunker version 2. A fresh empty manifest is stamped without rebuilding.
Changing indexer.hierarchical.enabled, finding malformed hierarchy metadata,
or opening an active hierarchy built with another hierarchy metadata version
also requests a full rebuild automatically. An indexer.only_ocr run stops;
run a normal full index first so all dense, BM25, and parent-link metadata are
replaced together. If one source cannot produce a valid hierarchy, Codexa logs
hierarchy_decomposition_failed, leaves that source pending, and continues;
inspect the source and rerun indexing instead of accepting flat fallback rows.
The chunk worker discarded three consecutive extraction attempts because the
source's device, inode, modification time, metadata-change time, or size changed
between its pre- and post-extraction checks (source_snapshot_drift_retry
reports the first two
discarded attempts). Codexa does not place this failure in the chunk-timeout
cooldown: stop or pause the process writing that source, then run indexing
again normally. Any prior manifest row is removed immediately because its old
chunks may already have been purged; the stable retry will repopulate it with a
matching content hash, modification time, and size.
A Chroma query found either a dangling vector/document ID or an HNSW segment that cannot be loaded, typically after an interrupted write, partial disk write, or incompatible Chroma upgrade. The query returns a typed incomplete outcome and logs chroma_query_corrupt_id (corruption_family=dangling_id) or chroma_query_hnsw_load_failed (corruption_family=hnsw_load) with rebuild remediation. Sharded search keeps rows from healthy shards, but Streamlit warns that retrieval is incomplete and RAG refuses to use partial evidence. Unknown Chroma internal errors are never swallowed. Back up the store and rebuild it with codexa index --force-reindex.
The doc collection's HNSW index is in the legacy l2 (squared-euclidean) space instead of cosine. Codexa's relevance scoring (scoring.distance_to_similarity's 1 - d/2) is written for cosine distance ∈ [0, 2]; on un-normalized MiniLM vectors an L2 distance routinely exceeds 2, so every score clamps to ~0 and good passages look irrelevant. This happens to indexes built before the ML-1 fix: get_or_create_collection returns the existing L2 collection and ignores the metadata={"hnsw:space": "cosine"} on a hit, so a re-open can't migrate it in place.
The fix is a one-time rebuild that recreates the collection in cosine space:
codexa index --force-reindex --config config.ymlFor risky migrations, set indexer.backup_before_reindex: true first. Codexa
copies the current store.persist_dir plus manifest/index-info into a stable
corpus namespace below <persist_dir>.backups/ (or below
indexer.reindex_backup_dir when set) before it clears the manifest or resets
Chroma. This lets multiple corpora safely share one backup root; retention and
stale-temporary cleanup stay inside each corpus namespace. Legacy backups stored
directly below the root remain untouched. A backup is safe to restore only after
its .complete marker exists; Codexa publishes through a temporary directory
and keeps the newest five complete backups per corpus. To roll back,
stop Codexa, move the broken store.persist_dir aside, copy the backup's
store/ directory back to store.persist_dir, and restore any files under
metadata/ to the configured metadata paths.
You'll see this surfaced three ways: a chroma_space_mismatch WARNING in the indexer log, an entry in the dashboard sidebar's get_health_status probe (probe_doc_space) flags it on every health refresh until the rebuild lands.
If the configured embedding model no longer matches the one the index was built with, semantic retrieval falls back to lexical search rather than returning nonsense scores. The get_health_status now report "Semantic retrieval is degraded to lexical search" once a search has hit the mismatch — the flag is per-process and set on the first degraded query, so it appears exactly when the degradation starts costing you answers. Either restore the model the index was built with, or rebuild with codexa index --force-reindex.
Production has two different timeout surfaces:
indexer.chunk_file_timeout_s(default1800, 30 minutes) is the total extract-and-chunk deadline for one file. It records an overdue file as an error, logschunk_worker_timeout, reaps that pool generation, and retries pending files in a fresh pool.indexer.chunk_pool_max_restartslimits consecutive restarts without a successful file completion, so healthy progress resets the streak. Raise the deadline for legitimately large files or rerun with--retry-failed; set the file timeout to0only to opt into an intentionally unbounded wait.indexer.chunk_timeout_max_attempts(default3) stops the retry loop. An input that has timed out that many times on unchanged bytes under the same processing contract is quarantined: it is skipped instead of costing another full deadline every run, andchunk_timeout_quarantinednames the files. Editing the file clears the quarantine on its own;--retry-failedoverrides it for one run;0restores the retry-forever behaviour. A deferred or quarantined input does not invalidate a paused generation's resume plan: the plan row stays pending and a later pass indexes it once the cooldown expires.ocr.engine_timeout_s(default0, disabled) is a separate deadline for each OCR-engine call inside that file. Set it below the file deadline when a native OCR engine can wedge; expiry returns an empty engine result and logsocr_engine_timeout.indexer.stall_timeout_s(default1800, 30 minutes) logslane_stalled,indexer_stalled, orembed_flush_stalledwhen no progress is observed for that long. Set0to opt out. The producer-lane deadline behindlane_stalledis raised to clear the per-file budget —max(stall_timeout_s, chunk_file_timeout_s + 60)— because a lane is legitimately silent for as long as one file may take, so at the shipped defaults it fires at 1860s, not 1800s. The daemon-statusIndex daemon progress is stalecheck uses the same derived deadline:progress_atis stamped per checkpoint off the embed loop, so one slow file starves it for that file's whole budget.indexer.stall_fail_after_intervals(default2) turns a producer-lane stall into a boundedlane_stall_timeoutfailure after two effective windows (about 62 minutes at the shipped per-file budget) and reaps the phase's child workers. The index daemon applies the same semantic-progress deadline to its one-shot child and reaps the child's whole process group, so fresh worker heartbeat timestamps cannot hide a stuck native call. Set0for warn-only operation.embed_flush_stalledremains warn-only inside a non-daemon run, but the watchdog marks the flush so remaining embed batches in the same run shrink automatically after that call returns.
If the log says embed_flush_stalled, the main process is inside one native embed call that cannot be interrupted safely; once it returns, Codexa logs embed_flush_split_retry and halves the remaining batch size for this run. Lower embeddings.batch_size or embeddings.main_thread_cap for the next run if the host is still CPU-bound. If it says lane_stalled / indexer_stalled, inspect the latest chunk_error / chunk_worker_timeout rows. A single overdue file is not a reason to abort: the pool recycles the timed-out worker itself, so look for chunk_pool_restarted and chunk_timeout_checkpointed following the warning and let the run continue. Intervene only when the warning repeats with no restart and no new files: set stall_fail_after_intervals if you want automatic abort/reap on the next run, or interrupt once to use the top-level teardown now.
A non-UI plugin that exceeds its configured timeout_s can keep running in its worker thread because Python cannot safely kill a thread. Codexa immediately quarantines the plugin. A later unload, reload, or registry reset also detaches its hook routing and owned strategies, but logs plugin_lifecycle_drain_timeout and retains a non-routable tombstone while the worker still owns a lease; teardown() is not called under executing plugin code. Wait for the hook to return and retry the lifecycle operation. If runtime_stats() continues to show lifecycle_state: draining with non-zero leases, restart the process. A replacement using the same entry point or plugin name remains blocked until the tombstone drains.
During a native Chroma reset or purge, the first Ctrl+C records cancellation but retains the indexer writer lock until that native mutation returns. This prevents a second writer from entering the same store. If the native call is wedged and immediate termination is worth a potentially partial store, press Ctrl+C a second time to force-kill the process; rerun indexing afterward.
The _install_stall_debugger watcher is one of the four sanctioned UI daemon threads in
the canonical architecture inventory. It exposes an
on-demand all-thread traceback for diagnosing a wedged search (Streamlit runs inside a
ScriptRunner thread so the usual signal.signal / faulthandler.register
main-thread-only hooks don't reach the dispatcher). Touch the trigger file and the daemon
writes every thread's stack to stall_traceback.txt:
touch "$PERSIST_DIR"/logs/STALL_DUMP # the log dir the run resolved to
cat "$PERSIST_DIR"/logs/stall_traceback.txt$PERSIST_DIR stands for store.persist_dir from config.yml; the same rules as every other log apply — CODEXA_LOG_DIR wins if set, and with no store configured the dump lands in $XDG_STATE_HOME/codexa/logs (else ~/.local/state/codexa/logs). The dump uses a tmp + atomic rename (ROB-27) so a kill mid-dump preserves the previous trace instead of zero-byte overwriting it. The trigger file is unlinked on success so the next touch fires a fresh dump.
The search computation itself runs on codexa-search-turn. The script thread stores an
authenticated TurnHandle in session_state; the worker publishes locked plain data into
that handle. The worker never reads or writes session_state. The script thread renders
the buffered canonical grounded answer. Stop is cooperative: it sets the handle's Event,
which the worker observes only when provider iteration returns to Python.
A blocked provider read or native operation may not yield. After the grace interval, the
render thread can abandon the handle and release the parked UI owner, but it cannot kill a
running Python thread or native operation; the worker retains its own owner until actual
exit. Use STALL_DUMP to identify the blocked stack and restart the UI process when the
underlying call never returns.
AppContext.current().reset() walks registered_reset_hooks(), the authoritative
full-process lifecycle inventory. Its registered_caches lifecycle hook then walks
cache_registry.registered_caches(), the authoritative cache membership; Diagnostics
cache statistics read that same cache registry. Each registration's topic supplies its
Diagnostics category; only registrations with a stats hook appear there, while reset-only
registrations still participate in a drain. The lifecycle registry owns hook dependency
ordering through ResetHook.after; the cache registry owns cache membership, reset
callbacks, and optional stats/topic metadata. This runbook deliberately does not copy
either inventory. The Diagnostics tab's Drain caches button and test fixtures that
require a full-process drain route through this entry point. There is deliberately no
codexa reset subcommand: a fresh CLI process holds no caches to drain.
For migrated LazyKeyedCache consumers, the registry hook detaches lookup-visible entries
immediately. A fresh query loads a fresh handle, while an active CacheBorrow keeps its
exact retired handle valid: close is deferred until the final borrower releases, so reset
does not interrupt the old query. The close callback runs outside the cache lock. A
successful drain therefore means retired entries are unavailable to new work, not that
every native resource closed synchronously. The ownership and generation close order are
ADR 0010 and
ADR 0018.
Reset is best-effort: one failing lifecycle hook does not prevent later hooks from running.
Individual cache callback failures emit cache_reset_failed; lifecycle-hook failures emit
app_context_reset_failed. The Diagnostics warning displays valid cache and lifecycle-step
identifiers verbatim, while unsafe or private-looking names use bounded translated labels;
raw exceptions are never displayed. Later lifecycle hooks still run. Retry the drain; if
state remains stale, finish active searches and restart the UI process.
Executable evidence covers the
borrow/deferred-close contract,
Diagnostics reset during active plain and sharded queries,
and the shared registry reset/stat seam.
Every metadata write (manifest stubs, index_info.json, cfg patch from cli/_base.patch_yaml_scalar, pilot store compact, faulthandler dump) routes through utils.io.atomic_write_text (ROB-22..25; the posture is ADR 0011). The helper runs tmp + fsync + rename + parent-dir fsync so a kill -9 mid-write can't leave the file truncated. The next process load reads either the old file or the new one, never zero-length.
If a process did die between fsync and rename on a non-durable filesystem (CI tmpfs), the helper degrades best-effort — the file lands without the durability guarantee, but the load path still sees the right shape.
Pilot JSONL mutations require a native cross-process advisory lock. A
pilot_lock_unavailable event means Codexa refused the operation before
touching pilot_runs.jsonl; verify the sidecar directory is writable and move
metadata.pilot_runs_file to a local filesystem that honours advisory locks.
With store.publish: generations, codexa store compact acquires the indexer
lock, copies CURRENT, mutates only the staging copy, and publishes it after
every shard succeeds. Plain codexa store gc is a read-only preview of orphan
segments plus abandoned or out-of-retention gen_NNNNNN directories.
codexa store gc --apply also acquires the indexer lock, but it does not always
clone the store. When there are no orphan segments, it runs the direct
lease-protected retention sweep without copying CURRENT; active readers make
a candidate busy and retained. When orphan segments need mutation, GC creates a
staging copy, removes them there, publishes only on success, and then runs the
same retention sweep. A failed shard leaves CURRENT and PUBLISHED
unchanged. If either maintenance command reports an active indexer lock, wait
for that run to finish; do not delete the lock manually.
On a large generation, compaction temporarily needs space for a full store
copy plus SQLite's VACUUM workspace. Clone telemetry appears in
generation_maintenance_staging_ready; compare copied bytes and fallback bytes
against the thresholds in ADR 0001.
This is normal retention, not corruption. An active search, answer stream,
cross-panel read, consistency probe, or evaluation can hold a shared generation
lease after CURRENT has advanced. GC asks for the candidate's exclusive lease
without waiting; when that lock is busy, the generation is retained intact and
its journal entry is not pruned.
Let the active search and any timed-out background work finish, then retry
codexa store gc --apply. A later automatic or startup sweep also retries it.
Do not remove the generation directory, PUBLISHED, or lock files manually.
If the same generation remains busy with no visible work, inspect the structured
search logs for an unfinished worker and restart the UI only after saving the
session.
IndexerLock is separate: it admits writers and reports indexer liveness.
Readers never take it, so clearing or waiting on IndexerLock cannot release a
reader-held generation. The SH/EX lifetime and nonblocking retry contract is
ADR 0018.
Run codexa health --config config.yml to expose readiness on
<store.persist_dir>/health.sock, then probe it locally:
curl --unix-socket ./chroma_db/health.sock http://localhost/healthz/healthz asks can this install serve? — readiness. Its consumer is a
machine, and the HTTP code is the answer: 200 serve, 503 do not. The body
exists to explain a 503.
/status asks what is the indexer doing right now? — a diagnostic for a
person, where the body is the answer.
Keep them apart: readiness has to stay cheap enough to poll on a timer, while a diagnostic is free to grow fields. Merging them would either make the probe expensive or make the diagnostic answer a question it cannot — an install with no run in progress is perfectly ready, and "no run" is not a failure.
Two known rough edges (OBS-72): /status also returns 503 when it reports bad
news, which is a category error — the endpoint worked, so an operator's curl
exits non-zero for a functioning route; and neither route carries both
readiness and live-run state, so neither answers "healthy and progressing",
which is the question a long rebuild actually raises. During a run, prefer the
run_health event in <store.persist_dir>/logs/indexer.jsonl: it is emitted
every 60 s with throughput, ETA, the phase-cost table and process-tree
CPU/RSS, and it keeps reporting when the pipeline stalls — a rising
counters_age_s with a live process is the stall signal.
The response separates search_ready from daemon. A failed or dead daemon
returns HTTP 503 while search_ready: true can still confirm that the last
published index remains queryable. The daemon block reports liveness, state,
failure streak, last-success age, active phase/progress age, next retry, and
bounded progress counters. No daemon-status file is neutral for one-shot
deployments. Restart a dead updater with codexa index --daemon --config config.yml; inspect indexer logs before retrying a persistent pass failure.
For a daemon on another host sharing the store, process liveness is not locally
observable. Readiness instead accepts heartbeats through the configured watch
interval or retry deadline plus a clock/jitter grace window. A stale heartbeat
returns 503; check that host and the shared store before restarting the daemon.
The endpoint returns JSON with HTTP 200 when ready and HTTP 503 when the
manifest, index metadata, disk headroom, configured Ollama provider, or vector
space check is unhealthy. This is readiness—not pure liveness—so a fresh,
unindexed install correctly returns 503. The explicitly launched sidecar binds
no TCP port and creates its socket with mode 0600. Concurrent requests share
one status evaluation. If that evaluation exceeds two seconds, the endpoint
returns 503 and reuses that failure until the blocked evaluation exits; inspect
the logs before retrying.
After deploying new code or editing settings that apply only in a fresh process, ask the running supervisor to replace its active indexing child:
codexa index --config config.yml --reload-daemonThe portable control request works during a pass or the idle interval. During a
pass, status moves through reload_pending / draining; the child stops taking
new work, waits for an accepted native operation to return, flushes its durable
checkpoints, pauses an unpublished generation, and exits with the handoff status.
The supervisor records reloading / handoff and starts the successor only
after the old process has exited and released the writer lock. The published
CURRENT generation remains queryable throughout. A successor failure follows
the ordinary daemon retry/backoff policy, and its next attempt resumes compatible
paused work.
On POSIX, sending SIGHUP to the supervisor PID in the daemon health/status record requests the same handoff. Prefer the CLI control request in scripts because it is portable. Use SIGTERM only when the supervisor itself should stop.
Logs are JSON-lines under <store.persist_dir>/logs/indexer.jsonl (rotated daily, 7-day retention) plus a stdout handler at INFO. Pass --verbose to codexa-index for stdout DEBUG. Event keys are stable English identifiers — translation only applies to user-facing CLI/UI text. CODEXA_LOG_DIR redirects the file output, which the benchmark uses to keep per-rung logs isolated. With no configured store the logs fall back to the platform state directory ($XDG_STATE_HOME/codexa/logs, else ~/.local/state/codexa/logs); if neither is writable, setup warns on stderr and logs to a temp directory rather than failing the run.
Selected event keys operators commonly grep for:
| event | level | when it fires |
|---|---|---|
run_start / run_complete |
INFO | indexer boot + tear-down |
effective_budget |
INFO | pre-flight resource-budget snapshot |
change_detection |
INFO | per-run add/mod/del / rehash_skipped counters |
flat_manifest_recovery_refused |
ERROR | an unreadable flat manifest was left untouched because automatic in-place rebuilding is destructive; restore a backup or explicitly choose the documented rebuild path |
generation_staging_ready |
INFO | incremental generation clone accounting: logical/copied/hardlinked/fallback bytes and clone duration |
generation_maintenance_staging_ready |
INFO | full-copy staging is ready for compact/GC; includes the same clone accounting |
generation_published |
INFO | atomic pointer/journal commit; includes previous generation and commit duration |
index_daemon_reload_requested |
INFO | the supervisor forwarded a reload request to the active child |
index_daemon_reload_handoff_complete |
INFO | the drained child exited and the supervisor may start its successor |
generation_gc_removed / generation_gc_remove_failed |
INFO / WARNING | retention sweep removed or could not remove a generation |
generation_interrupted_recovered |
WARNING | a build that died without a graceful unwind (kill -9, OOM, power loss) was re-armed as paused and offered to the resume path instead of being discarded; interrupted_from names the state it died in |
generation_resume_rejected_retained |
WARNING | a resume was rejected but the staging generation was KEPT — the data is intact and only this run declined to adopt it. retained_bytes is the disk cost; reclaim with codexa store gc when you no longer want it. Only the newest rejected generation is kept; older ones are reclaimed as resume_rejected:*, so retention costs at most one extra generation |
generation_resume_rejected |
WARNING | this run declined to adopt a staged generation; the build is kept (ROB-72). reason=contract_mismatch:release_contract means the contract definition changed in a Codexa upgrade, so plans frozen by an older build are invalid — nothing is wrong with your config |
resume_preview |
INFO / WARNING | codexa index --dry-run verdict for the staged generation (resume / reject + reason). Runs the whole validation read-only — use it before restarting a long build |
resume_eligibility_mismatch |
WARNING | why reason=live_eligibility_changed fired: excluded plan paths are neither eligible nor vetoed this run, with a bounded sorted sample. Check failed-input cooldowns, include/exclude globs, and whether the files still exist |
resume_source_mismatch |
WARNING | why reason=pending_source_changed / frozen_sources_changed fired: path moved since the plan was frozen and kind says how (content_hash, mtime_or_size, unreadable:<OSError>) |
generation_interrupted_state_unreadable |
WARNING | a staging generation's GENERATION_STATE.json could not be parsed, so it is neither resumed nor reclaimed — repair or remove the directory by hand |
orphan_chunk_gc_delete_failed |
WARNING | one dense orphan source could not be deleted; sparse sidecars are preserved and fields include retry/force-reindex remediation |
orphan_chunk_gc_complete |
INFO / WARNING | orphan-source GC totals (orphans_found, orphans_purged, orphans_failed); partial failure includes a deterministic sample and remediation |
chunk_error |
WARNING | extractor failure on one file (run continues) |
embedding_cache_summary |
INFO | hits / misses / total entries / bytes |
embed_oom_backoff |
WARNING | CUDA OOM mid-encode (batch halved + retried) |
auto_batch_resized |
INFO | RAM/VRAM pressure crossed a threshold |
workers_downscaled |
WARNING | configured workers: clamped under the per-kind cap |
device_probe_failed |
WARNING | requested GPU backend unavailable |
chunk_worker_timeout |
WARNING | one file exceeded indexer.chunk_file_timeout_s; the event includes the configured deadline and remediation, and the pool generation is reaped/restarted while chunk_pool_max_restarts consecutive generations make no successful progress |
chunk_timeout_quarantined |
WARNING | inputs that hit indexer.chunk_timeout_max_attempts timeouts on unchanged bytes; the event names them and they are no longer retried until edited or --retry-failed |
pdf_pages_dropped |
WARNING / INFO | pypdf could not decode some pages of this PDF; the indexed text is incomplete. WARNING once more than a tenth of the file is missing |
lane_stalled / indexer_stalled / embed_flush_stalled / index_daemon_child_progress_stalled |
WARNING | indexer.stall_timeout_s observed no semantic progress; warn-only unless stall_fail_after_intervals is enabled |
lane_stall_timeout |
ERROR | stall_fail_after_intervals converted a repeated producer-lane stall into a bounded failure |
index_daemon_child_progress_deadline_exceeded |
ERROR | The daemon child repeated the same phase/counters through the configured windows; Codexa interrupts and reaps the owned process group, then enters normal daemon backoff/recovery |
plugin_loaded / pre_extract_vetoed |
INFO | plugin lifecycle + veto |
plugin_lifecycle_tombstoned / plugin_lifecycle_drain_complete |
INFO | routing and owned strategies detached, then teardown completed after leases reached zero |
plugin_lifecycle_drain_timeout / plugin_load_blocked_tombstone |
WARNING | a worker still holds a plugin lease; fields include lifecycle state/lease count and retry-or-restart remediation |
plugin_lifecycle_teardown_timeout |
WARNING | another lifecycle operation is still inside the plugin's teardown(); the waiting reload/reset gives up rather than starting the replacement against resources the outgoing plugin has not released. Retry, or restart the process if it repeats |
chroma_query_corrupt_id |
WARNING | dangling vector/document ID; marks retrieval incomplete with corruption_family=dangling_id and rebuild remediation |
chroma_query_hnsw_load_failed |
WARNING | HNSW segment load failure; marks retrieval incomplete with corruption_family=hnsw_load and rebuild remediation |
rag_refused_incomplete_retrieval / rag_flare_retrieval_incomplete |
WARNING | primary retrieval refused before cache/provider use, or a corrupt FLARE follow-up discarded its buffered draft and stopped further generation; repair or reindex the affected shard |
flare_retrigger |
DEBUG | FLARE iterative retrieval triggered on low-confidence sentence |
rerank_skipped |
DEBUG | cross-encoder rerank was disabled, had too few rows, or could not load/predict; fields include the reason and input count |
citation_orphaned |
INFO | drop_orphan_citations removed marker(s) past the passage set (FAITH-5) |
ungrounded_regenerate / ungrounded_regenerate_outcome |
INFO | RAG-7 second pass fired + post-regen score delta (FAITH-9) |
stream_grounding_scored |
INFO | progressive streaming answer finished and was scored for grounding; fields include score/threshold and whether regeneration was enabled |
pin_dropped |
INFO | BOT-13 pinned source fell below the relevance horizon |
pilot_objective_rebased |
INFO | PilotRunStore.update(cfg=) rebased the persisted objective snapshot (BOT-28) |
pilot_lock_unavailable |
ERROR | Pilot JSONL access was refused because its cross-process lock could not be acquired; verify permissions and filesystem lock support |
pilot_lock_release_failed |
WARNING | Pilot JSONL mutation completed, but explicit unlock failed; closing the handle releases the native lock |
validator_sentinel_leak |
WARNING | _check_writable couldn't unlink its sentinel file (ROB-29) |