Skip to content

Latest commit

 

History

History
469 lines (319 loc) · 20.3 KB

File metadata and controls

469 lines (319 loc) · 20.3 KB

🩺 Troubleshooting

Keyed on the messages the harness actually emits. Start with slmcode doctor.


Endpoint pre-flight

The harness probes the model server before starting a run and refuses to start with a doctor-quality block, rather than marching through every phase emitting per-agent failures. slmcode run exits 4 when the endpoint is unreachable.

cause: tip:
connection refused Nothing is listening. Start your server (ollama serve, LM Studio, oMLX) or point --endpoint elsewhere.
host not found The hostname does not resolve — check --endpoint / SLMCODE_ENDPOINT for a typo.
timed out The endpoint accepted the connection but did not answer in time. Is the model still loading?
TLS handshake failed Use http:// for a local server, or fix the CA bundle.
HTTP 401/403 unauthorized The provider rejected the API key. Set SLMCODE_API_KEY, or store it in .slmcode/auth.json.
HTTP 404 — model not found That model id is not served by this endpoint. slmcode config set model <id>.
HTTP 404 (no model named) The endpoint path is wrong — most OpenAI-compatible servers need the /v1 suffix.
HTTP 429 rate limited Retry shortly, or lower max_parallel.
HTTP 5xx from the provider The server is up but failing — check its logs.
no endpoint configured slmcode config set endpoint <url>
HTTP 404 on /models (amber) The server answered but does not list models. Fine for some backends; slmcode doctor runs the deeper check.
slmcode status --json | jq .connection
slmcode doctor --json

Edit failures

These arrive in-band, as the tool result the model sees. They are also what you will see in the Live view and the run trace.

Edit refused — old_str still contains ws_read's line-number prefix (like 42|)

The commonest small-model failure. ws_read renders a 42| gutter for navigation; it is not in the file. The message shows a before/after. pkg/evolve ships a rule (transform_args: strip_line_number_prefix) that fixes this automatically after the first time.

Edit refused — old_str is empty (or only whitespace)

An empty search used to pass strings.Contains and silently prepend new_str. The message names the three real intents: create → ws_write; append → anchor on the last 2–3 lines; insert → repeat the anchor at the end of new_str.

old_str found N times in <path>

The search text is not unique. Add 2–3 surrounding lines, or pass replace_all: true if you really mean every occurrence. Not a reason to fall back to ws_write.

Ambiguous edit refused — the search text matches N places in <path> (<strategy> match)

One of the tolerant match strategies found several candidates. An ambiguous edit is a wrong edit, so nothing is applied. Same fix: more context.

old_str not found in <path>

None of the five strategies matched uniquely. The result includes a fuzzy hint at the closest span. Re-read and retry with the exact text — never with ws_write.

[matched on first+last line anchors — … verify the result with ws_read]

The edit did apply, but only the first and last lines of old_str matched; the middle drifted. Read the file back before trusting it. Seeing this repeatedly means your model is paraphrasing spans instead of copying them — consider edit_format: search_replace or a different fast_model.

EDIT REVERTED — <path> parsed correctly before your change and does NOT parse after it

The post-edit syntax check caught a break the edit introduced, and the file is unchanged on disk. Fix the replacement text; retrying the identical edit will be reverted again. Disable with disable_syntax_check: true if a checker is misbehaving on your codebase.

⚠ syntax check failed (<tool>) on <path>

The file did not parse before either — the edit was applied and the parse error is reported so it can be fixed on the next turn rather than three tool calls later.

No-op edit refused — old_str and new_str are identical

Usually a model that has lost track of what it already did. Check the run trace for a loop.

A ws_patch per-hunk report

Multi-hunk patches are all-or-nothing. The report names which hunks anchored (anchored@120..164 exact) and which missed. A repeated multi-hunk failure is what makes pkg/evolve switch edit_format to search_replace.


Shell refusals

shell refused — "<cmd>" can execute arbitrary code, so it needs explicit operator approval

An executor (python, node, make, npx, sh, awk, go run, …) is not auto-allowed. This is a deliberate tightening: python on an allowlist is functionally identical to no allowlist. Use an allowed verification form (python -m pytest, python -m py_compile, node --check, go test), or allow it explicitly:

slmcode config set shell_allow '["make ","npx vitest"]'
export SLMCODE_BASH_ALLOW="make ,npx vitest"

shell refused — "<cmd>" modifies files outside the tool layer

A mutator (sed, cp, mv, rm, tee, patch, git checkout…). Those edits could not be checkpointed, reviewed or reverted. Use ws_edit / ws_patch / ws_write / ws_mv / ws_delete.

shell refused — command substitution "$(" is not allowed

$(…), backticks, <(…) and >(…) hide a nested command from every safety check. Run the inner command as its own ws_shell call and use its output. This one cannot be allowlisted.

shell refused — a bare & backgrounds the command

One command per call, and wait for its output. &&, 2>&1 and &>file are fine.

shell whitelist: this command writes to "<path>" (<kind>)

shell_write_guard caught a cat > file / tee clobber. Appends (>>) are allowed.

shell whitelist: "<cmd>" is not an allowed command

Not in any tier. Either it is genuinely unusual (allowlist it) or the model invented it.

shell denied by permission mode (shell=deny) / shell denied by user

shell_permission: deny, or you answered no at the gate. The model is told not to retry the same command.

command timed out after 2m0s and was killed (whole process group)

Output captured before the kill is included. The message suggests a narrower command (go test ./pkg/foo -run TestBar -short) and warns that a command needing stdin will always time out. Raise shell_timeout, or pass timeout_sec on the call (ceiling 15m).


Write refusals

write refused — .slmcode/<file> is harness control state, not project source

Tools may not write under .slmcode/ except .slmcode/scratch/. This is a privilege boundary, not a heuristic — see Permissions.

path escapes workspace: <path>

.. or a symlink pointing out of the tree. Use a project-relative path.

Write refused — <path> uses a reserved device name

Windows device names (nul, con, com1…).

review: staged <path> → …patch.json (run \slmcode apply`)`

Not an error — you are in permission: review. Run slmcode apply.


JSON and decoding

The model emits prose around its JSON

The repair ladder handles fences, prose extraction, single quotes, Python literals, trailing commas and missing braces. If it is happening constantly, the endpoint is probably at prompt_only: check whether constrained decoding was negotiated at all, and see Constrained decoding.

repair: json truncated mid-string (raise max_tokens or re-ask)

Distinct from malformation on purpose: a truncated string has no recoverable content, and appending closing braces produces a document that parses and lies. Raise max_tokens (or the role's max_tokens in model_profiles). pkg/evolve maps this fingerprint to action: raise_max_tokens.

repair: unrepairable json

The ladder ran out of rungs. Usually a model that answered in prose entirely. Check that the role has a schema contract, and that structured_decoding is not off.

Structured output silently got worse after working fine

A live demotion. When a server returns a permanent 4xx for a request that differs from a plain one only by its constrained-decoding field, that capability is demoted for that provider+endpoint+model key. Common with OpenAI-compatible proxies that 400 on unknown body fields, or servers that advertise json_schema but reject strict: true.


Timeouts

Role timeouts (context deadline exceeded)

role "context" timed out after 75s (budget 75s of the 5m0s task_timeout ceiling);
no latency measured yet for model family qwen3.8 (1/3 samples).

Role budgets are derived from the measured p95 for that role on that model family, clamped to a per-class floor and to your task_timeout. Two things make a role blow its budget:

1. Too much parallelism for a single local endpoint. This is the common one, and it is easy to miss because nothing looks misconfigured. Role budgets are wall-clock, and on one local model server every concurrent call shares one GPU. Measured on a single oMLX endpoint, running at 4-way concurrency inflates each role's observed latency about 2.5-2.7× versus running it alone — so a role that finishes in 60s solo needs roughly 160s, and blows a 75s budget it would otherwise have met comfortably.

Check what you are actually running at:

slmcode doctor | grep parallel
  parallel        4 (project)  [calibration: measured, knee 2 — concurrency knee 2
                  (4-way runs at 41% efficiency), p95 3.3s, 5 tok/s, ctx 262144]

If the measured knee is below your setting, you are paying latency for throughput you are not getting. Remove max_parallel from your config and slmcode will use the measured value; see Calibration and max_parallel is measured, not guessed. If you have never calibrated, slmcode calibrate takes about 10-25 seconds.

2. No latency evidence yet. With fewer than three observations for a (role, model family) pair the policy has no basis to be stingy and hands the role the whole task_timeout ceiling — which is also the case where the ceiling itself may be too small for the model. Calibration seeds the store from the measured decode rate so the first run is already informed, but if calibration is off (calibrate: off, SLMCODE_NO_CALIBRATE) the first runs on a new model start cold. Either calibrate, or raise task_timeout; the timeout message names the value to set.

A timed-out call is itself recorded as evidence (a censored lower bound), so a budget that was measured too low widens on the next run rather than failing forever.


Context and budget

The model behaves as if it cannot see the file

Check the effective budget. If no model profile matches your model, the pack falls back to max_context_kb (16 KB ≈ 4K tokens) regardless of the model's real window. Set a profile:

model_profiles:
  qwen2.5-coder:
    context_limit: 32768

context_length_exceeded

Classified as context_overflow and not retried — the same prompt will not fit on a second attempt either. The fix is to shrink the pack (context_role_budget, repo_map_tokens, excerpt_window_lines, skill_disclosure: cards) or raise the window.

Time to first token is seconds instead of milliseconds

KV-cache prefix reuse is not hitting. Anything you add to a prompt must be byte-deterministic and stable-prefix-first. A per-turn timestamp, a randomly ordered map, or a shuffled skill list is enough to break it for every call.

max_task_calls=10 used=10 blocked=review

The per-task LLM call budget is exhausted. It replaced an unbounded worst case (~16 calls for one task), and it is derived from max_retries, not picked: worker + self-critique + max_retries × (review + correct), which is 1 + 1 + 8 = 10 at the shipped max_retries: 4.

So raising max_retries without raising max_task_calls does nothing — the budget caps the retries first, and the run warns when you have configured that combination. Raise both together, or split the task. slmcode task show <id> names this gate under Gate, with the number of times it fired and the used= / llm_requests= counters behind it.


Gates and runs

A run finishes but nothing changed

The run says so now — the closing block reads ⚠ no files changed — nothing was created, modified or deleted on disk and gives a reason. Three reasons, in order of likelihood:

  1. The edit was refused for lack of evidence. The model returned {"status":"done","files_changed":["x.go"]} without ever writing x.go. The line under the warning says so, and slmcode task show <id> prints the reviewer's verdict, the gate that refused the task, and the (unchanged) diff of its focus files. Shrink the scope and sharpen the acceptance line — slmcode task edit T1 --acceptance "…" then slmcode run "…" again.
  2. permission: review. The edits exist as proposals in .slmcode/pending/; the block says N proposed edit(s) are held for review and offers slmcode apply.
  3. permission: dry-run. Nothing is ever written.

The change set is what this run did: files that were already modified before the run started and that the run did not touch are excluded, and .slmcode/ harness state never counts.

slmcode board says a task is done but the work is not there

Look for ⚑ forced done next to it. That marks a task closed because a human answered [d]one at the escalate gate, which overrides the evidence gate that refused it. The run summary counts those separately (1 human override — you answered [d]one at the escalate gate) and slmcode task show <id> says so in the header.

T1 needs human review and you do not know why

slmcode task show T1. It renders the scope, the acceptance criteria, the agent's last output (with its files_changed claim labeled as a claim), the reviewer's verdict and issues, the gate that refused the task, and the diff of the task's focus files — then lists what you can do from the terminal. slmcode board flags the tasks worth opening and names one in its tip.

The reviewer approves work that is not there

It should not: disk state is authoritative, hallucinated edits do not auto-approve, and repo dirt unrelated to the task does not count as evidence. If you see it anyway, capture the run trace (GET /api/queries/{id}/trace) and open an issue — that is a real defect, not a tuning problem.

The run stops at the plan gate in CI

With --on-gate-timeout unset, a headless run no longer stops there: nobody can answer and you asked for work to be done, so the plan gate auto-approves at run start and logs no TTY: auto-approving plan gate (override with --on-gate-timeout=stop).

If you passed --on-gate-timeout=stop or =reject on purpose, the run refuses before the first model call with exit code 6 and names the flag or config key — it never spends the whole budget planning and then throws the plan away.

A run that does stop always names what it kept: .slmcode/queries/<runID>/ holds board.json, PLAN.md and TASKS.md, and slmcode session resume <runID> continues from them.

shell_permission=ask is a safety gate: it never auto-approves. Headless it refuses the run up front — set shell_permission to allow or deny for unattended use.

slmcode apply exits 2

interactive review needs a TTY (use --all / --list / --json).

The QA gate wants to install dependencies

qa_bootstrap is ask by default: an agent that invented a requirements.txt should not get an unattended network install. Set it to auto if you trust the sandbox, off to forbid it.


Studio

forbidden: studio only serves loopback hosts

The request's Host was not 127.0.0.1 / ::1 / localhost. This is the DNS-rebinding guard.

forbidden: cross-origin request rejected

An Origin that is not same-origin, or Sec-Fetch-Site: cross-site. Studio emits no permissive CORS headers. For the Vite dev server, enable the dev-origin allowance (SLMCODE_STUDIO_DEV_CORS=1) — it permits exactly :5173, nothing else.

unauthorized: missing or invalid studio session token

Open the URL the CLI printed (it carries ?t=…), or send the token as X-SLMCode-Token / Authorization: Bearer.

The live feed shows a gap

An explicit event: gap {from,to} frame means events could not be replayed from the 1500-entry ring buffer. The run is fine; the log is incomplete. Token deltas are evicted first so the structural timeline survives.

Studio shows "The Studio UI has not been built"

The binary embeds no SPA, so the server is serving the placeholder page compiled into pkg/server. slmcode studio prints the same warning on startup. Only the web page is missing — the CLI, the TUI and the whole Studio API are working. Build it:

make bootstrap      # installs web/ npm deps (needs Node 22+), then builds the UI
make build

Nothing appears at cmd/slmcode/ui/ in git, and that is correct: everything the Vite build writes there is gitignored output. The one tracked file is .gitkeep, which keeps //go:embed all:ui compiling on a clone that has never built the UI.

make bootstrap / make ui-react fails with Cannot find module 'vitest'

src/api/session.test.ts:1:50 - error TS2307: Cannot find module 'vitest' or its
corresponding type declarations.

Two things were wrong, and both are fixed — if you still see this, your web/node_modules is stale and make bootstrap will replace it.

  1. web/node_modules was never installed or refreshed. make bootstrap used to short-circuit whenever cmd/slmcode/ui/assets/ already existed, so a months-old build artifact made it a no-op. It now always ensures dependencies (via make web-deps) and then builds.
  2. The production build was typechecking test files. npm run build runs tsc -b, and web/tsconfig.json now excludes src/**/*.test.ts(x) and src/test, so a missing test devDependency can no longer block shipping the app bundle. Tests are still typechecked, by npm run typecheck:test (web/tsconfig.test.json).

npm ci can only install packages when your package.json and package-lock.json are in sync

npm error `npm ci` can only install packages when your package.json and
npm error package-lock.json are in sync. Please update your lock file with
npm error `npm install` before continuing.

This is expected right now: web/package-lock.json is out of date with web/package.json. The lock predates vitest, @testing-library/*, eslint and the rest of the test toolchain, and npm ci installs strictly from the lock, so it refuses to run at all.

make bootstrap handles it — it reports the mismatch and falls back to npm install, which resolves from package.json and rewrites web/package-lock.json.

Commit the regenerated web/package-lock.json. That is the real fix. Until it is committed, every clone and every CI run pays for the fallback; once it is, npm ci works again and is both faster and reproducible.

If npm install itself fails, the npm registry is unreachable (offline, proxy, or an egress allowlist). The Go build does not need it: make build still works and the binary serves the placeholder page.

port 7420 is in use

slmcode studio --kill (only ever signals a process named exactly slmcode), or let it move to a free port, or --no-port-auto to fail instead.


Update

checksum mismatch for <asset> / <asset> is not listed in SHA256SUMS — refusing to install

The self-updater downloads the release's SHA256SUMS first and verifies the binary against it before replacing anything. A mismatch means the download was corrupted or tampered with — nothing was installed.

installing to <path>: permission denied

Try sudo, or slmcode update --user to install into ~/.local/bin.


Getting more detail

slmcode run --vv ""                     # debug-level rendering
slmcode status --json
slmcode memory show --role worker        # what the model was actually told
slmcode evolve rules                     # which repairs the harness has learned
slmcode metrics show --last 10           # pass rate, edit-apply rate, calls per task
cat .slmcode/memory/REFLECTION.md        # what happened last run

Still stuck → FAQ.