harness: run outside HTCondor, and stop the eval reaping other tenants' GPU processes

[tangxiangru]

Six fixes found while running the benchmark on a shared Slurm cluster instead of an HTCondor whole-node claim. Every new knob defaults to today's behaviour — a site that sets nothing gets exactly what it gets now. No task, prompt, scoring or judge logic is touched.

Two changes are not behind a knob and do change what a current site sees; they are listed under "Unconditional changes" below rather than folded into the sentence above.

Three files: src/run_task.sh, src/baselines/run_baseline.sh, src/trace_parsing/parse_trace.py.

1. The pre-eval GPU reap kills every user's processes

nvidia-smi --query-compute-apps=pid --format=csv,noheader | xargs -r kill -9

Under a whole-node HTCondor claim nothing else is running, so this is fine. On any shared node it is not: it kills other tenants' training jobs. Replaced with reap_gpu_processes, scoped to -i $POST_TRAIN_BENCH_VISIBLE_GPUS when that is set, and switchable via POST_TRAIN_BENCH_EVAL_GPU_REAP=all|own|none. With both unset the command issued is the one above.

2. --env HF_HOME does not actually redirect the hub cache

huggingface_hub resolves HF_HUB_CACHE, else HUGGINGFACE_HUB_CACHE, else $HF_HOME/hub — HF_HOME is the lowest priority of the three. These apptainer execs do not pass --cleanenv, so on a host that exports HUGGINGFACE_HUB_CACHE globally that value wins inside the container, the overlay at TMP_HF_CACHE is never read, and the host path is not bound in. The hub then materialises the cache on the container root, which --writable-tmpfs caps at sessiondir max size (64 MiB by default), and a multi-GB download dies as

OSError: [Errno 28] No space left on device

from inside file_download.py, four frames below anything that mentions a cache. All three variables are now named. Same fix in run_baseline.sh.

3. XDG_CACHE_HOME is bound through when it exists

vLLM's VLLM_CACHE_ROOT defaults to $XDG_CACHE_HOME/vllm, and triton's kernel cache lands under the same root. Unbound, both write to the 64 MiB container root: the first is survivable (Error saving model info cache), the second is torch._inductor.exc.InductorError: OSError: [Errno 28], which kills EngineCore and takes the vLLM server with it. One bind covers both and keeps compiled kernels across runs; the HF cache deliberately still points at the overlay, so the scorer cannot write into a shared hub.

4. The evaluation container never bound the results dir

EVAL_DIR and metrics.json live under POST_TRAIN_BENCH_RESULTS_DIR, but only REPO_ROOT and the HF cache are bound into the scoring container. example.env's relative results happens to fall inside REPO_ROOT, which is why this has not bitten before; an absolute results dir writes metrics.json where nothing can read it. Adds --bind "${EVAL_DIR}:${EVAL_DIR}", and makes a missing metrics.json after the last retry fatal rather than a green exit.

This is the one intentional behaviour change in the PR. Today a run whose every evaluation attempt failed exits 0 with no metrics.

5. source resolves against the script, not the working directory

bash reads a script incrementally and seeks back to the byte after the last command it parsed, so a long-running script holds an open handle on its own inode for its whole run. Replacing src/run_task.sh mid-run killed two 10-hour jobs with

src/run_task.sh: error reading input file: Stale file handle

— each after its agent phase had finished and before it was scored. Both source lines now resolve against ${BASH_SOURCE[0]}, so a launcher can run a node-local copy of src/ while leaving the working directory on the shared checkout. The working directory still has to be the checkout: REPO_ROOT comes from pwd and the scoring container binds it by that path.

6. parse_trace.py dispatches on a substring of the agent name

Any agent directory named claude_* gets the Claude CLI's stream-json parser regardless of what it writes, producing a stub output plus one NOT PARSABLE line per input line on stderr — into error.log, the file meant to hold the harness's own errors. New --raw-only takes the path already there for an unrecognised agent (copy verbatim, still sanitize); run_task.sh passes it when the agent ships a payload/ directory.

---

Also included, inert for every agent in the repo today

Both guarded on files no current agent has:

• agents/<agent>/payload/ is copied to /home/ben/agent and the sandbox gets BENCHMARK_ID, MODEL_TO_TRAIN and NUM_HOURS — for an agent that is a repository rather than a single solve.sh.
• agents/<agent>/env_passthrough.txt lists variable names only, no values, so an agent that authenticates against a cloud endpoint with ambient host credentials can be run without inventing a provider API key.

And two opt-in knobs:

• POST_TRAIN_BENCH_TMP_ROOT (default /tmp) — the HTCondor submit file asks for request_disk=400G and gets it on /tmp; a scheduler that makes no such reservation may have far less there.
• POST_TRAIN_BENCH_ISOLATE_GPUS=1 — adds -c --nvccli so the agent sandbox gets a device cgroup. CUDA_VISIBLE_DEVICES is an environment variable, not a fence: --nv binds all /dev/nvidia*, so the agent's own shell sees every card and is one export away from using them. Measured on an 8×H100 node with NVIDIA_VISIBLE_DEVICES=0: --nvccli alone leaves all eight devices in the sandbox, -c --nvccli leaves one.

Unconditional changes

These two are not opt-in and will change behaviour on an existing HTCondor site. Both are
reporting fixes — no task, prompt or scoring logic moves — but they are called out separately
because the "defaults to today's behaviour" line above does not cover them.

• SOLVE_EXIT now reports the agent's exit code. The brace group around the agent ended
  with kill $MONITOR_PID, so its status was the kill's, and the group was piped into
  timestamp_lines.py, so the pipeline's status was python's. Both are 0 essentially always,
  so a cell whose check_cuda.py refused to start the agent recorded exit_code: 0 / status: exited normally next to final_model_files: 0. set -o pipefail plus an explicit
  exit $SOLVE_RC makes the field mean its label. Nothing downstream branches on it, so the
  only visible effect is that some runs which recorded 0 will now record the real code.

• run_task.sh exits 1 when no metrics.json was written. The script previously ended on
  an echo, so six failed evaluation attempts still exited 0 and the scheduler recorded the
  job as successful over an empty result. The check is on the artifact rather than on either
  retry ladder's return value, because the first ladder failing and the second succeeding is a
  normal run. If your tooling treats a nonzero exit as an alert, runs that silently produced no
  score will start alerting.

Happy to drop either one if you would rather they landed separately.

Testing

These have carried a full campaign on 8×H100 Slurm nodes — GSM8K post-training runs at the 1 h and 10 h budgets, scored end to end. Not run against HTCondor, which is why every change is default-off or byte-equivalent to the current command; the HTCondor path is what the defaults reproduce.

Happy to split this into six PRs if that is easier to review.