Add Harbor Framework Support

[hrdkbhatnagar]

Adds Harbor framework support to PostTrainBench, enabling anyone to run our benchmark on cloud GPUs (Modal, Daytona) without needing access to our internal HTCondor cluster.

At the moment:

• Generate Harbor-compatible task directories from PostTrainBench benchmarks
• Almost full parity with original pipeline

Tested:

• Generated task for gsm8k + qwen3-1.7b
• Ran 1-hour test with Claude Code on Sonnet 4 on Modal
• Verified end-to-end pipeline (including eval + contam judge)
• Confirmed accuracy metrics extracted correctly

Usage

  cd src/harbor_adapter
  uv sync

  # Generate a task
  python run_adapter.py --benchmark gsm8k --model qwen3-1.7b --output ./tasks

  # Run with Harbor
  harbor run \
      --path ./tasks/posttrainbench-gsm8k-qwen3-1.7b \
      --agent claude-code \
      --model anthropic/claude-sonnet-4 \
      --env modal

See src/harbor_adapter/README.md for detailed parity tracking. Key points:

• Agent timeout, GPU access, evaluation: Full parity
• Contamination judge: Parity
• Agent duration: Tracked by Harbor in result.json
• timer.sh: Minor difference (created at task generation vs job start)

Note: Right now I have skipped the installation of flash-attn in the container as we need to have a CUDA runtime for it. In modal the GPU is attached to the sandbox after the container is built, so installation doesn't occur.

Note: I have added a uv environment for us to use in PTB. This is used for using modal and harbor, and is useful in general for reproducibility

Todos:

• directly before agent is run, install flash_attn and build timer.sh
• huggingface cache in a modal storage
• before running evaluation, uninstall and reinstall major dependencies (like transformers, inspect-ai, ...). Make sure NOT to use the cache. ((Alternatively we can look into docker in docker))

[hrdkbhatnagar]

pin versions like the current images

[hrdkbhatnagar]

things that are remaining to get full parity with the original PTB implementation:

1. Separate verifier container (eval integrity)
   In our setup, we evaluate the agent's post-trained model in a different container than the one the agent trained in. This prevents reward hacking, the agent could modify eval files in its workspace to inflate its score. In Harbor, the verifier runs in the same sandbox as the agent, so it uses whatever files the agent may have tampered with. We need a way to run the verifier in an isolated environment.

2. Pre-agent shell command inside the container
   We have a timer script that the agent calls to check remaining time (out of 10 hours). It needs to know when the agent actually started. In our original setup, the host orchestrator writes the start timestamp before launching the agent container. In Harbor, I see lifecycle hooks on the Trial object (TrialEvent.AGENT_START etc.), but those run on the orchestrator side, not inside the sandbox. we need a way to execute a shell command inside the container right before the agent starts , like a pre-agent hook.

3. Downloading additional directories after a run
   After the agent finishes, we'd like to download its full workspace (/home/agent/workspace/), including the code it wrote and the fine-tuned model weights. Currently Harbor only downloads /logs/agent and /logs/verifier.

[hrdkbhatnagar]

after discussing with Alex from Harbor/tbench:

1. we could put the verifier in the tests/ directory which only gets uploaded after the agent runs

2. this is not yet supported be we should look into this, potentially make a PR to harbor

3. Artifact collection is supported now in harbor, so we should use that

[rank-and-file]

Added modal storage for hf-cache for harbor in the branch add_harbor_support_with_hfcache. This is necessary to have the pre-cached models and datasets. This is necessary for full parity.

Although there are some other changes as well, so you can probably clone the repo with this branch in another directory and ask your agent:

Consider PostTrainBench_harbor_modal and port all changes which are related to the modal storage integration for the huggingface cache to PostTrainBench_harbor.

[rank-and-file]

Also there is an upcoming change to the judge which will need to be integrated. Will post here.

[rank-and-file]

We need to hardcode baseline values to a json, instead of fetching them from the POST_TRAIN_BENCH_RESULTS_DIR.

This is needed for harbor integration (harbor should output the baseline value, in case the judge flags the run).

[hrdkbhatnagar]

Merged main into Harbor branch, @rank-and-file maybe we should push the new judge to main soon so we can pull it here. The new judge would require some major changes for Harbor

[lewtun]

Apologies for lurking, but I noticed this comment:

Right now I have skipped the installation of flash-attn in the container as we need to have a CUDA runtime for it

In case it's useful, we have flash-attn kernels available on the Hub (link) which are matched to the hardware at runtime and skip the annoying / long / brittle install of flash-attn itself. If you want to run PostTrainBench on cloud GPUs, this could be the fastest way to support such kernels across e.g. different CUDA versions

[hrdkbhatnagar]

Apologies for lurking, but I noticed this comment:

Right now I have skipped the installation of flash-attn in the container as we need to have a CUDA runtime for it

In case it's useful, we have flash-attn kernels available on the Hub (link) which are matched to the hardware at runtime and skip the annoying / long / brittle install of flash-attn itself. If you want to run PostTrainBench on cloud GPUs, this could be the fastest way to support such kernels across e.g. different CUDA versions

Hey Lewis, thanks for your comment! I didn't know about this, it would be very useful for us, especially when running on cloud providers and having full parity with the local version. Adding it to our todo for Harbor :)

[hrdkbhatnagar]

update

Pushed a substantial round of changes to the harbor adapter since last review. end to end run is now working at parity with our local pipeline for everything except verifier sandbox isolation, which is being addressed natively upstream (harbor-framework/harbor#1607).

main changes

• Build mirrors the image we used for the leaderboard
• Live log streaming (agent + verifier) via PID-1 entrypoint
• Timer parity (healthcheck-driven, absolute path)
• system_monitor.sh ported (GPU/CPU/mem/disk every 60s)
• /tests/ placement closes file-tampering attack vector
• final_model/ + agent workspace/ downloaded as artifacts after each trial
• Single sandbox model leaves package tampering open; [verifier_environment] support upstream (Feature: isolated verifier sandbox ([verifier_environment]) harbor-framework/harbor#1607) lets us drop in a fresh sandbox for verification — fork branch validated, will integrate once landed

build

template/environment/Dockerfile is now a faithful port of the production apptainer image with two intentional changes:

• --torch-backend=cu128 instead of auto — Modal's build VM has no nvidia-smi, so auto resolves to CPU torch and breaks vllm's CUDA-only xformers requirement.
• I decided not to go with the HF kernels as suggested by Lewis since that would deviate from our original setting, as the kernels lib does not re-export under the flash_attn module name AFAICT. Agents either route through kernels directly or use the transformers attn-implementation string. But this is very useful for the future versions of PTB we build!

log streaming

template/environment/entrypoint.sh runs as PID 1 and:

1. backgrounds tail -F -q /logs/agent/*.txt /logs/verifier/*.txt so agent and verifier output stream live to the Modal dashboard via the sandbox's main process stdout, mirroring our local setup.
2. backgrounds system_monitor.sh (port of src/utils/system_monitor.sh), which writes GPU utilization / memory / disk to /logs/agent/system_monitor.log every 60s.

timer (now reliable)

Old design was START_FILE="$(dirname "$0")/.timer_start" — dirname resolves relative to invocation cwd, so each new directory the agent cd'd into got a fresh sentinel and the timer reset.

New design (native Harbor healthcheck):

• [environment.healthcheck] in task.toml writes /timer_start once before agent setup begins:

  [environment.healthcheck]
  command = "pgrep -f 'tail -F' > /dev/null && (test -f /timer_start || date +%s > /timer_start)"                                            
  interval_sec = 2                  
  timeout_sec = 5              
  start_period_sec = 5                    
  retries = 3                      

tamper resistance: /tests/ placement

The single sandbox means evaluate.py originally lived in the same writable directory the agent worked in: agent could trivially modify it to inflate scores. To match condor's "verifier reads from a separate trusted source" pattern within harbor's constraints:

• Adapter's _copy_eval_files() helper now populates both environment/ (so agent has copies for iterative testing during training,
  matching condor) and tests/ (so verifier reads untampered copies).
• Harbor uploads tests/ to /tests after the agent process exits — the agent has no opportunity to write there.

Closes: agent modifying evaluate.py, templates/, evaluation_code/, contamination_judge.py, metadata.json.

Doesn't close: agent pip installing tampered packages, addressed natively in harbor-framework/harbor#1607.

artifact collection

task.toml now declares two artifacts:

[[artifacts]]
source = "/home/agent/workspace/final_model"
destination = "final_model"
                                                                                                                                              
[[artifacts]]
source = "/home/agent/workspace"                                                                                                             
destination = "workspace"
exclude = ["final_model", "__pycache__", "*.pyc", ".git", ".venv", "venv"]

After each trial, <trial_dir>/artifacts/ contains final_model/ (the trained weights the verifier evaluated) and workspace/

tested e2e on modal with 1hr tasks (bfcl, gsm8k on qwen3 1.7b)

[hrdkbhatnagar]

Hi @schneidergithub, currently the way artifact download works in harbor causes the runs to fail in PostTrainBench.

Since we need to download the posttrained model weights to the host (and then upload them to the verifier sandbox for evaluation), the artifact download for the model weight fails because of the large size. The current way of downloading the artifacts for Modal is done via their SDK, and I think the file download through it is a bit flaky.

We are now thinking of skipping the artifact download completely and instead use Modal volumes as persistent storage across the agent and the verifier sandbox. However there are some things that are missing in Harbor natively that would help in this:

• Task-level mounts - task.toml doesn't expose [[environment.mounts]] on the task-level EnvironmentConfig.
• Mounts are TRIAL-level, set via --mount CLI flags , we can't just put [[verifier.environment.mounts]] in task.toml right now
• Cross-env volume sharing - even if you set mounts on both sides, there's no concept of "this volume is shared between the agent and verifier of this trial; create it before, delete it after." That lifecycle would be on the person running the job, which could be bad
• Per-trial volume naming - would need to generate something like ptb-{trial_id} and inject it into both env configs.

[think-step-by-step]

hey @hrdkbhatnagar any timeline to have this PR merged? respect and appreciated 🙏

[think-step-by-step]

n00b q @hrdkbhatnagar , possible to let the verifier sandbox reuse the agent container? so that it avoids the weights download

[schneidergithub]

Sorry for the delayed response. The harbor maintainers are aware of this PR, just a crazy amount of work happening at the moment with trying to get some new benchmarks out in the next week or two.

[hrdkbhatnagar]

hey @hrdkbhatnagar any timeline to have this PR merged? respect and appreciated 🙏

We are currently talking with Harbor folks to get native features relevant for this PR upstreamed very soon

n00b q @hrdkbhatnagar , possible to let the verifier sandbox reuse the agent container? so that it avoids the weights download

Unfortunately we cannot allow that, as that's not how the original setup of the benchmark is, and that is by design.

I talk about it more in my earlier comment in this PR #8 (comment)

[think-step-by-step]

hey @hrdkbhatnagar any timeline to have this PR merged? respect and appreciated 🙏

We are currently talking with Harbor folks to get native features relevant for this PR upstreamed very soon

n00b q @hrdkbhatnagar , possible to let the verifier sandbox reuse the agent container? so that it avoids the weights download

Unfortunately we cannot allow that, as that's not how the original setup of the benchmark is, and that is by design.

I talk about it more in my earlier comment in this PR #8 (comment)

Thanks @hrdkbhatnagar that makes sense; Pier, a Harbor variant, supports the patch-isolation feature and maybe can use that, in case Harbor takes too much time to adopt this feature

[komikat]

@hrdkbhatnagar have you considered using modal snapshots? Lets you pull only the files in a separate sandbox for verification. If your verifiers are not in the task harbor image then you can pull them from a registry during verification time as well. Though volumes are perfectly good as well, have noticed models sometimes don't save at the right location.

[hrdkbhatnagar]

Status update: end to end on Harbor, full v1.1 pipeline parity

Big update since the last round: the adapter now runs the complete PostTrainBench v1.1 pipeline (new judging system) on stock Harbor 0.22 on Modal, with no Harbor patches, and has been tested end to end with Claude Code, Codex, and OpenCode.

The model hand off (the old blocker) is now solved

The final_model transfer failures came from two hard limits in Harbor's artifact download on Modal: a fixed 120s gzip timeout on the archive step and a 5 GiB per file cap on sandbox downloads: every PTB base model exceeds the cap. Also made an upstream fix for it here: harbor-framework/harbor#3046

The adapter now hands weights to the separate verifier through a shared Modal Volume instead:

• A [[verifier.collect]] hook (ptb_collect.sh, in the agent image) runs after the agent exits and copies final_model onto a volume mounted in both sandboxes (~7 GB in under 90s). The agent's workspace stays exactly the same as original condor layout.
• The same hook stages a size budgeted code + data snapshot (≤ 512 MiB/file, ≤ 2 GiB total) into /logs/artifacts/, which Harbor transfers to the verifier natively, so the judges always see the agent's code even when the workspace holds 10+ GB of random things. Exclude patterns by themselves couldn't do it
• The trained model conveniently persists on the volume after the run (modal volume get <vol_name> / ./final_model).

Volumes are created per run by the run_modal_task.sh wrapper for now; would be good to make make this task declared and per trial natively in Harbor.

v1.1 parity

• Prompt: instruction.md is now rendered by calling src/eval/general/get_prompt.py itself - same as the condor prompt, including the new rules and the decontamination tool section. The agent gets ../contamination_check.py + ../test_data.json at the same relative paths as condor.
• Judges: all four src/judges/ judges (contamination, API-usage, PTB-lookup, general) run in the separate verifier with the upstream prompts and tools. Traces are parsed with src/trace_parsing/. Verdicts land as judgement_<id>.json. One difference: judges authenticate with OPENAI_API_KEY instead of the ChatGPT-subscription auth.json.
• Scoring: the Harbor reward is the pre-fallback accuracy. harbor_to_results.py exports trials into the standard results layout, so scripts/collect.py (baseline fallback), find_flagged_runs.py, and src/judges/run_judges.sh (judge reruns) work on Harbor runs unchanged.
• Resources: 16 CPUs / 128 GB RAM / 1x H100 in both sandboxes, matching single_task.sub, with Modal's H100! opt-out so runs aren't silently upgraded to H200. (Modal ignores the disk request; sandboxes get more than condor's 400 GB.)
  -Agent params: effort high, BASH_MAX_TIMEOUT_MS, CLI versions pinned to the opus_5.def by default (--cli-version latest mimics condor's autoupdate, recorded in result.json). Thinking display parity uses the new thinking_display kwarg we upstreamed in feat(claude-code): add thinking_display kwarg (--thinking-display) harbor-framework/harbor#3030 (merged today; needs the next Harbor release).

Validation on Modal (all on Harbor 0.22)

Run	Agent	Result
gsm8k, 1 h	Claude Code (Opus 4.8)	0.689 full GSM8K (base: 0.127); SFT on train split; agent used the decontamination checker; all 4 judges clean
gsm8k, 1 h	Claude Code (Opus 4.8)	0.597; showed the workspace tar issue that led to the snapshot design
infra test	Codex (gpt-5.4)	full pipeline: transcript parsed, 4 judges, eval - ~30 mins end to end
infra test	OpenCode (zai/glm-5)	full pipeline incl. PTB provider config
export	—	collect.py on exported Harbor runs produces the standard final_*.csv

Remaining / known differences

• gemini-cli untested (upstream Harbor doesn't pass --output-format stream-json, so trace parsing would degrade); _reprompt variants and cursor/grok/kimi agents have no Harbor equivalent yet (but one could easily build on the existing implementations). We don't need to include them
• Judge auth is API-key (condor: subscription).
• Per-run volume lifecycle (one has to manually remove the volumes after the runs are finished)

Full details and some known issues are in src/harbor_adapter/README.md.