HPC hardening: single-GPU saturation, downstream-WE parallelism, and cleanups

[namus]

Summary

Hardens PathGennie for HPC testing on CPU and GPU nodes. The headline change lets the OpenMM backend saturate a single GPU by running many swarm walkers concurrently, and makes the downstream Weighted Ensemble spread across a node's GPUs/cores. It also fixes an executable-resolution bug that broke module load binaries, speeds up on-the-fly SPIB, and removes the non-functional MPI/Dask executors. The path-refinement review fixes and repo hygiene ride along.

10 commits; 107 unit tests pass and the HPC self-check is 7/7.

What changed & why

HPC scaling (the focus of this branch)

• Single-GPU saturation (OpenMM). OpenMMEngine now backs a pool of interchangeable Contexts on one card, handed out one-per-thread via a queue with a shared, lock-guarded State cache (States are portable across Contexts of the same System). PathGennieMD sizes the pool from workers_per_device — an int, or auto (CPU cores capped by free GPU memory, probed while the pool is built). n_workers=1 is exactly the old single-Context path (no regression). Previously trials ran one at a time, leaving the GPU idle for small systems.
• Downstream WE parallelism. The backend's device pool is now forwarded through run_downstream → make_stage → WeightedEnsembleStage, so WE walker propagation spreads across GPUs/cores instead of silently running serially. No algorithm change.
• Executable resolution. AMBER/GROMACS runners resolve executable with shutil.which, so a bare module load name (gmx, pmemd.cuda, sander) works instead of failing Path.exists(); example configs use bare names.
• SPIB performance/memory. SPIBProgress caches features incrementally (was re-featurizing the whole growing buffer each refresh, ~O(N²)) with an optional bounded sliding window.
• Removed MPIExecutor/DaskExecutor and the [hpc] extra. They never worked (per-cycle work closes over live, unpicklable engine state) and were wired to no backend.
• tau2 on the winning device. The commit segment runs on the device of the softmax-selected sampler.

Path refinement (review fixes)

• Snap refined NN path nodes to the nearest physical trajectory frame (project_to_real); fix the converged flag so a walker-generation failure isn't reported as convergence; use a true discrete Fréchet distance for path convergence (was mislabeled Hausdorff); renumber the example pipelines.

Repo hygiene

• Ignore Manuscripts/ and .DS_Store; remove the superseded amberengine.py and unused standalone we/ scratch scripts.

Scaling model (reviewer note)

Path discovery is single-GPU, saturated by concurrent walkers. Multi-GPU/CPU belongs to the downstream WE via the device pool. Multi-node is Slurm/PBS array jobs over independent pathways/replicates (how the path-resolved-kinetics workflow already runs channels). There is deliberately no in-process multi-node executor; a work-queue manager for a single tightly-coupled run stays on the roadmap.

Testing

• pytest -q tests --ignore tests/hpc → 107 passed (torch installed so the SPIB lane runs).
• python tests/hpc/hpc_selfcheck.py → 7/7, including the new executable_resolution and concurrent_openmm checks.
• OpenMM concurrent pool verified on the CPU platform: a 4-worker run reproduces the serial result exactly with a deterministic integrator; WE walker spread verified across a mock device pool.
• Real multi-GPU spread and OpenMM single-GPU saturation are to be validated on the cluster via the tests/hpc/ Slurm/PBS job scripts and DEBUGGING.md.

Not included / follow-ups

• Real-GPU throughput numbers (run on the cluster).
• The OpenMM example still ships platform: CPU; switch to CUDA + workers_per_device: auto for the saturation test.
• No version bump (changes recorded under CHANGELOG.md [Unreleased]).

🤖 Generated with Claude Code