Skip to content

oif bench: one run per node (B-1a) - #67

Merged
manmohanbrahma merged 1 commit into
mainfrom
bench/b1a-lock
Nov 16, 2026
Merged

manmohanbrahma merged 1 commit into
mainfrom
bench/b1a-lock

Conversation

@manmohanbrahma

Copy link
Copy Markdown
Collaborator

In B-1 a second oif bench started while the first was between points and removed its server: 6 points lost.

  • A node-wide lock (/run/lock/oif-bench.lock, BENCH_LOCK for tests) refuses a second run: another oif bench is running on this node (PID N). Per node, not per GPU: ADR-009 is one GPU under load at a time, and every run's server listens on port 8000.
  • The lock file holds the running bench's PID; containers do not inherit the lock (ct closes fd 8), so it frees itself when the run ends.
  • The fixed server name oifb-srv stays: with the lock no second run can remove it, and a crashed run's server is cleaned up by the next run.
  • New tests/unit/test_bench_lock.sh (fails without the lock; checked on Linux).
  • Guide 18 troubleshooting row; TASKS B-1a [~] until the lab check (a second run on gpu-amd is refused).

In B-1 a second oif bench started while the first was between points and
removed its server (6 points lost). A node-wide lock now refuses a second run
and names the PID of the one running (ADR-009: one GPU under load at a time;
one fixed server port). Containers do not inherit the lock.
New tests/unit/test_bench_lock.sh (fails without the lock).
@manmohanbrahma
manmohanbrahma merged commit d5a83ab into main Nov 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant