[WIP] feat(minimaxm3): add B200 disaggregated configs by xinli-sw · Pull Request #2230 · SemiAnalysisAI/InferenceX

xinli-sw · 2026-07-15T23:00:31Z

No description provided.

claude · 2026-07-15T23:19:32Z

+slurm:
+  account: coreai_comparch_inferencex
+  partition: batch
+  time_limit: "01:00:00"


🟡 All four new B200 minimax-m3 recipes set slurm.time_limit: "01:00:00", well below the launcher's 4h default and the 3-8h used elsewhere in the tree, and none define a health_check block, so the launcher's sed that bumps max_attempts 360→720 (meant to absorb cold-cache loads off shared FS) is a no-op here. The model loads from shared scratch FS, so a slow cold load could eat the 1h walltime and kill the job in TIMEOUT with no results — worth double-checking against the sibling B300 recipes (no time_limit override, health_check present) before merge.

Extended reasoning...

The bug: all four new recipes (2p2d-c128, 3p2d-c256, 3p2d-c512, 4p2d-c1024) set slurm.time_limit: "01:00:00". This overrides the generated srtslurm.yaml's default_time_limit: "4:00:00" (set in runners/launch_b200-dgxc.sh) downward to a quarter of that. Across the whole srt-slurm-recipes tree, the only other explicit time_limit values used are 3h and 8h — 1h is an outlier by a wide margin, and it's the only class of minimax-m3 recipe that sets it this low.\n\nWhy the launcher's usual safety net doesn't apply: runners/launch_b200-dgxc.sh has a sed step specifically to absorb slow cold-cache model loads off shared FS:\n\nThis only works if the recipe already has a health_check.max_attempts line to rewrite (going from 360×10s=3600s to 720×10s=7200s). None of the four new recipes define a health_check section at all, so this sed matches nothing and is a silent no-op. Compare to the sibling B300-fp4 minimax-m3 recipes (same checkpoint, same container image), which set no slurm.time_limit override (inheriting the 4h default) and still carry health_check.max_attempts: 360, letting the sed extend their health-check budget to 2h. The new B200 recipes get neither protection.\n\nWhy this matters here specifically: MODEL_PATH for minimaxm3+fp4 resolves to /scratch/fsw/models/MiniMax-M3-NVFP4, shared scratch FS — the same class of cold-cache-slow-load risk that motivated both the sed and the 8h time_limit/1440 max_attempts used by the sibling deepseek-v4 disagg recipes (comment there: "to absorb cold-cache model loads"). If a cold cache causes a slow multi-worker MoE weight load (the 4p2d/c1024 config spans 4 nodes and has the most startup coordination), the health-check window and remaining walltime could both be exhausted before srtctl ever runs the benchmark sweep, and SLURM kills the job in TIMEOUT — a multi-node B200 allocation consumed with zero results collected, which is a much worse failure mode than a clean error.\n\nProof sketch: launch job → sed looks for ^ max_attempts: [0-9]* in the recipe, finds no match (none of the 4 files have a health_check key), no-op → srtctl applies the recipe with its default health-check window (360×10s=3600s=1h) → model load from /scratch/fsw/models/MiniMax-M3-NVFP4 under cold cache takes long enough to approach that 1h health-check budget → SLURM's own 1h wall-clock limit (time_limit: "01:00:00") expires around the same time the load might still be settling → job terminates in TIMEOUT before the benchmark concurrency point runs → no results_concurrency_*.json produced, wasted allocation.\n\nCounterpoint worth weighing: a reviewer flagged that each of these recipes runs a single concurrency point (skip_initial_test: true, one value in concurrencies) rather than a multi-point sweep like the B300 sibling that relies on the 4h default, and that MiniMax-M3-NVFP4 is a much smaller 4-bit checkpoint than the ~680GB DSR1-FP8 case the sed comment references — so 1h may be a deliberate, proportionately-sized choice rather than an oversight, and the perf-changelog even labels these "validated" configurations. That's plausible, but it doesn't fully explain why these are the only minimax-m3 recipes with no health_check block at all (even the B300 siblings running fewer/no sweep points keep one), and a single occurrence of cold cache during CI reruns would still zero out the run. Given the WIP status and that this is easy to align with the siblings, it's worth a second look before merge.\n\nFix: either drop the slurm.time_limit override (inherit the 4h default like the B300 siblings) or add a health_check block (e.g. max_attempts: 360) so the launcher's sed can extend it, matching the pattern used everywhere else in the tree.