standard.def installs transformers unpinned, vllm_debug.def pins 4.57.3. The two images therefore end up on different major versions:
standard.sif: transformers 5.14.1 (whatever pip resolves at build time)
vllm_debug.sif: transformers 4.57.3 (pinned)
Tokenizers saved under 5.x write extra_special_tokens as a list; 4.57.3 crashes loading it (AttributeError: 'list' object has no attribute 'keys'), so the eval-side vLLM server never starts, all 9 retries fail, no metrics.json — valid deliveries fall back to the base-model score. Also unfair wrt prompt Rule 6: agents complying with "must run in the current environment" still fail, since the scoring env differs from theirs.
Fix: install requirements-direct.txt in standard.def too, or at least pin transformers/tokenizers to the same versions as the eval image.
standard.definstalls transformers unpinned,vllm_debug.defpins 4.57.3. The two images therefore end up on different major versions:Tokenizers saved under 5.x write
extra_special_tokensas a list; 4.57.3 crashes loading it (AttributeError: 'list' object has no attribute 'keys'), so the eval-side vLLM server never starts, all 9 retries fail, nometrics.json— valid deliveries fall back to the base-model score. Also unfair wrt prompt Rule 6: agents complying with "must run in the current environment" still fail, since the scoring env differs from theirs.Fix: install
requirements-direct.txtinstandard.deftoo, or at least pin transformers/tokenizers to the same versions as the eval image.