Skip to content

Reminder: standard.def doesn't pin transformers → training/eval env mismatch breaks final evaluation #65

Description

@CeciRyy

standard.def installs transformers unpinned, vllm_debug.def pins 4.57.3. The two images therefore end up on different major versions:

standard.sif:   transformers 5.14.1   (whatever pip resolves at build time)
vllm_debug.sif: transformers 4.57.3   (pinned)

Tokenizers saved under 5.x write extra_special_tokens as a list; 4.57.3 crashes loading it (AttributeError: 'list' object has no attribute 'keys'), so the eval-side vLLM server never starts, all 9 retries fail, no metrics.json — valid deliveries fall back to the base-model score. Also unfair wrt prompt Rule 6: agents complying with "must run in the current environment" still fail, since the scoring env differs from theirs.

Fix: install requirements-direct.txt in standard.def too, or at least pin transformers/tokenizers to the same versions as the eval image.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions