[nv-alicheng]

What does this PR do?

Adds a cohort-driven validator for submission bundles: src/inference_endpoint/validation/ (fourteen modules), run as

uv run python -m inference_endpoint.validation <cohort-dir> <submission-root>

A cohort is a directory that declares what one benchmark round requires: a cohort.yaml manifest (hint, ruleset or cohort-bundled catalog), one YAML file per scope (system.yaml, point.yaml, pareto.yaml), and a valid_refs.yaml naming the artifact fields a row may address. Checks are declarative rows — scope, dotted reference into the submission's artifacts, operator (comparison, membership, regex, aggregate over lists/regions), severity, and the policy rule it enforces. Each row prints one finding per instance with the rule citation and the fix, and the run exits 0 (pass), 1 (a fail row failed) or 2 (refused before any check ran). warn rows are reported but do not fail the run.

What the module covers:

• Declared vocabulary — valid_refs.yaml gives every addressable field its expected type. A present value that contradicts its declaration, a name no declaration covers, and an artifact the upload walk would silently skip are all refused up front, so rows only judge facts that actually exist.
• Upload walk — results/<system>/<model>/r<N>/ with the scope files (point.yaml, system_desc.json, result_summary.json, accuracy_results.json, system_power.json); directories that do not match the point shape, symlinks, and entries resolving outside the submission are refused once, naming each.
• Regions and pareto arithmetic — region declarations, per-pareto folds, and derived facts (point cap, out-of-range points) computed a single time and shared by every row that names them.
• Catalog — model and dataset vocabularies, either from a packaged ruleset or from a catalog that ships inside the cohort.
• Honest reporting — each row reports whether it ran and at how many instances; a row whose when gated every instance off prints NEVER-RAN instead of looking like a pass, so a cohort cannot quietly stop checking things.

Docs: docs/validation/README.md is the usage contract (how to write a cohort, run it, and read findings); docs/validation/DESIGN.md holds the internals, invariants, and the engineering calls with their rationale. AGENTS.md maps the new package.

The change is additive: nothing outside the new package changes except the AGENTS.md component map and one .gitignore entry.

Type of change

• New feature

Related issues

None.

Testing

• Tests added/updated — nine unit test modules under tests/unit/validation/ (~4.4k lines) cover ref resolution and refusals, comparison and aggregate operands, region arithmetic, the upload walk, runner reporting, and the CLI, with stub cohort assets under tests/assets/cohorts/stub/.
• All tests pass locally — uv run pytest tests/unit/validation -q → 407 passed.
• Manual testing completed — end-to-end CLI runs against the stub cohort, against deliberately malformed uploads to exercise every refusal path, and against a real MLPerf submission bundle with the first real cohort (built on top of this branch): 11 points, one system, one model, 291 findings, 0 failing, exit 0.

Checklist

• Code follows project style — ruff and ruff-format clean.
• Pre-commit hooks pass — locally with SKIP=mypy: the mypy hook trips on three pre-existing macOS-typeshed errors unrelated to this branch.
• Documentation updated (if needed) — README, DESIGN, and the AGENTS.md map.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[nv-alicheng]

DERIVATION_NAMES is used: check.py:74 imports it and check.py:112 checks every
derived.<name> ref against it at cohort load, and
tests/unit/validation/test_submission.py asserts the registry matches it. Leaving the constant
where it is.

[nv-alicheng]

Kept as ValueError, with a comment. I ran the suggested KeyError recipe: __contains__ false +
__getitem__ raising KeyError + a get that raises ValueError makes the verdict
expected derived.x to exist, got absent (broken=False) and the test fails —
Verdict.evaluate maps ValueError/OverflowError to the fail-loud verdict this test pins, and
nothing calls Mapping.get on the way there. The stub is deliberately non-standard.

[nv-alicheng]

Thanks — both the CI break and the four code-quality notes are settled, plus five review-council
findings on the decoding commit.

CI test job (fixed, now green). The full suite imports every test file by module basename, so
tests/unit/validation/test_runner.py collided with the pre-existing
tests/unit/evaluation/swebench_service/test_runner.py (and test_cli.py with
tests/integration/commands/test_cli.py). CI died at collection with
import file mismatch: imported module 'test_runner' has this __file__ attribute …. The module's own
suite never caught it because it always collected one directory at a time; a whole-tree
--collect-only is now part of the local gate. Both files are renamed to
test_validation_runner.py / test_validation_cli.py: the tree collects clean (3272/3279 collected,
7 deselected, 0 errors), the module suite passes (439), and the test job is green (8m50s) with
every other job on this PR.

Code-quality notes, triaged:

• DERIVATION_NAMES reported unused — not accurate, no change: check.py imports it (line 74) and
  uses it at line 112 to refuse a derived.<name> ref at cohort load, and
  tests/unit/validation/test_submission.py asserts the derivation registry against it.
• _Fact.matches mixing explicit and implicit returns — fixed: a dict arm is spelled out beside
  the catch-all, so every declared type name has a branch.
• LoudDerived.__getitem__ raising ValueError — intentional, kept, with a comment. The bot
  re-flagged it with a KeyError recipe, so I ran that recipe: with __contains__ false and
  __getitem__ raising KeyError, the verdict becomes
  expected derived.x to exist, got absent (broken=False) and the test fails — Verdict.evaluate
  maps ValueError/OverflowError to the fail-loud verdict this test pins, and a KeyError never
  reaches it. The stub is deliberately non-standard; that is its job here.
• derived["nope"] statement has no effect — fixed: the lookup binds to _ inside pytest.raises.

Review-council findings on the decoding commit (all five addressed):

1. (high) Point.from_node discarded the decoded max_supported_concurrency, so a stringified
   maximum passed the parser and was refused later by the pareto shape check. The decoded maximum
   now rides on the carried descriptor (with a stringified c_min_override decoded alongside),
   while the artifact document stays as written so the ref table still reports the decoding.
2. (medium) _check_values matched declared names exactly while Ref.resolve reads a
   numeric-equivalent key, so a declared 90 skipped an artifact key "90.0" and a reachable value
   was never type-checked. Both paths now share _key_for.
3. (medium) The int decode judged integrality on a float pass, so "512.9999999999999999"
   rounded to a whole number and "9007199254740993.0" lost a digit. Integrality is now read from
   the spelling with Decimal, exactly.
4. (low) The TestResolve class statement had been dropped in an earlier edit, leaving its
   docstring as a bare string inside TestMapRefs; restored.
5. (low) DESIGN's contract-validation paragraph had lost its declaration-refusal clause; restored.

Verification: module suite 439 passed; whole tree collects with 0 errors; the cohort still reports
291 findings / 0 failing against the reference submission bundle, with the P90 row reading the
submission's "90.0" key at every measurement point; and a copy of that bundle with stringified
concurrency and max_supported_concurrency now runs clean, reporting both decodings
(coerced: … point.config.concurrency: '512' read as 512, coerced: … point.system_desc.max_supported_concurrency: '1024' read as 1024) where the stringified maximum
previously failed the shape check.