[CI] Make benchmark baselines fork-safe and bounded

[jhinpan]

Summary

Make benchmark baseline selection deterministic for fork pull requests and prevent one unusable baseline from expanding into eight full benchmark runs.

Fixes #856.

Motivation

The current workflow checks out the PR head repository and later fetches origin/main. For fork PRs, origin is the contributor fork rather than ROCm/FlyDSL.

In run 29366839827, the fork main was 11 commits behind the PR base. The same AITER API mismatch caused all eight fallback commits to fail, but every candidate still performed an editable install and almost the entire benchmark suite:

Runner	Baseline time	Result
MI355	76m06s	8 failures; no comparison
MI325	57m23s	8 failures; no comparison

The workflow remained green after skipping both main and tag comparisons. Superseded GPU jobs could also continue because their job-level conditions used always() despite cancel-in-progress: true.

Changes

• Resolve the canonical baseline repository from pull_request.base.repo.full_name.
• Fetch the exact pull_request.base.sha for PRs and github.event.before for main pushes; manual dispatch retains canonical main behavior.
• Run the main baseline exactly once instead of walking main through main~7.
• Fetch release tags from the base repository rather than the head fork.
• Apply the same exact-base selection to the label-triggered multi-GPU baseline.
• Reduce the combined baseline timeout from 180 to 45 minutes.
• Emit GitHub warning annotations when optional baseline comparisons cannot run.
• Replace job-level always() with !cancelled() on the real single- and multi-GPU jobs. Cleanup steps retain always().
• Temporarily quarantine the unhealthy linux-flydsl-navi-2 GPU 1 by running that job on proven healthy GPU 3; tracked for infrastructure repair in [CI] linux-flydsl-navi-2 GPU 1 consistently hangs the test suite #858.
• Cap the single-GPU test step at 60 minutes so a bad device cannot occupy a runner for the default six-hour limit.
• Preserve the comparison label main for compatibility with the existing dashboard parser, while logging the exact repository and commit.

Baseline selection after this change:

Event	Repository	Ref
Pull request	PR base repository	Exact pull_request.base.sha
Push to main	github.repository	Exact pre-push github.event.before
Manual dispatch	github.repository	main

Expected impact

For the observed failure mode, removing seven redundant retries should save roughly 65 minutes on MI355 and 50 minutes on MI325. It also makes the comparison target correct for every fork PR and allows concurrency cancellation to release GPU runners.

This intentionally remains a surgical workflow fix. AITER pinning, partial baseline CSV recovery, reusable main/release artifacts, test/performance workflow separation, and MLIR cache redesign are documented as follow-ups in #856.

Testing

• git diff --check
• Parsed .github/workflows/flydsl.yaml with PyYAML
• Ran bash -n over the modified embedded shell blocks
• Ran actionlint v1.7.7 (ignoring only the repository's known self-hosted runner labels)
• Verified GitHub fetches both current and historical FlyDSL commits by exact SHA with --depth=1
• Confirmed the eight-commit rev-list fallback and git fetch origin main are absent
• Correlated 26 recent Navi jobs: GPU 1 hung 3/3 times; GPU 2 passed 6/6 and GPU 3 had 16 successful runs
• Fork PR CI: confirm logs show ROCm/FlyDSL@<base SHA> and only one main attempt

No local GPU suite was run because the change is limited to GitHub Actions orchestration; this PR's fork CI is the functional integration test.

Dependencies

• No new third-party dependencies

Breaking Changes

None.

Related work

• Report benchmark ratios against main #526 introduced the baseline comparison and eight-commit fallback.
• [Refactor] Harden and speed up the JIT cache-key computation #597 added the latest-tag comparison while retaining that fallback.
• [Fix]: mla_decode test aborts whole collection on aiter API drift #840 and [CI] run_benchmark: treat runtime pytest.skip as skipped, not failed #842 fixed AITER-drift skip handling for current source but not stale fork baseline selection.
• [CI] skip build/test for docs-only changes #852 is an orthogonal open docs-only CI optimization. Its lightweight required-status shim intentionally retains always(); this PR only changes real GPU jobs and is compatible with it.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[jhinpan]

CI integration update from run 29380724467:

• MI325 completed successfully.
• The workflow fetched ROCm/FlyDSL@fa4f8ae2d5b0a08ca2b428247fd0ff5a7863558d, exactly matching the PR base.
• The log contains one actual Trying exact benchmark baseline ... entry and no main~N fallback attempts.
• The exact-base suite completed with 45 success / 1 skip / 0 failures, wrote 75 CSV rows, and current vs main executed.
• The baseline step took 15m49s versus 57m23s in the affected [layernorm] Add backward pass for training (PR 3/3, #769) #801 run (about 72% lower).
• Tags now come from upstream and resolve v0.2.4. That historical package still fails its known MLA/AITER API-drift case; the workflow reports the unavailable optional tag comparison as a warning.

MI355 and Navi are still in progress. The core fork/base selection and bounded-execution acceptance criteria are validated.

[jhinpan]

Navi follow-up validation on head c1a35c29 passed in run 29387022374:

• The job environment contained CI_HIP_VISIBLE_DEVICES=3, and the container received HIP_VISIBLE_DEVICES=3.
• Run tests completed in 1m31s: 927 passed, 3370 skipped, 3 warnings.
• The two prior GPU 1 attempts had hung at the same test boundary for 1h29m and 30m before cancellation.
• Current benchmarks completed in 20s.
• Baselines completed in 1m47s with one exact ROCm/FlyDSL@fa4f8ae2... attempt and zero historical fallbacks.
• Both current vs main and current vs v0.2.4 comparisons executed successfully.

The unhealthy GPU and removal criteria for this temporary quarantine are tracked in #858.