ci: smoke-test Lemonade candidate prereleases so breaking changes surface before they ship

[kovtcharov]

Lemonade v2026.39.1 started publishing prerelease candidate builds — GitHub prereleases tagged candidate-v<version>, plus a candidate Docker tag. GAIA can now find out that a Lemonade release breaks it before that release is stable.

Why this matters here. The v2026.39.1 bump (#4169) is the argument. That release changed the version format from semver to CalVer, which quietly disabled GAIA's version gate for development builds — five separate parsers returned "can't tell" instead of comparing. It was caught by reading the release notes by hand. A candidate existed a week earlier (candidate-v2026.39.1, published 16 Sep) and would have surfaced it on a schedule instead.

The existing lemonade-version-bump.yml deliberately tracks releases/latest so it never chases a prerelease, and that should stay as it is — this is a separate, non-blocking smoke job, not a change to what GAIA pins.

Worth scoping:

• A scheduled job that installs the current candidate and runs the version-gate and health-schema contract tests against it, reporting rather than gating.
• Decide the failure channel: a rolling issue like the bump workflow's, so a breaking candidate is visible without turning main red.

Proposed from the Lemonade v2026.39.1 release scan; maintainer triage decides scope. cc @kovtcharov-amd

[github-actions]

Most of the pieces already exist — the gap is one missing knob and a tier split.

The break you want caught was pure string parsing: a dev-build version string GAIA's parsers choked on. Catching that needs no Lemonade install at all — just the candidate's version string fed through the same regression test #4169 adds. Seconds on a free runner, so it can run daily rather than weekly.

Only the health-schema half needs a real server, and that's where the cost lands: the existing Lemonade smoke job runs on the self-hosted Ryzen pool, and the install helper is hard-pinned to the version GAIA ships with no way to point it at a candidate. The candidate Docker tag you mentioned may let that tier run on a free runner instead — worth settling before wiring anything to the self-hosted pool, since a weekly candidate job would contend with PR CI for the same machines.

For the failure channel, the bump workflow already has exactly the rolling-issue behaviour you described. Copy it rather than inventing a second one.

Cadence: the candidate landed ~a week before the stable release, so the same weekly cron shifted a day earlier buys the lead time.

🔍 Technical details

Tier 1 — parser gate, ubuntu-latest, no install. The #4169 failure mode is int("0~12") raising on a 2026.39.0~12.abc1234 dev string, so all five parsers return None and every caller reads that as "can't tell". tests/unit/test_lemonade_calver.py (new in #4169) is already the assertion body — it needs the version string as a parameter instead of a hardcoded fixture, then the job is:

tag=$(gh api repos/lemonade-sdk/lemonade/releases \
  --jq '[.[] | select(.prerelease and (.tag_name | startswith("candidate-v")))][0].tag_name')

…and feed the stripped version through the parsers. That alone would have caught this one, a week early, for free.

Tier 2 — health schema, needs a running server. Two blockers:

• .github/actions/install-lemonade/action.yml:30-35 reads LEMONADE_VERSION straight out of src/gaia/version.py and exact-match-verifies it. It needs an optional version input (defaulting to today's behaviour) before it can install a candidate.
• The MSI URL derives from a v<version> tag (lemonade_installer.py:GITHUB_RELEASE_BASE); candidate-v<version> is a different tag shape.

.github/workflows/test_lemonade_server.yml:54 pins [self-hosted, Windows, devlab-dispatch, strix-halo]. If the candidate Docker tag serves /api/v1/health with the same schema, this tier is a container step on a hosted runner and skips both blockers — worth confirming first.

Failure channel. .github/workflows/lemonade-version-bump.yml:362-368: search open issues by title prefix, comment if one exists, else create assigned to kovtcharov-amd with bug + devops. Reusable as-is with a different prefix.

Scoping note. Tier 1 catches contract drift in version strings; it will not catch a candidate that changes the health payload or an endpoint's response shape. If tier 2 turns out to need the self-hosted pool, tier 1 is still worth shipping standalone — it is the cheap 80% of this issue.