Skip to content

Add golden store publish scripts + nightly CI wiring - #703

Draft
yuhuchua-qti wants to merge 15 commits into
dev/yuhuchua/migrate-clip-utfrom
dev/yuhuchua/qnn-ut-golden-publish
Draft

Add golden store publish scripts + nightly CI wiring#703
yuhuchua-qti wants to merge 15 commits into
dev/yuhuchua/migrate-clip-utfrom
dev/yuhuchua/qnn-ut-golden-publish

Conversation

@yuhuchua-qti

@yuhuchua-qti yuhuchua-qti commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Stacked on PR1 (migrate-clip-ut, #399). Delivers the producer side of the QNN EP unit-test golden pipeline: scripts that regenerate goldens, keep only accuracy-passing op groups, version-stamp them, and publish to Artifactory — plus the nightly CI job that runs it and the retention policy that prunes old archives.

Pure additive. No production op-builder / snapshot / accuracy source is touched. The consumer (per-PR accuracy-routing gate) is a separate later PR.

What's in it

Version resolver — single source of truth

  • qcom/scripts/linux/resolve_tool_versions.sh — source-able and executable. Resolves QAIRT version (from sdk.yaml across QAIRT_SDK_ROOT/QNN_SDK_ROOT/SNPE_ROOT) and ORT runtime version (from $ORT_PREBUILT_ROOT/VERSION_NUMBER, then /VERSION). Deliberately does not read repo-root VERSION_NUMBER (that's the plugin/wheel version, not the ORT runtime). Undeterminable → exit 3 (graceful), so callers treat it as a safe signal. Writer and future gate stay consistent by sharing this code, never a process.

Publisher — thin orchestrator

  • qcom/scripts/linux/publish_goldens.sh — regen goldens via run_snapshot_accuracy.sh --generate-goldens → read the accuracy report → package only groups whose every QnnUnit_Accuracy_<Group>Test case PASSED → write a version-stamped manifest.json → upload a write-once archive/<date>-<sha>/ and the mutable latest/ pointer. Default is dry-run; --publish is required to actually upload (it overwrites the shared latest/ pointer — never implicit).
  • qcom/scripts/linux/filter_accuracy_pass_groups.py — extracted pass-group selection (a group passes iff all its accuracy cases COMPLETED with no failures).

Nightly CI wiring

  • .github/workflows/qualcomm-internal-golden-publish.yml — reusable workflow (workflow_call + workflow_dispatch). Builds the coverage-instrumented binary, then runs publish_goldens.sh. dry_run defaults true so manual dispatch never clobbers the shared store; the scheduled producer opts into the real upload explicitly.
  • .github/workflows/qualcomm-internal-release-nightly.yml — adds a golden-publish job (9am UTC cron) passing dry_run: false.
  • qcom/build_and_test.py — new coverage_build_linux_x86_64 task (coverage build only, no HTML report — snapshot/accuracy tiers only compile under --enable-coverage).

Golden store layout + retention

  • Store lives under qnn-ep-test-store/ut-snapshot/{archive,latest}/. The qnn-ep-test-store/ parent is shared with future siblings (e.g. modelzoo goldens).
  • qcom/scripts/artifactory/aql/expired_golden_artifacts.json — AQL that matches qnn-ep-test-store/*/archive/* older than 10d. Retention keys off the archive/ segment only, so latest/ pointers (and any permanent data outside archive/) are never matched — retention can never leave a store without a current golden.
  • .github/workflows/qualcomm-internal-clean-artifacts.yml — new "Delete old golden archives" step (runs on the existing daily cleanup cron, 1h before the publish cron).

Test plan

  • pytest qcom/scripts/linux/tests — 17 tests pass (resolver priority order, sdk.yaml variants, undeterminable → exit 3).
  • publish_goldens.sh dry-run end-to-end against a real coverage build-dir: regen → parse accuracy report → stage passing groups → build goldens.zip + manifest.json → prints the two jf rt upload commands without executing.
  • manifest.json qairt_version/ort_version match resolve_tool_versions.sh both; git_sha matches git rev-parse --short=10 HEAD.
  • Empty-pass guard: zero passing groups → die, no zip/upload.
  • Live Artifactory upload verified via nightly (post-merge).

🤖 Generated with Claude Code

@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch 2 times, most recently from e434afc to e45fe62 Compare August 18, 2026 05:55
yuhuchua-qti and others added 4 commits August 19, 2026 16:46
@yuhuchua-qti @claude
Restructure QNN EP tests into tier-first layout; migrate Clip
261d495 
Flatten onnxruntime/test/providers/qnn/ into tier-first siblings:
component/, snapshot/, session_snapshot/, accuracy/, integration/,
with shared infra in test_infra/.

- Migrate Clip op to spec-shared tiers (one op spec set drives
  component dtype-dispatch, snapshot + session_snapshot graph goldens,
  and accuracy inference checks)
- Add OpBuilderTestContext: stub-backed QnnModelWrapper factory shared
  by the component + snapshot tiers
- Goldens are NOT committed: the harness reads them from
  $QNN_UT_SNAPSHOT_GOLDEN_DIR (Artifactory-fetched). Unset/absent ->
  snapshot skips (GOLDEN_ABSENT) and accuracy always runs -> safe
  baseline that can never red the build on golden freshness
- Golden path logic anchored on /providers/qnn/ with per-tier prefix
- CMake tier globs; generate_coverage.sh is the single CI coverage
  orchestrator running three phases (component -> snapshot -> accuracy)
  into one lcov capture, with component + accuracy gating and snapshot
  non-gating (drift is a routing signal). run_snapshot_accuracy.sh is
  the local-only two-pass runner

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti @claude
Rename --update-goldens to --generate-goldens in run_snapshot_accurac…
d55c596 
…y.sh

The flag writes golden files from scratch regardless of whether they
previously exist. "generate" reflects the actual semantics (create from
nothing) better than "update" (which implies modifying existing state).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti @claude
[QNN EP] Complete Clip migration: fill Path A gap, HTP reback, drop P…
f9a341e 
…lain naming

gcov shows two integration tests, Clip_U8/U16_FloatData_QDQConstMinMax,
are the only cases (old or new) that hit clip_op_builder.cc:45-52's
folded-constant fallback. The existing quantized-min/max test cases
look similar but don't hit it — their min/max keep a quant_param on
the initializer and go through the QUANT switch instead. Add a spec +
snapshot + accuracy suite that replays the qdq_constant_folding
end-state (via AddTensorWrapper + MarkTensorAsFoldedConstant) to
cover the gap.

Also reback Clip_{4D_f32_DefaultMinMax,5D_f32} to HTP (QnnCpu dropped
from the wheel; HtpOpDefSupplement confirms RELUMINMAX supports FP32
rank 5, no code change needed — both PASS), and drop the `Plain`
suffix from the default spec/suites/helpers since it read as an
unnecessary negative marker.

51 Clip unit tests: 35 PASS, 16 GOLDEN_ABSENT skips (snapshot tier,
no golden store configured), 0 failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yuhuchua-qti
Rename QNN UT suites to op-first to fix per-op filtering
7c7bbb7 
Tier-first names (QnnUnit_Snapshot_ClipQDQFloatTest) glue the variant
onto the op with no delimiter, so tooling that recovers the op segment
from the suite name (run_snapshot_accuracy.sh --filter, generate_coverage.sh)
cannot split "ClipQDQFloat" back into op=Clip. This caused
run_snapshot_accuracy.sh --filter=Clip to match 0 tests in CI.

Switch to QnnUnit_<Op>_<Tier>[_<Variant>]Test: the op is recovered as
the segment(s) between "QnnUnit_" and the first tier token (Component/
Snapshot/SessionSnapshot/Accuracy), so op names containing underscores
(e.g. Gelu_Fusion) stay unambiguous and per-op filters are a clean
prefix (QnnUnit_Clip_*).

Verified via a fresh coverage build: all four Clip tiers run under
their new names (component/accuracy PASS, snapshot SKIPPED pending an
external golden store as before), and run_snapshot_accuracy.sh
--filter=Clip now matches all 16 snapshot cases across 5 suites
(previously 0).
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch from e65be32 to 7c7bbb7 Compare August 19, 2026 09:13
yuhuchua-qti and others added 11 commits August 20, 2026 11:12
@yuhuchua-qti @claude
Add resolve_tool_versions.sh + publish_goldens.sh for golden store pu…
a8f60e6 
…blishing

PR2 of the QNN EP UT migration stack. Adds:
- resolve_tool_versions.sh: single source of truth for QAIRT + ORT runtime
  version strings (file-only detection, no env override). Dual-mode
  (sourceable + executable). Shared by the manifest writer and the future
  accuracy gate reader.
- publish_goldens.sh: thin orchestrator that regenerates goldens, selects
  only accuracy-PASSING op groups, packages them with a version-stamped
  manifest.json, and uploads to Artifactory (default dry-run).
- pytest suite (17 tests) covering version resolution precedence, sdk.yaml
  parsing variants, undeterminable exit codes, and edge cases.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti @claude
Update publish_goldens.sh to use renamed --generate-goldens flag
6250bb3 
Follows the PR1 rename of --update-goldens to --generate-goldens in
run_snapshot_accuracy.sh.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti @claude
Extract inline accuracy-group filter into standalone Python script
5243a34 
Move the pass-group selection logic from an inline bash heredoc in
publish_goldens.sh into filter_accuracy_pass_groups.py. This makes
the classification logic independently testable and reusable by the
future accuracy gate (PR4).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti @claude
Wire nightly golden-publish CI job
3ed983d 
Adds the CI half of PR2 (merged PR2+PR3): a scheduled job that runs
publish_goldens.sh to refresh the Artifactory golden store nightly.

- qualcomm-internal-golden-publish.yml: reusable workflow (workflow_call +
  workflow_dispatch) that builds a coverage-instrumented binary, then runs
  publish_goldens.sh. Inputs: filter, repo_subpath, skip_regen, dry_run.
  dry_run defaults true so a manual dispatch never clobbers the shared store
  by accident; the scheduled producer opts into the real upload explicitly.
- qualcomm-internal-release-nightly.yml: adds a parallel golden-publish job
  on the 9am cron, passing dry_run=false.
- build_and_test.py: adds coverage_build_linux_x86_64 (coverage build only,
  no HTML report). publish_goldens.sh does its own golden regen + accuracy
  verify, so the job needs the instrumented binary but not the report,
  avoiding a wasted second test run.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti @claude
Add golden archive retention and finalize store layout
795f849 
Golden stores live under qnn-ep-test-store/, grouped by subject:
ut-snapshot/ (this suite) and, later, modelzoo/. Retention keys off the
path segment, not the top folder:

  qnn-ep-test-store/<subject>/latest/   -> never pruned (gates read it)
  qnn-ep-test-store/<subject>/archive/  -> pruned after 10d

- expired_golden_artifacts.json: AQL matching qnn-ep-test-store/*/archive/*
  created before 10d. latest/ and any permanent data (no archive/ segment)
  are never matched, so retention can't strand a store without a golden.
- qualcomm-internal-clean-artifacts.yml: new "Delete old golden archives"
  step on the existing nightly cleanup cron.
- publish_goldens.sh / golden-publish.yml: repo_subpath default moves from
  qnn-ut-goldens to qnn-ep-test-store/ut-snapshot.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti
Align filter_accuracy_pass_groups.py regex with op-first suite naming
6a11e13 
PR1 renamed accuracy suites from QnnUnit_Accuracy_<Group>Test to
QnnUnit_<Group>_Accuracy[_<Variant>]Test. The old regex no longer
matched any suite, so publish_goldens.sh would always find zero
passing groups. Update the regex and its two doc comments to match
the new naming; verified against a synthetic gtest JSON report.
@yuhuchua-qti
Resolve QAIRT/ORT versions from build artifacts instead of dead env vars
08af944 
No CI workflow ever exports QAIRT_SDK_ROOT/QNN_SDK_ROOT/SNPE_ROOT/
ORT_PREBUILT_ROOT, so the old env-var-based resolver always fails with
"version undeterminable" in real CI. CMakeCache.txt already records the
actual onnxruntime_QNN_HOME/onnxruntime_ORT_HOME used by the build (or
_deps/ort_core-src when ORT is FetchContent-ed), so read from there instead.
@yuhuchua-qti @claude
TEMP: push-trigger dry-run for pre-merge golden inspection
29406b5 
Temporary wrapper: on push to the PR2 branch, run golden-publish in
dry-run (nothing touches Artifactory) and upload the packaged zip as a
downloadable artifact. Passes dry_run + filter explicitly because a push
event has no inputs context. DELETE THIS COMMIT BEFORE MERGE.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti @claude
TEMP: add permissions to wrapper so reusable call starts
8096db1 
The wrapper called golden-publish.yml (which requests checks: write) with
default token permissions, causing a startup_failure. Match the reusable
workflow's permissions. DELETE WITH THE OTHER TEMP COMMIT BEFORE MERGE.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti
Add --keep-zip support to publish_goldens.sh for CI artifact upload
3dad35a 
Lets a golden-publish run also surface the packaged goldens.zip as a
downloadable GitHub Actions artifact, gated behind keep_zip_artifact
(off by default) for manual inspection.
@yuhuchua-qti
TEMP: wire keep_zip_artifact into dry-run so goldens.zip is downloadable
a8010de 
Verify the new --keep-zip flag end-to-end on the next push before
merge.
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/qnn-ut-golden-publish branch from b0154fa to a8010de Compare August 20, 2026 05:11
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch 3 times, most recently from 0407fe9 to 7717d38 Compare August 31, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@qti-ashwshan qti-ashwshan Awaiting requested review from qti-ashwshan qti-ashwshan will be requested when the pull request is marked ready for review qti-ashwshan is a code owner
@qti-shubham qti-shubham Awaiting requested review from qti-shubham qti-shubham will be requested when the pull request is marked ready for review qti-shubham is a code owner
@minfhong-qti minfhong-qti Awaiting requested review from minfhong-qti minfhong-qti will be requested when the pull request is marked ready for review minfhong-qti is a code owner
@qti-yuduo qti-yuduo Awaiting requested review from qti-yuduo qti-yuduo will be requested when the pull request is marked ready for review qti-yuduo is a code owner
@tirupath-qti tirupath-qti Awaiting requested review from tirupath-qti tirupath-qti will be requested when the pull request is marked ready for review tirupath-qti is a code owner
@qti-chuteng qti-chuteng Awaiting requested review from qti-chuteng qti-chuteng will be requested when the pull request is marked ready for review qti-chuteng is a code owner
@qti-mbadnara qti-mbadnara Awaiting requested review from qti-mbadnara qti-mbadnara will be requested when the pull request is marked ready for review qti-mbadnara is a code owner
@qti-kromero qti-kromero Awaiting requested review from qti-kromero qti-kromero will be requested when the pull request is marked ready for review qti-kromero is a code owner
@huaychou huaychou Awaiting requested review from huaychou huaychou will be requested when the pull request is marked ready for review huaychou is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@yuhuchua-qti