Skip to content

Add accuracy-routing gate for QNN EP coverage runs - #684

Draft
yuhuchua-qti wants to merge 4 commits into
dev/yuhuchua/migrate-clip-utfrom
dev/yuhuchua/accuracy-routing-gate
Draft

yuhuchua-qti wants to merge 4 commits into
dev/yuhuchua/migrate-clip-utfrom
dev/yuhuchua/accuracy-routing-gate

Conversation

@yuhuchua-qti

@yuhuchua-qti yuhuchua-qti commented Jul 30, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Replaces the unconditional QnnUnit_Accuracy_* full run in the coverage script's accuracy phase with a gate that skips accuracy cases whose paired snapshot already passed at a matching golden version, and runs the rest.

Skip a case only when both hold:

  • its paired snapshot passed (graph structure unchanged), AND
  • the golden manifest's QAIRT backend and ORT runtime versions both equal the current ones (either up-level can shift numerics even when the emitted structure is byte-identical).

Any other situation runs accuracy: snapshot drift (failed), golden absent / backend unavailable (skipped, no PASS proof), no paired snapshot (unmapped), or version mismatch / no manifest. Reasons are collected as a union with no short-circuit, so drift is always surfaced in the summary even when a version mismatch already forced a full run.

What's here

  • qcom/scripts/linux/accuracy_gate.py — pure decision functions + a thin CLI.
  • qcom/scripts/linux/generate_coverage.sh — wires the gate between the snapshot and accuracy phases; prints a developer-facing summary. Any gate failure (missing snapshot JSON, list failure, gate error, empty filter) falls back to a full run, so coverage is never silently dropped.
  • qcom/scripts/linux/tests/{conftest.py,test_accuracy_gate.py} — fixture unit tests (synthetic gtest JSON + accuracy lists; no binary, no build).

Stacked on #<PR1: dev/yuhuchua/migrate-clip-ut>.

Fix included

The snapshot→accuracy suite-name pairing regex required an underscore on both sides of the tier token (Snapshot/SessionSnapshot), so bare (no-variant) op-first suites like QnnUnit_Clip_SnapshotTest never matched and fell through as unmapped — forcing those accuracy cases to always run instead of being correctly paired against the snapshot result. Fixed derive_accuracy_suite's regex to not require a trailing underscore, with a regression test covering both the bare and variant suite forms. Confirmed via a real generate_coverage.sh run: unmapped went from 5 to 0 for Clip.

Test plan

  • pytest qcom/scripts/linux/tests — 30 passed
  • python qcom/build_and_test.py lint — clean
  • End-to-end CLI against the local golden cache (manifest with both qairt+ort) + real 14-case accuracy list:
    • both versions match -> run 0, skip 14
    • ORT mismatch -> full run (14)
    • QAIRT mismatch -> full run (14)
  • Full coverage_linux_x86_64 build + run on this branch (rebased directly onto PR1's latest tip): component 2504 passed / 12 skipped (env-limited), accuracy 16 passed / 0 failed, gate summary shows unmapped=0 (no manifest locally -> full run, as expected: absent=16 version-mismatch=16)

@yuhuchua-qti
yuhuchua-qti marked this pull request as draft July 30, 2026 09:41
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch 3 times, most recently from 1efe46d to fd14652 Compare August 24, 2026 10:33
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch from 0407fe9 to 7717d38 Compare August 31, 2026 07:30
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch 2 times, most recently from ab7c34a to 8db7faf Compare September 8, 2026 04:01
…p op

Flatten onnxruntime/test/providers/qnn/ into tier-first siblings:
component/, snapshot/, session_snapshot/, accuracy/, integration/,
with shared infra in infra/.

- Migrate Clip op to spec-shared tiers: one op spec set drives
  component dtype-dispatch tests, snapshot + session_snapshot
  graph-JSON goldens, and accuracy inference checks.
- Add OpBuilderTestContext (stub-backed QnnModelWrapper factory shared
  by component + snapshot) and a shared CompareOrWriteGolden helper
  used by both AssertSnapshotJson and AssertSessionSnapshotJson.
- Drop all QnnCpu-backed test infra (QnnRealCpuBackendContext/Manager,
  CPU branches in Clip tests) since libQnnCpu.so is no longer shipped
  with the wheel; reback Clip_4D_f32_DefaultMinMax / Clip_5D_f32 to
  HTP (HtpOpDefSupplement confirms RELUMINMAX supports FP32 rank 5).
  Keep the SnapshotBackend enum (now HTP-only) for interface stability.
- Close the folded-constant coverage gap: add a spec + snapshot +
  accuracy suite that replays the qdq_constant_folding end-state
  (AddTensorWrapper + MarkTensorAsFoldedConstant) gcov showed
  Clip_U8/U16_FloatData_QDQConstMinMax were the only cases hitting
  clip_op_builder.cc's folded-constant fallback, and no spec covered
  it directly.
- Rename QNN UT suites to op-first
  (QnnUnit_<Op>_<Tier>[_<Variant>]Test) so run_snapshot_accuracy.sh
  --filter and generate_coverage.sh can recover the op segment; the
  prior naming glued the variant onto the op with no delimiter, so
  --filter=Clip matched 0 tests in CI.
- Goldens are not committed: the harness reads them from
  $QNN_UT_SNAPSHOT_GOLDEN_DIR. Unset/absent -> snapshot tests skip
  (GOLDEN_ABSENT) and accuracy always runs, so golden staleness can
  never redden the build.
- generate_coverage.sh is now the single CI coverage orchestrator,
  running three phases (component -> snapshot -> accuracy) into one
  lcov capture, with component + accuracy gating and snapshot
  non-gating (drift is a routing signal, not a failure).
  run_snapshot_accuracy.sh is the local-only two-pass runner; its
  --update-goldens flag is renamed to --generate-goldens to match
  what it actually does (writes from scratch, not incrementally),
  and its embedded-python test-name extraction is factored out into
  extract_snapshot_groups.py.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch from 8db7faf to 096e681 Compare September 8, 2026 04:54
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/accuracy-routing-gate branch from 0a2b8a3 to b04074e Compare September 8, 2026 11:02
yuhuchua-qti and others added 2 commits September 9, 2026 09:51
Skip accuracy cases whose paired snapshot passed at a matching golden
version; run the rest (drift, golden absent, unmapped, or version mismatch).
A version match requires BOTH the QAIRT backend and ORT runtime versions in
the golden manifest to equal the current ones -- either up-level can shift
numerics even when the emitted graph structure is unchanged. This replaces
the unconditional full QnnUnit_Accuracy_* run in the coverage script's
accuracy phase.

accuracy_gate.py is pure functions + a thin CLI, wired into
generate_coverage.sh between the snapshot and accuracy phases. Any gate
failure falls back to a full run so coverage is never silently dropped.
Unit-tested with synthetic fixtures: pytest qcom/scripts/linux/tests

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_SNAPSHOT_TIER_RE required an underscore on both sides of the tier
token, so bare (no-variant) op-first suites like QnnUnit_Clip_SnapshotTest
never matched and fell through as unmapped, forcing their accuracy
cases to always run instead of being correctly paired against the
snapshot result. Confirmed via a live generate_coverage.sh run showing
unmapped=5 matching exactly the bare Clip snapshot suite's case count.
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/accuracy-routing-gate branch from 0ab37c3 to edf9c0f Compare September 9, 2026 02:26
detect_ort_version's NOTE referenced a nonexistent qnn_ut_version.sh;
the real shared getter is PR2/PR3's resolve_tool_versions.sh. Also add
the same NOTE to detect_qairt_version, which was missing one entirely.
@yuhuchua-qti
yuhuchua-qti force-pushed the dev/yuhuchua/migrate-clip-ut branch 3 times, most recently from d937314 to 84b714b Compare September 16, 2026 08:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant