Add accuracy-routing gate for QNN EP coverage runs - #684
Draft
yuhuchua-qti wants to merge 4 commits into
Draft
yuhuchua-qti wants to merge 4 commits into
yuhuchua-qti wants to merge 4 commits into
Conversation
yuhuchua-qti
requested review from
qti-ashwshan,
qti-chuteng,
qti-jkilpatrick,
qti-kromero,
qti-yuduo,
tirupath-qti and
yath1
as code owners
July 30, 2026 09:31
yuhuchua-qti
marked this pull request as draft
July 30, 2026 09:41
yuhuchua-qti
force-pushed
the
dev/yuhuchua/migrate-clip-ut
branch
3 times, most recently
from
August 24, 2026 10:33
1efe46d to
fd14652
Compare
yuhuchua-qti
force-pushed
the
dev/yuhuchua/migrate-clip-ut
branch
from
August 31, 2026 07:30
0407fe9 to
7717d38
Compare
yuhuchua-qti
force-pushed
the
dev/yuhuchua/migrate-clip-ut
branch
2 times, most recently
from
September 8, 2026 04:01
ab7c34a to
8db7faf
Compare
…p op Flatten onnxruntime/test/providers/qnn/ into tier-first siblings: component/, snapshot/, session_snapshot/, accuracy/, integration/, with shared infra in infra/. - Migrate Clip op to spec-shared tiers: one op spec set drives component dtype-dispatch tests, snapshot + session_snapshot graph-JSON goldens, and accuracy inference checks. - Add OpBuilderTestContext (stub-backed QnnModelWrapper factory shared by component + snapshot) and a shared CompareOrWriteGolden helper used by both AssertSnapshotJson and AssertSessionSnapshotJson. - Drop all QnnCpu-backed test infra (QnnRealCpuBackendContext/Manager, CPU branches in Clip tests) since libQnnCpu.so is no longer shipped with the wheel; reback Clip_4D_f32_DefaultMinMax / Clip_5D_f32 to HTP (HtpOpDefSupplement confirms RELUMINMAX supports FP32 rank 5). Keep the SnapshotBackend enum (now HTP-only) for interface stability. - Close the folded-constant coverage gap: add a spec + snapshot + accuracy suite that replays the qdq_constant_folding end-state (AddTensorWrapper + MarkTensorAsFoldedConstant) gcov showed Clip_U8/U16_FloatData_QDQConstMinMax were the only cases hitting clip_op_builder.cc's folded-constant fallback, and no spec covered it directly. - Rename QNN UT suites to op-first (QnnUnit_<Op>_<Tier>[_<Variant>]Test) so run_snapshot_accuracy.sh --filter and generate_coverage.sh can recover the op segment; the prior naming glued the variant onto the op with no delimiter, so --filter=Clip matched 0 tests in CI. - Goldens are not committed: the harness reads them from $QNN_UT_SNAPSHOT_GOLDEN_DIR. Unset/absent -> snapshot tests skip (GOLDEN_ABSENT) and accuracy always runs, so golden staleness can never redden the build. - generate_coverage.sh is now the single CI coverage orchestrator, running three phases (component -> snapshot -> accuracy) into one lcov capture, with component + accuracy gating and snapshot non-gating (drift is a routing signal, not a failure). run_snapshot_accuracy.sh is the local-only two-pass runner; its --update-goldens flag is renamed to --generate-goldens to match what it actually does (writes from scratch, not incrementally), and its embedded-python test-name extraction is factored out into extract_snapshot_groups.py. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
yuhuchua-qti
force-pushed
the
dev/yuhuchua/migrate-clip-ut
branch
from
September 8, 2026 04:54
8db7faf to
096e681
Compare
yuhuchua-qti
force-pushed
the
dev/yuhuchua/accuracy-routing-gate
branch
from
September 8, 2026 11:02
0a2b8a3 to
b04074e
Compare
Skip accuracy cases whose paired snapshot passed at a matching golden version; run the rest (drift, golden absent, unmapped, or version mismatch). A version match requires BOTH the QAIRT backend and ORT runtime versions in the golden manifest to equal the current ones -- either up-level can shift numerics even when the emitted graph structure is unchanged. This replaces the unconditional full QnnUnit_Accuracy_* run in the coverage script's accuracy phase. accuracy_gate.py is pure functions + a thin CLI, wired into generate_coverage.sh between the snapshot and accuracy phases. Any gate failure falls back to a full run so coverage is never silently dropped. Unit-tested with synthetic fixtures: pytest qcom/scripts/linux/tests Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_SNAPSHOT_TIER_RE required an underscore on both sides of the tier token, so bare (no-variant) op-first suites like QnnUnit_Clip_SnapshotTest never matched and fell through as unmapped, forcing their accuracy cases to always run instead of being correctly paired against the snapshot result. Confirmed via a live generate_coverage.sh run showing unmapped=5 matching exactly the bare Clip snapshot suite's case count.
yuhuchua-qti
force-pushed
the
dev/yuhuchua/accuracy-routing-gate
branch
from
September 9, 2026 02:26
0ab37c3 to
edf9c0f
Compare
detect_ort_version's NOTE referenced a nonexistent qnn_ut_version.sh; the real shared getter is PR2/PR3's resolve_tool_versions.sh. Also add the same NOTE to detect_qairt_version, which was missing one entirely.
yuhuchua-qti
force-pushed
the
dev/yuhuchua/migrate-clip-ut
branch
3 times, most recently
from
September 16, 2026 08:24
d937314 to
84b714b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the unconditional
QnnUnit_Accuracy_*full run in the coverage script's accuracy phase with a gate that skips accuracy cases whose paired snapshot already passed at a matching golden version, and runs the rest.Skip a case only when both hold:
Any other situation runs accuracy: snapshot drift (failed), golden absent / backend unavailable (skipped, no PASS proof), no paired snapshot (unmapped), or version mismatch / no manifest. Reasons are collected as a union with no short-circuit, so drift is always surfaced in the summary even when a version mismatch already forced a full run.
What's here
qcom/scripts/linux/accuracy_gate.py— pure decision functions + a thin CLI.qcom/scripts/linux/generate_coverage.sh— wires the gate between the snapshot and accuracy phases; prints a developer-facing summary. Any gate failure (missing snapshot JSON, list failure, gate error, empty filter) falls back to a full run, so coverage is never silently dropped.qcom/scripts/linux/tests/{conftest.py,test_accuracy_gate.py}— fixture unit tests (synthetic gtest JSON + accuracy lists; no binary, no build).Stacked on #<PR1: dev/yuhuchua/migrate-clip-ut>.
Fix included
The snapshot→accuracy suite-name pairing regex required an underscore on both sides of the tier token (
Snapshot/SessionSnapshot), so bare (no-variant) op-first suites likeQnnUnit_Clip_SnapshotTestnever matched and fell through as unmapped — forcing those accuracy cases to always run instead of being correctly paired against the snapshot result. Fixedderive_accuracy_suite's regex to not require a trailing underscore, with a regression test covering both the bare and variant suite forms. Confirmed via a realgenerate_coverage.shrun:unmappedwent from 5 to 0 for Clip.Test plan
pytest qcom/scripts/linux/tests— 30 passedpython qcom/build_and_test.py lint— cleancoverage_linux_x86_64build + run on this branch (rebased directly onto PR1's latest tip): component 2504 passed / 12 skipped (env-limited), accuracy 16 passed / 0 failed, gate summary showsunmapped=0(no manifest locally -> full run, as expected:absent=16 version-mismatch=16)