feat(oracle): add --oracle autotuned-vs-OOTB benchmark comparison [ALMIOPEN-2462] - #32
Open
SamuelReeder wants to merge 7 commits into
Open
feat(oracle): add --oracle autotuned-vs-OOTB benchmark comparison [ALMIOPEN-2462]#32SamuelReeder wants to merge 7 commits into
SamuelReeder wants to merge 7 commits into
Conversation
Add an opt-in --oracle mode that measures each hipDNN engine row twice: the normal heuristic-selected run (OOTB) and a run of the plan hipDNN auto-tuning selects for that same engine. Both timings, the winning plan's identity, and the OOTB-vs-oracle delta appear in the console output and as separately addressable SuiteResult JSON fields. The oracle is measured per engine row, not per graph. A graph-level "best engine" number is derivable from per-engine rows by taking minima; per-engine detail is not recoverable from a single graph-level number. Candidates are restricted with AutotuneConfig.engine_id_filter, so no new ProviderEngineResult role is needed and pass/fail counts are unaffected. The OOTB build hard-selects one engine and compiles only the active plan; auto-tuning needs every plan compiled. The two are mutually exclusive, so the oracle uses a second Executor over a second graph build rather than mutating a prepared OOTB executor. Record the hipDNN cache/benchmarking environment under metadata.hipdnn_selection_env and warn once when the exact-match engine-ranking cache is left enabled: a persisted ranking makes the OOTB row reuse a previously benchmarked engine order and shrinks the reported gap. The tool never mutates those variables, so the OOTB number stays identical between an oracle run and a plain run of the same command. A tuning failure records oracle_error and leaves the OOTB row and the run's exit code intact. Runs without the flag emit the same output as before. JIRA ID : ALMIOPEN-2462
…fields
Review follow-ups on the --oracle pass.
Bar every other engine before build_plans(ALL). The ALL loop marks a
barred plan and skips it before finalizePlanDescriptor, so this drops
E-1 engine compiles per row, and get_autotune_workspace_size() ignores
barred plans, so the oracle workspace shrinks to the target engine's
plans. That matters because it is allocated while the OOTB workspace is
still live. The tuning candidate set is unchanged: engine_id_filter had
already restricted benchmarking to that engine, so barring the rest only
changes which non-benchmarked reason the others report.
Feed the run's --warmup into the sweep. The binding default is 1 warmup
iteration, which left a provider's first-execute kernel sampling inside
the window that ranks the candidates, so the winner could be picked on a
priming cost rather than steady-state time. strategy stays
RUN_UNTIL_STABLE.
Report the target engine's own failure when no candidate succeeds.
Candidates for other engines come back as non-benchmarked rejections
("Plan excluded by engineIdFilter." / "Plan barred ..."), so the previous
first-non-empty-message scan could surface the filter's message as
oracle_error instead of the real failure.
Drop OracleResult.engine_id/engine_name: engine_id_filter makes both
constant and equal to the enclosing row's engine_id/provider. Executor
.autotune now rejects a winner from another engine outright, which turns
the redundancy into a checked invariant rather than a duplicated field.
Also drop the redundant re-sort of the winners: rankAndSelectWinner
already returns succeeded candidates first in ascending rank order.
JIRA ID : ALMIOPEN-2462
Pass the handle to get_plan_name(). Newer hipDNN bindings need it to name plugin-supplied engines and fall back to a hex engine ID without it, which is exactly the engine class this tool benchmarks. Bindings at the pinned submodule revision take no handle and resolve plugin engines from a process-wide registry, so fall back to the no-handle form when the handle is rejected. That keeps the reported plan name correct on the current pin and on a bumped one, without folding a submodule bump into this PR: bumps land as their own chore PR here. Tighten the integration assertion accordingly. It asserted only that plan_name was truthy, which a hex ID satisfies; it now pins plan_name to the row's own registered engine name and rejects a "0x" prefix. Drop the input reload before tuning. Inputs are disjoint from the output set, and the OOTB pass already relies on them surviving its own warmup, timed loop, and correctness re-execution without a reload, so zeroing the outputs alone restores the comparable starting state. Removes one host-to-device copy per engine row and the now-unused input_data parameter. Note the source of the truthy-value set and the fact that the oracle row's cpu_build_time_ms is not comparable to the OOTB row's, since the autotune build enumerates every engine's plans before barring all but one. JIRA ID : ALMIOPEN-2462
…tune APIs Bump rocm-libraries from 7c311dc to e7b7720 (tip of develop, the branch .gitmodules tracks) so the oracle pass can call the autotune APIs directly instead of working around their absence. - get_plan_name() now takes the handle. hipDNN needs it to name plugin-supplied engines and reports a hex engine ID without it, which is the engine class this tool benchmarks. Replaces the try/except probe that supported both binding shapes. - AutotuneResult.excluded_by_caller marks candidates hipDNN's own caller-side filters rejected without benchmarking (engine_id_filter, deselect_engines, the workspace ceiling). Selecting the reported failure on that flag is what the field is for, and it also covers a target-engine plan dropped by the workspace ceiling, which keying on engine_id alone could not distinguish. Fix the library search order in setup_env.py, which the bump exposed. ROCM_PATH is set to install_prefix, but LD_LIBRARY_PATH listed the toolchain wheel's lib directory first, so the hipDNN that setup_env builds from the submodule was shadowed by the wheel's prebuilt copy and never actually loaded. That was invisible while the two agreed on symbols; with this bump the wheel's older libhipdnn_backend.so lacks hipdnnGetEngineNameById_ext and the frontend bindings fail to import with an undefined symbol. Listing install_prefix first makes the search path agree with ROCM_PATH and loads the libraries that were just built.
Add -DHIPDNN_ENABLE_KERNEL_INGESTOR=ON to the superbuild configure. HIPDNN_ENABLE_SDPA was already ON (plus ENABLE_ASM_SDPA_ENGINE); no change needed there. Enabling the ingestor pulls in two things setup_env.py must also handle: - The ingestor engine's descriptor-packaging step needs rocm_kpack, which is not part of this superbuild's own sources. -DHIPKERNELPROVIDER_KPACK_ALLOW_FETCH=ON does the sanctioned sparse, depth-1, pinned-commit checkout from rocm-systems (the same recipe the hipDNN dev container and rocm-libraries' own CI use) rather than requiring a host-local rocm-systems checkout. Re-entrant: a warm build directory touches the network only once. - Building the ingestor code for the first time on this host's toolchain hits a libstdc++-internal deprecation (std::stable_sort's get_temporary_buffer, IKernelHeuristic.hpp:63) promoted to a hard error by -Werror. Confirmed this is a local toolchain skew, not a hipDNN defect: rocm-libraries' own Linux superbuild CI builds this exact flag combination green. Downgrade it to a warning (-Wno-error=deprecated-declarations) for this bootstrap script's own build only, matching the -Wno-error=unused-command-line-argument idiom the same CMake target already uses; hipDNN's own CMakeLists and CI's warning policy are untouched. Verified: HIPDNN_ENABLE_KERNEL_INGESTOR and HIPDNN_ENABLE_SDPA both ON in the resulting CMakeCache; full rebuild succeeds; import through the generated activate script is clean; unit suite (937 passed) and oracle GPU integration tests (gfx90a) unaffected; oracle E2E run unchanged. JIRA ID : ALMIOPEN-2462
CI's rocm-build job failed after the previous commit:
CMake Error ... hkp: /opt/hostedtoolcache/Python/3.12.14/x64/bin/python3.12
cannot import rocm_kpack.compression, rocm_kpack.kpack (rocm_kpack
needs zstandard>=0.20.0 and msgpack).
HkpPackaging.cmake resolves its interpreter with a bare
find_package(Python3 COMPONENTS Interpreter REQUIRED) and imports
rocm_kpack under whatever that finds. Left unpinned, on a fresh runner
that lands on the ambient system Python, which has neither dependency.
It happened to succeed in this repo's own local verification only
because that shell's PATH put an unrelated venv first that already had
both packages installed for something else -- coincidence, not a fix,
and exactly why it broke on a clean runner.
Pin -DPython3_EXECUTABLE to the venv this script already manages, and
install zstandard/msgpack into that same venv before configuring, so
resolution is deterministic instead of depending on what happens to be
first on PATH. Matches the existing -DPython_EXECUTABLE={self.py}
precedent in build_and_install_bindings().
Verified the fix is real, not incidental: rebuilt with a scratch PATH
exposing only cmake/ninja (no Python), confirmed the ambient fallback
Python genuinely lacks zstandard/msgpack, and the build still succeeds
with Python3_EXECUTABLE correctly resolving to the managed venv in
CMakeCache.txt. Unit suite (937 passed) and oracle GPU integration
unaffected.
The reported speedup was measuring warmup, not tuning. The sweep executes the engine's plans up to a hundred times, so the tuned run was timed on a much hotter device than the OOTB pass ever saw. Comparing the two reported a speedup even when the sweep had a single candidate and could not have changed anything. Measured on gfx90a across the 21 sample graphs, every row had candidates_benchmarked == 1, yet the tool printed: --warmup 10 (default): mean 1.053x, max 1.351x --warmup 5: mean 1.170x, max 2.191x --warmup 200: mean 1.000x, max 1.018x The OOTB number converged down onto a stable oracle number as warmup rose, and a plain non-oracle run agreed with OOTB, which pinned the oracle side as the privileged one rather than OOTB as inflated. Re-time the heuristic plan between the sweep and the tuned run, so both operands carry the same warmup history and the ratio isolates the plan change. The row's own OOTB columns keep the untouched out-of-the-box number; the comparison uses the new warm baseline, carried on the oracle payload as warm_baseline_gpu_kernel_stats / warm_baseline_host_stats and surfaced as oracle_delta.baseline_mean_ms. After the fix, same host and graphs: --warmup 10 (default): mean 1.001x, max 1.019x --warmup 1: mean 1.020x, max 1.158x build_oracle_delta now takes only the OracleResult: the row can no longer supply the baseline by accident. A unit test pins the call order (ootb.benchmark, oracle.autotune, ootb.benchmark, oracle.benchmark) so a revert to the pre-sweep comparand fails, and another asserts the delta is None when no warm baseline was recorded. JIRA ID : ALMIOPEN-2462
SamuelReeder
marked this pull request as ready for review
September 1, 2026 06:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
--oraclemode that measures each hipDNN engine row twice: the normal heuristic-selected run (OOTB) and a run of the plan hipDNN auto-tuning selects for that same engine. This turns "how much performance is the heuristic leaving on the table for this engine, on this graph?" into a number the tool reports directly, instead of a manual two-run comparison. Runs without the flag emit the same output as before.JIRA ID : ALMIOPEN-2462
Risk Assessment
Risk 3. New opt-in feature path with real integration surface: it adds a second execution mode to
Executor.preparethat the existing OOTB path shares, and it extends theSuiteResultJSON schema. The blast radius on existing behavior is small and gated — every new JSON key is emitted only when populated, the flag defaults toFalse, and a tuning failure is caught and recorded asoracle_errorwithout touching the OOTB row, its correctness result, or the run's exit code. What keeps this off risk 2 is thatprepareis on every hipDNN benchmark path, so a mistake there would be broad rather than narrow.ASIC Coverage
Arch-independent host-side tool plumbing. The change adds no kernel, no dispatch rule, and no support-surface change; it calls existing hipDNN frontend bindings (
Graph.autotune,get_autotune_workspace_size,get_plan_name) and reports what they return. Behavior does not vary by ASIC, so a multi-arch sweep is not warranted.Repository CI runs
pytest -m "not gpu"only and has no GPU lane, so the on-device path was verified locally on gfx90a (MI210): the full GPU integration suite plus a manual end-to-end run of both the--oracleand the plain path. Passing PR CI plus that local gfx90a run is sufficient coverage for this change.Testing Summary
preparefork,Executor.autotunecandidate filtering/failure handling, the two-pass suite-runner flow, oracle JSON serialization and the delta-basis rules, reporter table/verbose rendering, CLI flag and TOML plumbing, and the startup warnings.--oracle) output is unchanged: nooracle/oracle_delta/oracle_errorkeys on rows, nohipdnn_selection_envin metadata, and no--oracleleaking into the profiling orchestrator's inner argv allowlist.Testing Checklist
pytest -m "not gpu" -q- Status: Passed (927 passed, 6 skipped)pytest tests/integration/test_suite_execution.py -k Oracle- ASICs: gfx90a (MI210) - Status: PassedHIPDNN_DISABLE_EXACT_ENGINE_CACHE=1 python -m dnn_benchmarking --graph ./graphs/sample_conv_fwd.json --warmup 10 --iters 100 --oracle -v -o /tmp/oracle.json- ASICs: gfx90a (MI210) - Status: Passed (oracle block, both table columns, and the footer aggregate rendered; no cache warning)HIPDNN_DISABLE_EXACT_ENGINE_CACHE- ASICs: gfx90a (MI210) - Status: Passed (warning printed;hipdnn_selection_envrecords the variable asnull)--oracle- ASICs: gfx90a (MI210) - Status: Passed (no oracle keys, nohipdnn_selection_env)black --check src tests- Status: PassedFlags / Guardrails
--oracle(and theoracleTOML key), defaultfalse. The entire feature is behind it; with the flag off, no new code executes and no new JSON key is emitted. Rejected with--backend pytorch, since auto-tuning is a hipDNN engine feature. No enable-by-default plan — the flag runs one tuning sweep per engine row and is meant to stay opt-in because of that cost.Adjacent Tests Considered
Executor.prepareis shared with the OOTB path, so the existing forced-engine tests intests/unit/execution/test_executor_forced_engine.pyare the guard that the heuristic path is unchanged; they pass unmodified, and the new autotune-mode tests were added beside them against the same graph stub.--oraclenever forwards into the child process.count_by_status/ role handling was deliberately left untouched: measuring per engine row rather than adding a new row role means pass/fail counts cannot shift, and the existing suite-results tests cover that.hipErrorInvalidImage("device kernel image is invalid") from the ROCm torch nightly on this host. None touch the oracle path, and every hipDNN-engine test passes.Technical Changes
execution/executor.py:prepare(..., for_autotune=True)creates all execution plans, builds withBuildPlanPolicy.ALL, sizes the workspace withget_autotune_workspace_size(), and skips engine pinning. Newautotune()restricts candidates withAutotuneConfig.engine_id_filter, drops failed candidates, returns the rest rank-sorted, and raisesExecutionErrorwhen none succeeded. Newplan_nameproperty.execution/suite_runner.py:_run_oracle_passruns the second pass inside the OOTBBufferManagerscope so both passes see identical input data, restores the buffers to the OOTB starting state before tuning, and never raises.reporting/suite_results.py:OracleResult,OracleDelta, andbuild_oracle_delta(prefers GPU-kernel basis, falls back to host, returnsNonewhen the oracle mean is zero). Three conditionally-emitted fields onProviderEngineResult;SuiteMetadata.hipdnn_selection_envemitted only for oracle runs.reporting/reporter.py: two summary-table columns and a verboseOracle (auto-tuned):block, both conditional on a row carrying oracle data.cli/,config/:--oracleflag andoracleTOML key,SuiteConfig.oracle,--backend pytorchrejection, a one-time cache warning and a--warmup 0warning, and the suite footer aggregate.Note on the OOTB baseline
hipDNN consults an on-disk exact-match engine-ranking cache before heuristic selection, so a previously persisted ranking can make the OOTB row reuse an already-benchmarked engine order and silently shrink the reported gap. This PR deliberately does not set or clear any hipDNN cache variable: doing so would make the OOTB number differ between an oracle run and a plain run of the same command, which is exactly the confusion the flag exists to remove. Instead it records the relevant variables under
metadata.hipdnn_selection_envand warns once, recommendingHIPDNN_DISABLE_EXACT_ENGINE_CACHE=1for a cold baseline.The oracle pass itself cannot poison that cache: only
autotune_exhaustive_sweep()writes ranking records, and this feature callsautotune(), which never writes.