fix(norm,moe): restore cuDNN source tensor rank; correct MoE grouped-matmul graphs - #39
Merged
Merged
Conversation
Regenerate Workloads/headline/norm.tar.gz from the latest main tarball while restoring the tensor shapes used by cudnn-frontend/benchmark/norms/ benchmark_single_norm.py: - activations and gradients: [N,C,1,1] - scale/bias and their gradients: [1,C,1,1] - saved RMSNorm/LayerNorm statistics: [N,1,1,1] - epsilon: [1,1,1,1], float/Float32Value The previous extraction squeezed trailing singleton dimensions to rank 2 and used a double epsilon. HIP MLOPS norm plans require rank 4 or 5, and the LayerNorm validator requires float epsilon. Keep the existing fp32 statistic/affine correction unchanged. Updated norm/MANIFEST.md with source references, corrected layout, and the reason for the regeneration. Validation: tools/check_deserialize.py --level json reports files=30 ok=30 fail=0 after fresh tar extraction. All tensors in all 30 graphs are rank 4; all epsilon tensors are float/Float32Value with dims [1,1,1,1].
Migrate four unique forward LayerNorm graphs from the legacy cudnn_frontend.tar.gz sample suite into norm.tar.gz: - fp16 8192x128 - fp16 2048x128 - fp16 4096x768 - fp16 4096x128 Exclude the legacy bf16 1024x12288 graph because it duplicates the model-derived GPT-3 LayerNorm shape already in norm.tar.gz. Preserve the legacy synthetic source metadata and add norm_type/pass annotations. Normalize the migrated graphs to the source-compatible rank-four layout and float32 epsilon representation. Expand norm.tar.gz from 30 to 34 graphs: 19 forward and 15 backward. Update MANIFEST.md with the selection and provenance. Validation: tools/check_deserialize.py --level json reports files=34 ok=34 fail=0 after a fresh tar extraction. DVC object pushed to storage.
Remove the four fp16 LayerNorm forward graphs migrated into Workloads/headline/norm.tar.gz from Workloads/microbench/cudnn_frontend.tar.gz: - 8192x128 - 2048x128 - 4096x768 - 4096x128 Keep the legacy bf16 1024x12288 LayerNorm sample because it was not migrated; the headline workload already covers that shape with the model-derived GPT-3 case. Update the legacy MANIFEST totals from 53 to 49 graphs and document the migration. No remaining legacy graph content changed. Revalidated both tarballs: - cudnn_frontend.tar.gz: 49/49, fail=0 - norm.tar.gz: 34/34, fail=0
All MoE graphs in moe.tar.gz and cudnn_bench_moe.tar.gz failed opgraph deserialization. Root causes: - Tensors were flattened; hipDNN's MoE node requires rank-3 shapes ([1, T, K] / [E, K, N] / [E, 1, 1]). - first_token_offset used an E+1-length cumulative-boundary array; hipDNN expects the E-length per-expert start-offset array. - top_k was set to the real routing value on NONE/GATHER nodes, but those modes require top_k == 0 (SCATTER only). - GATHER-mode dgrad nodes carried a token_ks input inherited from their paired SCATTER forward node; GATHER forbids token_ks. - token_index/token_ks used the activation dtype instead of INT32. - Some graphs had a disconnected router-GEMM component; hipDNN requires one connected component per graph, so these were split into separate *__router.json files. - SCATTER-mode output row count used the reduced final-token count; hipDNN's contract requires the dispatched-token count. Excluded 21 wgrad graphs whose token/doutput row counts mismatch (pre- vs post-scatter counts) -- MoeGroupedMatmulBwdAttributes has no scatter/token_index support today, so this isn't fixable by reshaping. Result: 191/191 graphs (165 moe.tar.gz + 26 cudnn_bench_moe.tar.gz) pass from_json + validate + build_operation_graph.
SamuelReeder
changed the title
3 tasks
SamuelReeder
added a commit
that referenced
this pull request
Sep 1, 2026
--level json only exercises dnn_benchmarking's pure-Python GraphLoader, which doesn't know hipDNN's real node contract (tensor rank, dtype, connected-components, per-node field constraints). Verified against the pre-fix moe.tar.gz blob: --level json passed 121/121 on graphs that --level opgraph correctly failed 113/121 on -- the exact bug class fixed in the moe.tar.gz PR. --level json stays as a fast pre-check, but the opgraph check (real hipdnn_frontend, built via the same no-GPU path as setup.yml's rocm-build job) is now the actual gate. Known red: one pre-existing failure (norm/bwd GPT3 LayerNorm-backward padding constraint, documented in applicability_results/STATUS.md) will be fixed in the norm/moe DVC PR (#39), not here.
The one layer_norm config (gpt3-175b) shipped scale/bias/dscale/dbias as [1,C,1,1], the same axis convention as every RMSNorm graph in this package. LayernormBackwardNode enforces a trailing-normalized-axis convention instead: with normalized_dim_count=1 it requires the real (C) axis to be the last dim of scale, not the second. The old combination failed opgraph-level backward validation with "LayernormBackwardNode requires that one-padded scale (input) conforms to the specified normalized_dim_count"; forward's node has no equivalent check, so the forward graph passed anyway with the same axis order. Fixed by reshaping scale/bias/dscale/dbias to [1,1,1,C] for both the forward and backward gpt3-175b graphs. x/y/dx/dy/mean/inv_variance are unaffected. Scoped to the one layer_norm config; every rms_norm graph is untouched. tools/check_deserialize.py --level opgraph: norm.tar.gz 34/34 (was 33/34), and the full Workloads/ corpus is now 8696/8696.
--level opgraph only calls from_json/validate/build_operation_graph, which assemble the backend graph descriptor from JSON and never touch an engine. hipdnn.Handle() otherwise eagerly loads every engine plugin on construction, and HIPBLASLT_ENGINE's plugin aborts hard with 'no ROCm-capable device is detected' on GPU-less machines (e.g. CI runners). Call set_engine_plugin_paths([], ABSOLUTE) before creating the Handle to skip all engine plugin loading -- confirmed no regression locally (8696/8696 still pass). Also copies the check-deserialize workflow (from #41) onto this branch temporarily so this PR gets its own CI run against the current fix commits before #41 merges to main.
SamuelReeder
marked this pull request as ready for review
September 3, 2026 20:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three DVC workload fixes, landed on the same branch:
Workloads/headline/norm.tar.gz, and migrates four unique fp16LayerNorm graphs out of the legacy
cudnn_frontend.tar.gzsamplesuite into the headline set.
Workloads/headline/moe.tar.gzandWorkloads/microbench/cudnn_bench_moe.tar.gzMoE grouped-matmulgraphs to match hipDNN's
MoeGroupedMatmulAttributesnode contract.layer_normconfig'sscale/bias axis order so its backward graph passes hipDNN's real
node validation, not just JSON schema validation.
Risk assessment
Low. All three changes update DVC-tracked workload archives and their
internal manifests only. No hipDNN or provider source code changes.
Testing summary
norm (rank)
tools/check_deserialize.py --level json: norm 34/34 valid,fail=0.tools/check_deserialize.py --level json: cleaned legacy cuDNN tar 49/49 valid,fail=0.[1,1,1,1]andfloat/Float32Value.after the four unique synthetic fp16 LayerNorm cases were added.
moe
tools/check_deserialize.py --level opgraph(built hipDNN,from_jsonvalidate+build_operation_graph): 191/191 graphs pass,fail=0.non-INT32 routing tensors, or a non-zero
top_koutsideSCATTERmode.norm (LayerNorm axis)
check-deserializeCI job (ci: check JSON-level graph deserialization on all DVC workloads #41):--level json(pure-Python schema check) passed 34/34 even before this fix, but
--level opgraph(realhipdnn_frontend) failed 1/34 on the gpt3-175bbackward graph -- exactly the gap that CI job exists to catch.
tools/check_deserialize.py --level opgraphreportsnorm.tar.gz34/34, and the fullWorkloads/corpus is 8696/8696.All three fixes pushed to the DVC storage remote.
Testing checklist
Workloads/corpus passes--level opgraph(8696/8696).Technical changes
norm (rank)
[N,C,1,1]activation and gradient tensors.[1,C,1,1]scale and bias tensors.[N,1,1,1]saved-statistic tensors.[1,1,1,1]epsilon tensors with float32 representation.moe
(
[1, T, K]/[E, K, N]/[E, 1, 1]).first_token_offsetfrom anE+1-length cumulative-boundaryarray to the
E-length per-expert start-offset array hipDNN expects.top_konNONE/GATHERmode nodes (onlySCATTERuses it).token_ksinput incorrectly carried byGATHER-modedgradnodes (forbidden in that mode).
token_index/token_kstensors to INT32.*__router.jsonfiles (hipDNN requires one connected component per graph).SCATTER-mode output row count to the dispatched-token count.wgradgraphs whosetoken/doutputrow counts mismatch(pre- vs post-scatter counts); the current backward node has no
scatter/token_index support, so this isn't a shape fix. Documented as a
known gap in the manifest.
norm (LayerNorm axis)
LayernormBackwardNodeenforces a trailing-normalized-axis convention:with
normalized_dim_count=1the real (C) axis of a rank-matched("one-padded")
scalemust be the last dim, not the second. Thegpt3-175b config (the only
layer_normcase; every other graph isrms_norm) shippedscale/bias/dscale/dbiasas[1,C,1,1],the same axis order used for RMSNorm -- which has no such check.
scale/bias/dscale/dbiasto[1,1,1,C]for both theforward and backward gpt3-175b graphs.
x/y/dx/dy/mean/inv_varianceare unaffected; this validator only inspectsscale'sshape. Scoped to the one
layer_normconfig -- everyrms_normgraphis untouched.