ci: check JSON-level graph deserialization on all DVC workloads - #41
Open
SamuelReeder wants to merge 10 commits into
Open
ci: check JSON-level graph deserialization on all DVC workloads#41SamuelReeder wants to merge 10 commits into
SamuelReeder wants to merge 10 commits into
Conversation
Regenerate Workloads/headline/norm.tar.gz from the latest main tarball while restoring the tensor shapes used by cudnn-frontend/benchmark/norms/ benchmark_single_norm.py: - activations and gradients: [N,C,1,1] - scale/bias and their gradients: [1,C,1,1] - saved RMSNorm/LayerNorm statistics: [N,1,1,1] - epsilon: [1,1,1,1], float/Float32Value The previous extraction squeezed trailing singleton dimensions to rank 2 and used a double epsilon. HIP MLOPS norm plans require rank 4 or 5, and the LayerNorm validator requires float epsilon. Keep the existing fp32 statistic/affine correction unchanged. Updated norm/MANIFEST.md with source references, corrected layout, and the reason for the regeneration. Validation: tools/check_deserialize.py --level json reports files=30 ok=30 fail=0 after fresh tar extraction. All tensors in all 30 graphs are rank 4; all epsilon tensors are float/Float32Value with dims [1,1,1,1].
Migrate four unique forward LayerNorm graphs from the legacy cudnn_frontend.tar.gz sample suite into norm.tar.gz: - fp16 8192x128 - fp16 2048x128 - fp16 4096x768 - fp16 4096x128 Exclude the legacy bf16 1024x12288 graph because it duplicates the model-derived GPT-3 LayerNorm shape already in norm.tar.gz. Preserve the legacy synthetic source metadata and add norm_type/pass annotations. Normalize the migrated graphs to the source-compatible rank-four layout and float32 epsilon representation. Expand norm.tar.gz from 30 to 34 graphs: 19 forward and 15 backward. Update MANIFEST.md with the selection and provenance. Validation: tools/check_deserialize.py --level json reports files=34 ok=34 fail=0 after a fresh tar extraction. DVC object pushed to storage.
Remove the four fp16 LayerNorm forward graphs migrated into Workloads/headline/norm.tar.gz from Workloads/microbench/cudnn_frontend.tar.gz: - 8192x128 - 2048x128 - 4096x768 - 4096x128 Keep the legacy bf16 1024x12288 LayerNorm sample because it was not migrated; the headline workload already covers that shape with the model-derived GPT-3 case. Update the legacy MANIFEST totals from 53 to 49 graphs and document the migration. No remaining legacy graph content changed. Revalidated both tarballs: - cudnn_frontend.tar.gz: 49/49, fail=0 - norm.tar.gz: 34/34, fail=0
All MoE graphs in moe.tar.gz and cudnn_bench_moe.tar.gz failed opgraph deserialization. Root causes: - Tensors were flattened; hipDNN's MoE node requires rank-3 shapes ([1, T, K] / [E, K, N] / [E, 1, 1]). - first_token_offset used an E+1-length cumulative-boundary array; hipDNN expects the E-length per-expert start-offset array. - top_k was set to the real routing value on NONE/GATHER nodes, but those modes require top_k == 0 (SCATTER only). - GATHER-mode dgrad nodes carried a token_ks input inherited from their paired SCATTER forward node; GATHER forbids token_ks. - token_index/token_ks used the activation dtype instead of INT32. - Some graphs had a disconnected router-GEMM component; hipDNN requires one connected component per graph, so these were split into separate *__router.json files. - SCATTER-mode output row count used the reduced final-token count; hipDNN's contract requires the dispatched-token count. Excluded 21 wgrad graphs whose token/doutput row counts mismatch (pre- vs post-scatter counts) -- MoeGroupedMatmulBwdAttributes has no scatter/token_index support today, so this isn't fixable by reshaping. Result: 191/191 graphs (165 moe.tar.gz + 26 cudnn_bench_moe.tar.gz) pass from_json + validate + build_operation_graph.
Adds a check-deserialize workflow that pulls every DVC-tracked workload tarball, extracts it, and runs tools/check_deserialize.py --level json across all of them. Catches graphs that fail to parse/validate against dnn_benchmarking's pure-Python GraphLoader before they reach a real benchmark run -- no hipDNN build or GPU required, so it runs on every PR/push touching Workloads/, the graph loader, or the check tool itself.
--level json only exercises dnn_benchmarking's pure-Python GraphLoader, which doesn't know hipDNN's real node contract (tensor rank, dtype, connected-components, per-node field constraints). Verified against the pre-fix moe.tar.gz blob: --level json passed 121/121 on graphs that --level opgraph correctly failed 113/121 on -- the exact bug class fixed in the moe.tar.gz PR. --level json stays as a fast pre-check, but the opgraph check (real hipdnn_frontend, built via the same no-GPU path as setup.yml's rocm-build job) is now the actual gate. Known red: one pre-existing failure (norm/bwd GPT3 LayerNorm-backward padding constraint, documented in applicability_results/STATUS.md) will be fixed in the norm/moe DVC PR (#39), not here.
The one layer_norm config (gpt3-175b) shipped scale/bias/dscale/dbias as [1,C,1,1], the same axis convention as every RMSNorm graph in this package. LayernormBackwardNode enforces a trailing-normalized-axis convention instead: with normalized_dim_count=1 it requires the real (C) axis to be the last dim of scale, not the second. The old combination failed opgraph-level backward validation with "LayernormBackwardNode requires that one-padded scale (input) conforms to the specified normalized_dim_count"; forward's node has no equivalent check, so the forward graph passed anyway with the same axis order. Fixed by reshaping scale/bias/dscale/dbias to [1,1,1,C] for both the forward and backward gpt3-175b graphs. x/y/dx/dy/mean/inv_variance are unaffected. Scoped to the one layer_norm config; every rms_norm graph is untouched. tools/check_deserialize.py --level opgraph: norm.tar.gz 34/34 (was 33/34), and the full Workloads/ corpus is now 8696/8696.
5 tasks
Collaborator
Author
Handle creation or applicability appears to need a GPU. I think I can get around this by not loading plugins.
|
--level opgraph only calls from_json/validate/build_operation_graph, which assemble the backend graph descriptor from JSON and never touch an engine. hipdnn.Handle() otherwise eagerly loads every engine plugin on construction, and HIPBLASLT_ENGINE's plugin aborts hard with 'no ROCm-capable device is detected' on GPU-less machines (e.g. CI runners). Call set_engine_plugin_paths([], ABSOLUTE) before creating the Handle to skip all engine plugin loading -- confirmed no regression locally (8696/8696 still pass). Also copies the check-deserialize workflow (from #41) onto this branch temporarily so this PR gets its own CI run against the current fix commits before #41 merges to main.
--level opgraph only calls from_json/validate/build_operation_graph, which assemble the backend graph descriptor from JSON and never touch an engine. hipdnn.Handle() otherwise eagerly loads every engine plugin on construction, and HIPBLASLT_ENGINE's plugin aborts hard with 'no ROCm-capable device is detected' on GPU-less machines -- exactly what this workflow's CI runner is. Call set_engine_plugin_paths([], ABSOLUTE) before creating the Handle to skip all engine plugin loading. Confirmed no regression locally on a machine with a real GPU (8696/8696 still pass).
…l end-to-end CI run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
check-deserializeGitHub Actions job that runstools/check_deserialize.py --level jsonagainst every DVC-trackedworkload tarball under
Workloads/. Catches graphs that fail toparse/validate before they reach a real benchmark run.
Risk assessment
Low. New workflow file only; no changes to existing CI, source, or
workload data.
Testing summary
tracked tarballs:
level=json files=8696 ok=8696 fail=0.dvc pullsucceeds anonymously (no AWS credentials in theenvironment) against the
storageremote'sallow_anonymous_loginconfig.
Testing checklist
Workloads/**/*.tar.gzand run the json-level check.dvc pullneeds no secrets.Technical changes
--level jsononly exercisesdnn_benchmarking.graph.GraphLoader(pure Python) -- no hipDNN build or GPU needed, so the job stays fast
and runs on every PR/push touching
Workloads/**, the graph loader,or the check tool itself.
rocm-librariesisn't needed for this level.