Optimize GLM5.2 MTP#1481
Closed
whx-sjtu wants to merge 24 commits into
Closed
Conversation
zejunchen-zejun
force-pushed
the
zejun/opt_GLM5.2_0701
branch
from
f4c157f to
18eaab4
Compare
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
GLM-5.2 (glm_moe_dsa) was hard-excluded from the fused indexer path added for DeepSeek-V3.2 in #788, forcing the indexer's rope + fp8-quant + kv-cache write to run as 5-6 separate ops per layer per token. GLM's indexer is structurally identical to V3.2 (index_head_dim=128, qk_rope_head_dim=64, per_1x128 fp8 quant, always-neox indexer rope), so the fused indexer_qk_rope_quant_and_cache kernel is math-equivalent to the per-op path. Notably the recent GLM acc fix (da360a2) fixed a bug that only exists in the per-op path (dropping q_nope on recombine); the fused kernel never had it. - Allow glm_moe_dsa through _supports_fused_indexer_kernel_config, gated by a new env ATOM_ENABLE_GLM_FUSED_INDEXER (default on) for easy A/B + rollback. - Keep the wk+weights_proj GEMM merge OFF for GLM (decoupled in _can_fuse_indexer_wk_weights_proj): that fusion requires merging two checkpoint tensors and GLM names weights_proj "indexers_proj" without a pre-mergeable layout. The dominant win (rope+quant+cache) needs no merge. DeepSeek-V3.2 behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the guard that kept the indexer wk+weights_proj GEMM-merge off for glm_moe_dsa. The guard was overly conservative: it assumed GLM checkpoints ship the projection under a non-standard "indexers_proj" name that couldn't feed the packed_modules_mapping merge. In fact "indexers_proj" only appears in the HF quant config's modules_to_not_convert (remapped via quant_exclude_name_mapping); the actual checkpoint tensors use the standard indexer.wk (fp8 block-scale) + indexer.weights_proj (bf16) paths — exactly the layout IndexerWkWeightsProjLinear's fp8-wk load and the merge expect. GLM now takes the same path as DeepSeek-V3.2: wk + weights_proj collapse into a single BF16 GEMM (IndexerWkWeightsProjLinear), on top of the already-enabled qk-rope+quant+cache fusion. Still gated by ATOM_ENABLE_GLM_FUSED_INDEXER for rollback. DeepSeek-V3.2 behavior unchanged. Verify on GPU: weights load without missing indexer.wk / indexer.weights_proj, accuracy unchanged, and the two small indexer GEMMs collapse to one in the decode breakdown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
* enable ar+norm+quant fusion * fix fusion with online quant
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
zejunchen-zejun
force-pushed
the
zejun/opt_GLM5.2_0701
branch
from
3fe3c11 to
5aacdc6
Compare
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
whx-sjtu
force-pushed
the
hexwang/glm52_opt
branch
2 times, most recently
from
8bcc7dd to
a542a2b
Compare
Restore the "Offline Inference with DP Attention + Expert Parallel" example and the fp8-kv-cache caveat that were dropped when the GLM-5 recipe was reworked into the server-style configs. Requested by maintainer review. Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
- recipes/GLM-5.md: the GLM-5.2 FP8 server recipes loaded zai-org/GLM-5-FP8 (GLM-5) despite the GLM-5.2 titles; point them at zai-org/GLM-5.2-FP8. - quantization/quark/utils.py: stop passing a bare fp8_e8m0 literal for the MXFP8 online-quant scale. E8M0 is the block-scale dtype mandated by the MX (microscaling) format for any per_1x32 scheme (FP4 or FP8), so derive it from the scheme via a single-source _mx_block_scale_dtype() helper and gate on the MX scheme rather than special-casing fp8. The MXFP4 path reuses the same helper so there is one source of truth. Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: zejunchen-zejun <zejun.chen@amd.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
whx-sjtu
force-pushed
the
hexwang/glm52_opt
branch
from
a542a2b to
7e0ff90
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary