Skip to content

[CICD] Add train image pipeline and real TE-FL smoke testCi/ascend train te smoke#1250

Open
AlexMa616 wants to merge 3 commits into
flagos-ai:mainfrom
BrianPei:ci/ascend-train-te-smoke
Open

[CICD] Add train image pipeline and real TE-FL smoke testCi/ascend train te smoke#1250
AlexMa616 wants to merge 3 commits into
flagos-ai:mainfrom
BrianPei:ci/ascend-train-te-smoke

Conversation

@AlexMa616

Copy link
Copy Markdown

Description

Add an Ascend training image pipeline and enable the Ascend CI suite with a
real TransformerEngine-FL training smoke test.

The change builds the training runtime from Ascend-specific Docker and installer
files, validates Megatron-LM-FL and TE-FL NPU initialization, and runs a
two-device Qwen3 0.6B smoke test through the TE-FL reference backend.

Chip-specific setup remains outside reusable common workflows. Ascend behavior
is defined through the platform configuration, Ascend installer and launcher,
Dockerfile, and dedicated image-build workflow. No FlagScale core source is
changed.

Prerequisite: TransformerEngine-FL must include 771a7488
(fix(ascend): include NPU backend in Python package) or an equivalent upstream
change before rebuilding the image from the official TE-FL repository.

Type of change

  • Infra/Build change (changes to CI/CD workflows or build scripts)
  • Bug fix
  • Code refactoring
  • New feature (non-breaking change which adds functionality)
  • Documentation change
  • Breaking change

Changes

  • Added a trusted Ascend training image build and Harbor publishing workflow.
  • Added the Ascend training Dockerfile and source dependency installation for
    Megatron-LM-FL and TransformerEngine-FL.
  • Added an Ascend launcher that restores TE-FL NPU compatibility after Megatron
    platform initialization.
  • Added a two-device Qwen3 0.6B mock-data smoke test using
    transformer_impl: transformer_engine and TE-FL reference.torch operators.
  • Added platform-neutral finite-loss smoke validation and parser tests.
  • Enabled Ascend through the standard all_tests workflow and selected the
    validated Harbor training image.
  • Aligned torchvision with the validated torch 2.7.1 and
    torch_npu 2.7.1.post2 runtime.
  • Improved installer cache path handling, resumable sccache downloads, and
    relative requirements include processing.

Validation

  • Repository-built training image smoke:
    • Completed 3 training iterations on two Ascend NPUs.
    • Dispatched TE-FL GEMM, RMSNorm forward/backward, and optimizer operations
      through reference.torch.
    • No skipped or NaN iterations.
    • 1 passed
    • FLAGSCALE_ASCEND_REPO_IMAGE=PASS
  • Validated Harbor image:
    • harbor.baai.ac.cn/flagos-dev/flagscale:manual-20260724-ascend-train-426ad116-te771a7488
    • Digest: sha256:2fe15cf8b339b6828395fc3ce56e67b004b7c5f251d9e6673f20579819fe2c16
  • GitHub Actions:
  • Local focused checks:
    • Shell, Python, YAML, and Git diff validation passed.
    • Ascend training matrix parsing passed.
    • Result parser tests: 2 passed.

Checklist

  • I have read and followed the contributing guidelines
  • The functionality is complete for the Ascend training pipeline and smoke scope
  • I have commented my code, particularly in CI workflow setup steps
  • My changes generate no new warnings
  • I have tested the training workflow on the Ascend platform

@AlexMa616
AlexMa616 requested a review from aoyulong as a code owner July 24, 2026 14:21
@CLAassistant

CLAassistant commented Jul 24, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@AlexMa616
AlexMa616 force-pushed the ci/ascend-train-te-smoke branch from a6bffa2 to be82b5a Compare July 24, 2026 14:44
@AlexMa616
AlexMa616 force-pushed the ci/ascend-train-te-smoke branch from be82b5a to c2a56d4 Compare July 24, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants