[fix] save_checkpoint_and_time error#1237
Open
Darryl233 wants to merge 3 commits into
Open
Conversation
Darryl233
requested review from
aoyulong,
heavyrain-lzy and
zhaoyinglia
as code owners
July 16, 2026 07:24
…#1240) Add standard Apache 2.0 copyright header (Copyright 2026 FlagOS Contributors) to all source files. - **703 files** stamped with Apache 2.0 headers - **80 files** with upstream copyright (HuggingFace, Physical Intelligence, NVIDIA) — left untouched - **104 files** with other licenses (51 MPL, 41 proprietary, 10 MIT, 2 BSD) — third-party/upstream code - **31 files** skipped by rule — binary, JSON, etc. This PR was written in part with the assistance of generative AI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Category
Train
PR Types
Bug Fixes
PR Description
Fix the issue when saving checkpoint under cuda 13.0
[default0]:[megatron_cicd:1610 :0:1610] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
[default0]:==== backtrace (tid: 1610) ====
[default0]: 0 /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7fab66d08654]
[default0]: 1 /opt/hpcx/ucx/lib/libucs.so.0(+0x3684c) [0x7fab66d0884c]
[default0]: 2 /opt/hpcx/ucx/lib/libucs.so.0(+0x36a88) [0x7fab66d08a88]
[default0]: 3 /lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x7fab7257d330]
[default0]: 4 /lib/x86_64-linux-gnu/libnvidia-ml.so.1(+0x7922e) [0x7faa4ba7822e]
[default0]: 5 /lib/x86_64-linux-gnu/libnvidia-ml.so.1(+0x7f056) [0x7faa4ba7e056]
[default0]: 6 /lib/x86_64-linux-gnu/libnvidia-ml.so.1(nvmlDeviceGetTotalEnergyConsumption+0x5b) [0x7faa4ba7e1cb]
[default0]: 7 /lib/x86_64-linux-gnu/libffi.so.8(+0x7b16) [0x7fab71eaab16]
[default0]: 8 /lib/x86_64-linux-gnu/libffi.so.8(+0x43ef) [0x7fab71ea73ef]
[default0]: 9 /lib/x86_64-linux-gnu/libffi.so.8(ffi_call+0x12e) [0x7fab71eaa0be]
[default0]:10 /usr/lib/python3.12/lib-dynload/_ctypes.cpython-312-x86_64-linux-gnu.so(+0xe11c) [0x7fab71ffd11c]
[default0]:11 /usr/lib/python3.12/lib-dynload/_ctypes.cpython-312-x86_64-linux-gnu.so(+0x92af) [0x7fab71ff82af]
[default0]:12 /opt/venv/bin/python3(_PyObject_MakeTpCall+0x75) [0x549185]
[default0]:13 /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa89) [0x5d73c9]
[default0]:14 /opt/venv/bin/python3(PyEval_EvalCode+0x15b) [0x5d58eb]
[default0]:15 /opt/venv/bin/python3() [0x608b42]
[default0]:16 /opt/venv/bin/python3() [0x6b4e93]
[default0]:17 /opt/venv/bin/python3(_PyRun_SimpleFileObject+0x1aa) [0x6b4bfa]
[default0]:18 /opt/venv/bin/python3(_PyRun_AnyFileObject+0x4f) [0x6b4a2f]
[default0]:19 /opt/venv/bin/python3(Py_RunMain+0x3b5) [0x6bca95]
[default0]:20 /opt/venv/bin/python3(Py_BytesMain+0x2d) [0x6bc57d]
[default0]:21 /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7fab725621ca]
[default0]:22 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7fab7256228b]
[default0]:23 /opt/venv/bin/python3(_start+0x25) [0x657ce5]
[default0]:=================================
[default7]:[megatron_cicd:1617 :0:1617] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
[default7]:==== backtrace (tid: 1617) ====
[default7]: 0 /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7f2d8ddc6654]
[default7]: 1 /opt/hpcx/ucx/lib/libucs.so.0(+0x3684c) [0x7f2d8ddc684c]
[default7]: 2 /opt/hpcx/ucx/lib/libucs.so.0(+0x36a88) [0x7f2d8ddc6a88]
[default7]: 3 /lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x7f2d8f8f3330]
[default7]: 4 /lib/x86_64-linux-gnu/libnvidia-ml.so.1(+0x7922e) [0x7f2c68df922e]
[default7]: 5 /lib/x86_64-linux-gnu/libnvidia-ml.so.1(+0x7f056) [0x7f2c68dff056]
[default7]: 6 /lib/x86_64-linux-gnu/libnvidia-ml.so.1(nvmlDeviceGetTotalEnergyConsumption+0x5b) [0x7f2c68dff1cb]
[default7]: 7 /lib/x86_64-linux-gnu/libffi.so.8(+0x7b16) [0x7f2d8f220b16]
[default7]: 8 /lib/x86_64-linux-gnu/libffi.so.8(+0x43ef) [0x7f2d8f21d3ef]
[default7]: 9 /lib/x86_64-linux-gnu/libffi.so.8(ffi_call+0x12e) [0x7f2d8f2200be]
[default7]:10 /usr/lib/python3.12/lib-dynload/_ctypes.cpython-312-x86_64-linux-gnu.so(+0xe11c) [0x7f2d8f37311c]
[default7]:11 /usr/lib/python3.12/lib-dynload/_ctypes.cpython-312-x86_64-linux-gnu.so(+0x92af) [0x7f2d8f36e2af]
[default7]:12 /opt/venv/bin/python3(_PyObject_MakeTpCall+0x75) [0x549185]
[default7]:13 /opt/venv/bin/python3(_PyEval_EvalFrameDefault+0xa89) [0x5d73c9]
[default7]:14 /opt/venv/bin/python3(PyEval_EvalCode+0x15b) [0x5d58eb]
[default7]:15 /opt/venv/bin/python3() [0x608b42]
[default7]:16 /opt/venv/bin/python3() [0x6b4e93]
[default7]:17 /opt/venv/bin/python3(_PyRun_SimpleFileObject+0x1aa) [0x6b4bfa]
[default7]:18 /opt/venv/bin/python3(_PyRun_AnyFileObject+0x4f) [0x6b4a2f]
[default7]:19 /opt/venv/bin/python3(Py_RunMain+0x3b5) [0x6bca95]
[default7]:20 /opt/venv/bin/python3(Py_BytesMain+0x2d) [0x6bc57d]
[default7]:21 /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7f2d8f8d81ca]
[default7]:22 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7f2d8f8d828b]
[default7]:23 /opt/venv/bin/python3(_start+0x25) [0x657ce5]
[default7]:=================================
[default7]: [2026-07-14 07:07:42.771017] iteration 10/ 10 | consumed samples: 80 | elapsed time per iteration (ms): 245.1 | learning rate: 1.000000E-06 | global batch size: 8 | lm loss: 1.141397E+01 | loss scale: 1.0 | grad norm: 33.223 | num zeros: 17020.0 | params norm: 286.003 | number of skipped iterations: 0 | number of nan iterations: 0 |
W0714 07:07:43.371000 1544 torch/distributed/elastic/multiprocessing/api.py:898] Sending process 1610 closing signal SIGTERM
W0714 07:07:43.377000 1544 torch/distributed/elastic/multiprocessing/api.py:898] Sending process 1611 closing signal SIGTERM
W0714 07:07:43.381000 1544 torch/distributed/elastic/multiprocessing/api.py:898] Sending process 1612 closing signal SIGTERM
W0714 07:07:43.386000 1544 torch/distributed/elastic/multiprocessing/api.py:898] Sending process 1614 closing signal SIGTERM
W0714 07:07:43.391000 1544 torch/distributed/elastic/multiprocessing/api.py:898] Sending process 1615 closing signal SIGTERM
W0714 07:07:43.398000 1544 torch/distributed/elastic/multiprocessing/api.py:898] Sending process 1616 closing signal SIGTERM
W0714 07:07:43.407000 1544 torch/distributed/elastic/multiprocessing/api.py:898] Sending process 1617 closing signal SIGTERM
E0714 07:07:44.653000 1544 torch/distributed/elastic/multiprocessing/api.py:870] failed (exitcode: -11) local_rank: 3 (pid: 1613) of binary: /opt/venv/bin/python3
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 893, in
main()
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 889, in main
run(args)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 880, in run
elastic_launch(
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 139, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: