fix(amdsmi): process visibility, per-GPU attribution, table alignment#8396
Merged
Conversation
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
marifamd
force-pushed
the
fix-amdsmi-process-visibility
branch
4 times, most recently
from
July 13, 2026 18:46
f10b509 to
e8d0054
Compare
adam360x
approved these changes
Jul 13, 2026
marifamd
force-pushed
the
fix-amdsmi-process-visibility
branch
2 times, most recently
from
July 15, 2026 21:23
42d6001 to
2842f82
Compare
gabrpham
approved these changes
Jul 15, 2026
marifamd
force-pushed
the
fix-amdsmi-process-visibility
branch
from
July 16, 2026 00:45
e33a73f to
6705416
Compare
…error IsKfdPidNamespaced() checked the first KFD PID via PidHasKfdOpen(), which opened /proc/<pid>/fd. When that PID belongs to another user (e.g. a root-owned GPU job seen by a non-root caller), the directory is unreadable and PidHasKfdOpen() returned false, which was misread as evidence of a different PID namespace. The enumeration then fell back to scanning /proc, where the same unreadable fd set caused the process to be dropped, so a non-root caller saw no running processes at all. Distinguish "cannot determine" (fd dir unreadable: another user's process or already exited) from "definitely has no /dev/kfd open" via a tri-state PidKfdOpenState(). Permission failures are now inconclusive and skipped rather than flagged as namespaced, so processes owned by other users are listed with N/A name instead of vanishing. Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
The process table header placed the SDMA label one column to the left of its value slot, and the CU%/SDMA value fields were too narrow: a valid "100.0 %" or a multi-digit SDMA time was truncated, and the N/A rows did not line up under their headers. Rebuild the header right-justified over the value columns, drop the redundant "%" suffix from the CU value (the unit is in the header), and widen SDMA to fit its unit by reclaiming two characters from the process name column. The row stays within the fixed 80-character box. Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
GetProcessGPUs() attributed a process to a GPU whenever the process had a KFD context on it. Opening /dev/kfd creates per-node vram_/stats_/counters_/sdma_<gpuid> files for the whole GPU topology, so a job with queues on a single GPU (or none) was listed under every GPU. Attribute a process to a GPU only when it has an active KFD queue on it or a non-zero VRAM allocation on it (vram_<gpuid>). The zero-valued stats/counters/sdma per-node files are ignored. This lists a process against the GPUs it actually uses, aligning with rocm-smi --showpids. The container (PID-namespace) vram_* fallback is unchanged. Adds a hardware-free unit test pinning the queue+VRAM attribution. Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
…ests Extract the default-output process row into AMDSMILogger._format_process_row and a shared PROCESS_TABLE_HEADER so the CU%/SDMA column widths can be pinned by a hardware-free unit test (test_process_table_format.py), which asserts the 80-character box, the dropped '%' suffix, and header-over-value alignment. Add a vram-unreadable case to the KFD attribution spec so the ReadSysfsStr failure -> skip path is covered, and document that the other-user visibility (namespace) path is hardware-validated rather than reproduced here. Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
…aring Condense the IsKfdPidNamespaced case list to the non-obvious inconclusive outcomes and drop the inline restatement of the kNoKfd branch. Note on the GetKfdGpuIdsForPid declaration that it replaces (clears) the output set. Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
marifamd
force-pushed
the
fix-amdsmi-process-visibility
branch
from
July 16, 2026 03:53
6705416 to
68f4b21
Compare
dmitrii-galantsev
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Three defects in
amd-smi process(and the process table in defaultamd-smioutput):
permission to inspect the running GPU jobs, even though the jobs were clearly
using the GPUs.
the GPU(s) it actually uses.
CU %andSDMAcolumns.Technical Details
rocm_smi/src/rocm_smi_kfd.ccIsKfdPidNamespaced()inspected the first KFD PID viaPidHasKfdOpen(),which opens
/proc/<pid>/fd. When that PID belongs to another user, thedirectory is unreadable and the helper returned
false, which wasmisinterpreted as evidence of a separate PID namespace. Enumeration then
fell back to scanning
/proc, where the same unreadable fd set dropped theprocess entirely, so the caller saw an empty list.
PidKfdOpenState()distinguishing "has/dev/kfdopen","definitely does not", and "inaccessible" (another user's process, or the
process already exited). Permission failures are now treated as inconclusive
and skipped rather than flagged as namespaced, so such processes are listed
with a redacted (
N/A) name instead of disappearing.GetProcessGPUs()ran the per-node KFD context-file fallback(
GetKfdGpuIdsForPid()) unconditionally. Opening/dev/kfdcreatesvram_/stats_/counters_/sdma_<gpuid>files for the whole GPU topology, so ajob with queues on a single GPU (or none) was attributed to every GPU.
Attribute a process to a GPU only when it has an active KFD queue on it, or
a non-zero
vram_<gpuid>allocation on it. The zero-valuedstats_/counters_/sdma_per-node files are ignored. This lists a processagainst the GPUs it actually uses, aligning with
rocm-smi --showpids. Thecontainer (PID-namespace)
vram_*fallback is unchanged.amdsmi_cli/amdsmi_logger.pyvalue columns (the
SDMAheader was one column too far left), drop theredundant
%suffix from the CU value (the unit is in theCU %header),and widen
SDMAto fit its unit by reclaiming two characters from theprocess-name column. The row stays within the fixed 80-character box. The
row rendering is extracted into
AMDSMILogger._format_process_row/PROCESS_TABLE_HEADERso the widths can be unit-tested.tests/python/unit/gpu/test_kfd_process_gpus.py: hardware-free spec pinning the queue + non-zeroVRAM attribution (including that an unreadable
vram_*entry is skipped).test_process_table_format.py: hardware-free coverage for the process-tablecolumn widths (80-character box, dropped
%suffix, header-over-valuealignment).
JIRA ID
JIRA ID: AILITOOLS-283
JIRA ID: ROCM-26000
Test Plan
amd-smi processand thedefault
amd-smiprocess table as a non-root user and as root, withthe installed library and with the fixed library, cross-checking against the
ground-truth KFD queue
gpuids in/sys/class/kfd/kfd/proc/<pid>/queues/andthe
vram_<gpuid>allocations.N/Aand populatedCU %/SDMAcases.Test Result
processes detected"), with the process name redacted to
N/A.single GPU is now listed only on that GPU (confirmed against RVS on a gfx125X
host: RVS with queues/VRAM on one GPU appeared under all 4 GPUs before the fix
and only the correct GPU after, matching
rocm-smi --showpidsGPU(s)=1).CU %andSDMAheaders align over their values within the 80-character box;valid values are no longer truncated.
tests/python/unit/gpu/test_kfd_process_gpus.pyandtest_process_table_format.pypass.