Skip to content

fix(amdsmi): process visibility, per-GPU attribution, table alignment#8396

Merged
marifamd merged 6 commits into
developfrom
fix-amdsmi-process-visibility
Jul 16, 2026
Merged

fix(amdsmi): process visibility, per-GPU attribution, table alignment#8396
marifamd merged 6 commits into
developfrom
fix-amdsmi-process-visibility

Conversation

@marifamd

@marifamd marifamd commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Motivation

Three defects in amd-smi process (and the process table in default amd-smi
output):

  1. It could return "No running processes detected" for a caller that lacks
    permission to inspect the running GPU jobs, even though the jobs were clearly
    using the GPUs.
  2. The same compute process was reported under every GPU, rather than only
    the GPU(s) it actually uses.
  3. The compute-process table mis-rendered its CU % and SDMA columns.

Technical Details

  • rocm_smi/src/rocm_smi_kfd.cc

    • IsKfdPidNamespaced() inspected the first KFD PID via PidHasKfdOpen(),
      which opens /proc/<pid>/fd. When that PID belongs to another user, the
      directory is unreadable and the helper returned false, which was
      misinterpreted as evidence of a separate PID namespace. Enumeration then
      fell back to scanning /proc, where the same unreadable fd set dropped the
      process entirely, so the caller saw an empty list.
    • Add a tri-state PidKfdOpenState() distinguishing "has /dev/kfd open",
      "definitely does not", and "inaccessible" (another user's process, or the
      process already exited). Permission failures are now treated as inconclusive
      and skipped rather than flagged as namespaced, so such processes are listed
      with a redacted (N/A) name instead of disappearing.
    • GetProcessGPUs() ran the per-node KFD context-file fallback
      (GetKfdGpuIdsForPid()) unconditionally. Opening /dev/kfd creates
      vram_/stats_/counters_/sdma_<gpuid> files for the whole GPU topology, so a
      job with queues on a single GPU (or none) was attributed to every GPU.
      Attribute a process to a GPU only when it has an active KFD queue on it, or
      a non-zero vram_<gpuid> allocation on it. The zero-valued
      stats_/counters_/sdma_ per-node files are ignored. This lists a process
      against the GPUs it actually uses, aligning with rocm-smi --showpids. The
      container (PID-namespace) vram_* fallback is unchanged.
  • amdsmi_cli/amdsmi_logger.py

    • Rebuild the process-table header so labels sit over their right-justified
      value columns (the SDMA header was one column too far left), drop the
      redundant % suffix from the CU value (the unit is in the CU % header),
      and widen SDMA to fit its unit by reclaiming two characters from the
      process-name column. The row stays within the fixed 80-character box. The
      row rendering is extracted into AMDSMILogger._format_process_row /
      PROCESS_TABLE_HEADER so the widths can be unit-tested.
  • tests/python/unit/gpu/

    • test_kfd_process_gpus.py: hardware-free spec pinning the queue + non-zero
      VRAM attribution (including that an unreadable vram_* entry is skipped).
    • test_process_table_format.py: hardware-free coverage for the process-table
      column widths (80-character box, dropped % suffix, header-over-value
      alignment).

JIRA ID

JIRA ID: AILITOOLS-283
JIRA ID: ROCM-26000

Test Plan

  • On multi-GPU hosts running a compute workload, ran amd-smi process and the
    default amd-smi process table as a non-root user and as root, with
    the installed library and with the fixed library, cross-checking against the
    ground-truth KFD queue gpuids in /sys/class/kfd/kfd/proc/<pid>/queues/ and
    the vram_<gpuid> allocations.
  • Verified column alignment for both the N/A and populated CU %/SDMA cases.
  • Ran the hardware-free unit tests.

Test Result

  • Non-root now lists the root-owned process (previously non-root saw "No running
    processes detected"), with the process name redacted to N/A.
  • Per-GPU attribution: on multi-GPU MI300 hosts, a job with queues + VRAM on a
    single GPU is now listed only on that GPU (confirmed against RVS on a gfx125X
    host: RVS with queues/VRAM on one GPU appeared under all 4 GPUs before the fix
    and only the correct GPU after, matching rocm-smi --showpids GPU(s)=1).
  • CU % and SDMA headers align over their values within the 80-character box;
    valid values are no longer truncated.
  • tests/python/unit/gpu/test_kfd_process_gpus.py and test_process_table_format.py
    pass.

@marifamd
marifamd requested a review from a team July 10, 2026 07:10
@therock-pr-bot

therock-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

✅ All Policy Checks Passed

Check Status Details
🌿 Branch Name ✅ Pass
📝 PR Title/Description ✅ Pass
Forbidden Files ✅ Pass
🧪 Unit Test ✅ Pass
🚫 Draft PR 🔜 To Be Enabled
🚩 Feature Flag 🔜 To Be Enabled
📊 Code Coverage 🔜 To Be Enabled

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot

therock-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ PR Title/Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

@marifamd
marifamd force-pushed the fix-amdsmi-process-visibility branch 4 times, most recently from f10b509 to e8d0054 Compare July 13, 2026 18:46
@marifamd
marifamd force-pushed the fix-amdsmi-process-visibility branch 2 times, most recently from 42d6001 to 2842f82 Compare July 15, 2026 21:23
@marifamd marifamd changed the title fix(amdsmi): list other users' compute processes and align process table fix(amdsmi): process visibility, per-GPU attribution, and process table alignment Jul 15, 2026
@marifamd
marifamd force-pushed the fix-amdsmi-process-visibility branch from e33a73f to 6705416 Compare July 16, 2026 00:45
@marifamd marifamd changed the title fix(amdsmi): process visibility, per-GPU attribution, and process table alignment fix(amdsmi): process visibility, per-GPU attribution, table alignment Jul 16, 2026
marifamd added 6 commits July 15, 2026 22:53
…error

IsKfdPidNamespaced() checked the first KFD PID via PidHasKfdOpen(), which
opened /proc/<pid>/fd. When that PID belongs to another user (e.g. a
root-owned GPU job seen by a non-root caller), the directory is unreadable
and PidHasKfdOpen() returned false, which was misread as evidence of a
different PID namespace. The enumeration then fell back to scanning /proc,
where the same unreadable fd set caused the process to be dropped, so a
non-root caller saw no running processes at all.

Distinguish "cannot determine" (fd dir unreadable: another user's process
or already exited) from "definitely has no /dev/kfd open" via a tri-state
PidKfdOpenState(). Permission failures are now inconclusive and skipped
rather than flagged as namespaced, so processes owned by other users are
listed with N/A name instead of vanishing.

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
The process table header placed the SDMA label one column to the left of
its value slot, and the CU%/SDMA value fields were too narrow: a valid
"100.0 %" or a multi-digit SDMA time was truncated, and the N/A rows did
not line up under their headers.

Rebuild the header right-justified over the value columns, drop the
redundant "%" suffix from the CU value (the unit is in the header), and
widen SDMA to fit its unit by reclaiming two characters from the process
name column. The row stays within the fixed 80-character box.

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
GetProcessGPUs() attributed a process to a GPU whenever the process had
a KFD context on it. Opening /dev/kfd creates per-node
vram_/stats_/counters_/sdma_<gpuid> files for the whole GPU topology, so
a job with queues on a single GPU (or none) was listed under every GPU.

Attribute a process to a GPU only when it has an active KFD queue on it
or a non-zero VRAM allocation on it (vram_<gpuid>). The zero-valued
stats/counters/sdma per-node files are ignored. This lists a process
against the GPUs it actually uses, aligning with rocm-smi --showpids.
The container (PID-namespace) vram_* fallback is unchanged.

Adds a hardware-free unit test pinning the queue+VRAM attribution.

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
…ests

Extract the default-output process row into AMDSMILogger._format_process_row
and a shared PROCESS_TABLE_HEADER so the CU%/SDMA column widths can be pinned
by a hardware-free unit test (test_process_table_format.py), which asserts the
80-character box, the dropped '%' suffix, and header-over-value alignment.

Add a vram-unreadable case to the KFD attribution spec so the ReadSysfsStr
failure -> skip path is covered, and document that the other-user visibility
(namespace) path is hardware-validated rather than reproduced here.

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
…aring

Condense the IsKfdPidNamespaced case list to the non-obvious inconclusive
outcomes and drop the inline restatement of the kNoKfd branch. Note on the
GetKfdGpuIdsForPid declaration that it replaces (clears) the output set.

Signed-off-by: Maisam Arif <Maisam.Arif@amd.com>
@marifamd
marifamd force-pushed the fix-amdsmi-process-visibility branch from 6705416 to 68f4b21 Compare July 16, 2026 03:53
@marifamd
marifamd merged commit d7fc3d2 into develop Jul 16, 2026
38 of 40 checks passed
@marifamd
marifamd deleted the fix-amdsmi-process-visibility branch July 16, 2026 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants