Skip to content

fix(rocjpeg): bind VA-API/VCN decode to the requested GPU by PCI BDF#8404

Merged
AryanSalmanpour merged 1 commit into
ROCm:developfrom
Xinyu302:fix/rocjpeg-render-node-bdf
Jul 14, 2026
Merged

fix(rocjpeg): bind VA-API/VCN decode to the requested GPU by PCI BDF#8404
AryanSalmanpour merged 1 commit into
ROCm:developfrom
Xinyu302:fix/rocjpeg-render-node-bdf

Conversation

@Xinyu302

Copy link
Copy Markdown
Contributor

Summary

On architectures where the HIP device UUID (hipDeviceProp_t.uuid) and the amdgpu sysfs unique_id come from different namespaces (observed on gfx950 / MI355X), RocJpegVappiDecoder::InitializeDecoder never finds a match in gpu_uuids_to_render_nodes_map_ (the map is keyed by unique_id but looked up by the HIP UUID) and hard-falls-back to renderD128. As a result every rocJpegCreate(HARDWARE, device_id), for any device_id, binds the VCN/JPEG decoder to renderD128 = GPU 0 — all hardware JPEG decode is funneled onto a single GPU.

Root cause

  • rocjpeg_decoder.cpp builds the lookup key from the HIP UUID: std::string gpu_uuid(hip_dev_prop_.uuid.bytes, ...).
  • rocjpeg_vaapi_decoder.cpp GetGpuUuids() keys the map by the sysfs unique_id, and InitializeDecoder() does render_node_id = map.count(gpu_uuid) ? map[gpu_uuid] : 128.
  • On gfx950 the two ID namespaces are disjoint, so the lookup always misses → renderD128 (GPU 0).

Fix

Select the DRM render node by matching the device's PCI bus ID (BDF) via hipDeviceGetPCIBusId/sys/class/drm/renderD*/device, which is consistent between the HIP runtime and the amdgpu sysfs nodes. The unique_id match is kept as a fallback (no behavior change where it already works, e.g. gfx942), and when nothing matches the first available render node is used instead of hard-coding renderD128 (this also lets single-GPU containers that expose a non-default render node work).

Relationship to prior work

This is the proper resolution of the issue #7349 tried to address by erroring out (Error on GPU UUID mismatch instead of silently using wrong render node), which was reverted in #7444 (issue #7442) because erroring broke rocJpegCreate for affected users. Matching by BDF binds to the correct GPU without erroring — it neither silently uses the wrong node nor fails initialization.

Testing (8x AMD Instinct MI355X, gfx950, ROCm 7.2)

Built the projects/rocjpeg library (rocJPEG 1.6.0) from this branch and validated with per-GPU amdgpu_fence_info jpeg_dec_* counters (co-tenant-immune):

  • Before (unmodified develop): decode requested on cuda:0 / cuda:4 / cuda:7 → JPEG-ring activity all on GPU 0.
  • After (this change): cuda:0 → GPU 0, cuda:4 → GPU 4, cuda:7 → GPU 7; 8 concurrent decoders spread across all 8 GPUs.
  • Output is byte-identical across devices and matches the CPU reference; single-GPU (cuda:0) behavior unchanged.

Notes

  • No public API/ABI change (the modified InitializeDecoder is the internal VA-API decoder method).
  • The same fix applies to rocDecode, which shares this render-node selection pattern.

@therock-pr-bot

therock-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
🌿 Branch Name ❌ Fail Branch name does not match allowed patterns.
Branch: fix/rocjpeg-render-node-bdf
Allowed patterns:
- ^users\/[A-Za-z0-9][A-Za-z0-9\-]*\/.+
- ^shared\/.+
- ^[A-Za-z0-9][A-Za-z0-9\-_]*$
- ^dependabot\/.+
- ^revert-[0-9]+-.+
📝 PR Title/Description ✅ Pass
Forbidden Files ✅ Pass
🧪 Unit Test ❌ Fail Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/rocjpeg/src/rocjpeg_decoder.cpp, projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp, projects/rocjpeg/src/rocjpeg_vaapi_decoder.h; no test file found
🔎 pre-commit ⏳ Pending ⏳ Still running…
🚫 Draft PR 🔜 To Be Enabled
🚩 Feature Flag 🔜 To Be Enabled
📊 Code Coverage 🔜 To Be Enabled

⚠️ 2 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ Branch Name
  • ❌ Unit Test

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot

therock-pr-bot Bot commented Jul 10, 2026

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ Branch Name
  • ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes rocJPEG’s VA-API/VCN render-node selection so hardware JPEG decode binds to the GPU requested by device_id (instead of silently falling back to renderD128 / GPU0 on certain architectures such as gfx950/MI355X). It does this by matching the HIP device to the DRM render node using the PCI bus ID (BDF), which is stable across HIP and sysfs.

Changes:

  • Extend RocJpegVappiDecoder::InitializeDecoder to accept a GPU PCI BDF string and preferentially select the DRM render node via BDF→render-node mapping.
  • Populate new BDF→render-node and BDF→compute-partition maps while scanning /dev/dri/renderD* and corresponding /sys/class/drm/... entries.
  • Replace the hard-coded renderD128 fallback with “first available render node” fallback (with renderD128 as a final fallback if none found).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
projects/rocjpeg/src/rocjpeg_vaapi_decoder.h Updates decoder init signature/docs and adds BDF-based mapping members + helper declarations.
projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp Implements BDF-based render-node selection, adds helper functions for bus-id detection and fallback node selection, and logs selected mapping.
projects/rocjpeg/src/rocjpeg_decoder.cpp Fetches the HIP device PCI BDF via hipDeviceGetPCIBusId and passes it into the VA-API decoder init.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp
Comment thread projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp
Comment thread projects/rocjpeg/src/rocjpeg_vaapi_decoder.h Outdated
Comment thread projects/rocjpeg/src/rocjpeg_vaapi_decoder.h Outdated
Comment thread projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp Outdated

@AryanSalmanpour AryanSalmanpour left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Xinyu302, Thank you for creating this PR and for your contribution. We appreciate your help.

I reviewed the changes and found an issue that causes the implementation to work only in the default partition mode (SPX) on MI3XX GPUs. In non-SPX partition modes, such as DPX, QPX, and CPX, the render node lookup can fail and incorrectly fall back to the default renderD128 node.

I have explained the root cause in my review comments below, along with a proposed fix that ensures the change works correctly across all MI3XX partition modes. Please take a look at the suggestions and let me know if anything is unclear or if you would like to discuss the details further.

Comment thread projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp
Comment thread projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp
@Xinyu302
Xinyu302 force-pushed the fix/rocjpeg-render-node-bdf branch 2 times, most recently from 9753732 to dd7c48f Compare July 13, 2026 09:04
Comment thread projects/rocjpeg/src/rocjpeg_decoder.cpp Outdated

@rrawther rrawther left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be merged after resolving all the comments

@Xinyu302
Xinyu302 force-pushed the fix/rocjpeg-render-node-bdf branch from dd7c48f to 9e2d215 Compare July 14, 2026 02:01
Match the render node by PCI BDF (hipDeviceGetPCIBusId <-> /sys/class/drm/renderD*/device),
keep unique_id as a fallback, and use the first available node instead of hard-coding renderD128.
The BDF function suffix is stripped so matching also works across MI3XX non-SPX partition modes.

Addresses review: strip .function in both places; use <stdlib.h> + realpath(path,nullptr);
const-ref params; rename gpu_bdf -> gpu_pci_bdf.

Fixes the wrong-render-node behavior from ROCm#7442 (error-out in ROCm#7349 was reverted in ROCm#7444).
@Xinyu302 Xinyu302 changed the title [rocjpeg] Bind VA-API/VCN decode to the requested GPU by PCI BDF fix(rocjpeg): bind VA-API/VCN decode to the requested GPU by PCI BDF Jul 14, 2026
@Xinyu302
Xinyu302 force-pushed the fix/rocjpeg-render-node-bdf branch from 9e2d215 to 2be414f Compare July 14, 2026 02:12
@Xinyu302

Copy link
Copy Markdown
Contributor Author

Thanks @AryanSalmanpour and @rrawther for the review. Pushed an update addressing all comments:

  • Partition modes (@AryanSalmanpour): strip the PCI .function suffix in both GetRenderNodeBusId and InitializeDecoder, so BDF matching also works in MI3XX non-SPX modes (DPX/QPX/CPX); the specific partition node is selected by GetDrmNodeOffset.
  • @rrawther: switched to <stdlib.h> for realpath, and renamed gpu_bdf -> gpu_pci_bdf.
  • Copilot: use realpath(path, nullptr) + free() (no more PATH_MAX); made device_name/gpu_uuid/gpu_pci_bdf const std::string&; the gpu_uuid doc no longer implies the sysfs unique_id namespace.

Rebuilt rocJPEG 1.6.0 from this branch and verified on 8x MI355X (gfx950): a decode requested on cuda:N now uses GPU N's VCN/JPEG engine (per-GPU amdgpu_fence_info jpeg_dec counters), vs all landing on GPU 0 before; decoded output is unchanged.

@AryanSalmanpour
AryanSalmanpour merged commit 544c3c0 into ROCm:develop Jul 14, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants