fix(rocjpeg): bind VA-API/VCN decode to the requested GPU by PCI BDF#8404
Conversation
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
There was a problem hiding this comment.
Pull request overview
This PR fixes rocJPEG’s VA-API/VCN render-node selection so hardware JPEG decode binds to the GPU requested by device_id (instead of silently falling back to renderD128 / GPU0 on certain architectures such as gfx950/MI355X). It does this by matching the HIP device to the DRM render node using the PCI bus ID (BDF), which is stable across HIP and sysfs.
Changes:
- Extend
RocJpegVappiDecoder::InitializeDecoderto accept a GPU PCI BDF string and preferentially select the DRM render node via BDF→render-node mapping. - Populate new BDF→render-node and BDF→compute-partition maps while scanning
/dev/dri/renderD*and corresponding/sys/class/drm/...entries. - Replace the hard-coded
renderD128fallback with “first available render node” fallback (withrenderD128as a final fallback if none found).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| projects/rocjpeg/src/rocjpeg_vaapi_decoder.h | Updates decoder init signature/docs and adds BDF-based mapping members + helper declarations. |
| projects/rocjpeg/src/rocjpeg_vaapi_decoder.cpp | Implements BDF-based render-node selection, adds helper functions for bus-id detection and fallback node selection, and logs selected mapping. |
| projects/rocjpeg/src/rocjpeg_decoder.cpp | Fetches the HIP device PCI BDF via hipDeviceGetPCIBusId and passes it into the VA-API decoder init. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
AryanSalmanpour
left a comment
There was a problem hiding this comment.
Hi @Xinyu302, Thank you for creating this PR and for your contribution. We appreciate your help.
I reviewed the changes and found an issue that causes the implementation to work only in the default partition mode (SPX) on MI3XX GPUs. In non-SPX partition modes, such as DPX, QPX, and CPX, the render node lookup can fail and incorrectly fall back to the default renderD128 node.
I have explained the root cause in my review comments below, along with a proposed fix that ensures the change works correctly across all MI3XX partition modes. Please take a look at the suggestions and let me know if anything is unclear or if you would like to discuss the details further.
9753732 to
dd7c48f
Compare
rrawther
left a comment
There was a problem hiding this comment.
Can be merged after resolving all the comments
dd7c48f to
9e2d215
Compare
Match the render node by PCI BDF (hipDeviceGetPCIBusId <-> /sys/class/drm/renderD*/device), keep unique_id as a fallback, and use the first available node instead of hard-coding renderD128. The BDF function suffix is stripped so matching also works across MI3XX non-SPX partition modes. Addresses review: strip .function in both places; use <stdlib.h> + realpath(path,nullptr); const-ref params; rename gpu_bdf -> gpu_pci_bdf. Fixes the wrong-render-node behavior from ROCm#7442 (error-out in ROCm#7349 was reverted in ROCm#7444).
9e2d215 to
2be414f
Compare
|
Thanks @AryanSalmanpour and @rrawther for the review. Pushed an update addressing all comments:
Rebuilt rocJPEG 1.6.0 from this branch and verified on 8x MI355X (gfx950): a decode requested on |
Summary
On architectures where the HIP device UUID (
hipDeviceProp_t.uuid) and the amdgpu sysfsunique_idcome from different namespaces (observed on gfx950 / MI355X),RocJpegVappiDecoder::InitializeDecodernever finds a match ingpu_uuids_to_render_nodes_map_(the map is keyed byunique_idbut looked up by the HIP UUID) and hard-falls-back torenderD128. As a result everyrocJpegCreate(HARDWARE, device_id), for anydevice_id, binds the VCN/JPEG decoder torenderD128= GPU 0 — all hardware JPEG decode is funneled onto a single GPU.Root cause
rocjpeg_decoder.cppbuilds the lookup key from the HIP UUID:std::string gpu_uuid(hip_dev_prop_.uuid.bytes, ...).rocjpeg_vaapi_decoder.cppGetGpuUuids()keys the map by the sysfsunique_id, andInitializeDecoder()doesrender_node_id = map.count(gpu_uuid) ? map[gpu_uuid] : 128.renderD128(GPU 0).Fix
Select the DRM render node by matching the device's PCI bus ID (BDF) via
hipDeviceGetPCIBusId↔/sys/class/drm/renderD*/device, which is consistent between the HIP runtime and the amdgpu sysfs nodes. Theunique_idmatch is kept as a fallback (no behavior change where it already works, e.g. gfx942), and when nothing matches the first available render node is used instead of hard-codingrenderD128(this also lets single-GPU containers that expose a non-default render node work).Relationship to prior work
This is the proper resolution of the issue #7349 tried to address by erroring out (
Error on GPU UUID mismatch instead of silently using wrong render node), which was reverted in #7444 (issue #7442) because erroring brokerocJpegCreatefor affected users. Matching by BDF binds to the correct GPU without erroring — it neither silently uses the wrong node nor fails initialization.Testing (8x AMD Instinct MI355X, gfx950, ROCm 7.2)
Built the
projects/rocjpeglibrary (rocJPEG 1.6.0) from this branch and validated with per-GPUamdgpu_fence_infojpeg_dec_*counters (co-tenant-immune):cuda:0/cuda:4/cuda:7→ JPEG-ring activity all on GPU 0.cuda:0→ GPU 0,cuda:4→ GPU 4,cuda:7→ GPU 7; 8 concurrent decoders spread across all 8 GPUs.cuda:0) behavior unchanged.Notes
InitializeDecoderis the internal VA-API decoder method).