feat(rocr): enforce required gfx1250 hotswap rewrites by harsh-amd · Pull Request #8387 · ROCm/rocm-systems

harsh-amd · 2026-07-10T01:12:03Z

Summary

Add HSA_HOTSWAP_STRICT_MODE, default off, to request COMGR AMD_COMGR_HOTSWAP_REWRITE_FLAG_STRICT_MODE for non-A0 gfx1250 HotSwap rewrites.
Keep A0 retargeting outside ROCR strict mode; A0 continues to use the normal B0-to-A0 hotswap path and does not set the COMGR strict flag.
Treat required rewrites as non-fallback operations: A0 B0-to-A0 retargeting and opt-in B0 strict rewrites return HSA_STATUS_ERROR_INVALID_CODE_OBJECT if COMGR cannot produce the rewrite or the rewritten code object cannot be loaded.
Keep entry-trampoline selection independent from strict-mode mask workarounds, so AMD_COMGR_HOTSWAP_ENTRY_TRAMPOLINES=0 only disables entry trampolines.
Preserve the original fallback behavior for optional rewrites and no-decision paths.
Document HSA_HOTSWAP_STRICT_MODE in the runtime environment variable table.
Depends on [comgr][hotswap] apply gfx1250 mask workarounds llvm-project#3312 for COMGR B0 strict mask workaround support.

JIRA ID

JIRA ID : ROCM-27304
JIRA ID : PLAT-204339

Testing

cmake --build build-rocrtst-hotswap-review --target hotswap_rewrite -j 8
ctest --test-dir build-rocrtst-hotswap-review -R hotswap_rewrite --output-on-failure
Result: hotswap_rewrite 30/30 passed.

therock-pr-bot · 2026-07-10T01:12:41Z

✅ All Policy Checks Passed

Check	Status	Details
🌿 Branch Name	✅ Pass	—
📝 PR Title/Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

therock-pr-bot · 2026-07-10T01:12:43Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Title/Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

shwetagkhatri

New tests (HotswapRewriteDecision., HotswapRewrite.RuntimeLoadRequired*) are not registered in rocrtst/suites/test_common/test_categories.yaml. Please take a note before adding any tests to rocrtsts next time.

powderluv · 2026-07-12T20:57:45Z

Do you want the strict mode to work even in the B0 -> A0 path ?

harsh-amd · 2026-07-12T21:02:49Z

These tests are built as the standalone hotswap_rewrite target and already have the quick, standard, pr, and pre-commit CTest labels. Since test_categories.yaml currently only applies to rocrtst64, adding these test names there would not affect the standalone target. Are the existing CTest labels sufficient, or should standalone targets also be added to the YAML?

harsh-amd · 2026-07-12T21:14:41Z

Do you want the strict mode to work even in the B0 -> A0 path ?

So far we haven't found a need for this. The D# WA is always required on A0, while it can be avoided on B0. Hence currently we only have strict mode for B0 -> B0.

shwetagkhatri · 2026-07-12T22:12:32Z

These tests are built as the standalone hotswap_rewrite target and already have the quick, standard, pr, and pre-commit CTest labels. Since test_categories.yaml currently only applies to rocrtst64, adding these test names there would not affect the standalone target. Are the existing CTest labels sufficient, or should standalone targets also be added to the YAML?

I see it now. Yes the existing CTest labels are the correct for standalone tests.

shwetagkhatri

LGTM.

cfreeamd

🤖 AI-assisted pre-review (for @cfreeamd)

The ROCr files look correct overall — hotswap.hpp, hotswap.cpp, and env_variables.rst all look consistent with the new rewrite_required enforcement semantics.

## Motivation #8387 made gfx1250 B0-to-A0 HotSwap rewrites mandatory. COMGR still rejects several valid production code objects, so those pre-existing rewrite failures changed from loading the original object to HSA_STATUS_ERROR_INVALID_CODE_OBJECT. The Jul-17 quick suite consequently regressed in hiptensor, rocPRIM, rocThrust, hipSPARSE, rocSPARSE, rocSOLVER, and hipBLASLt. A controlled A/B used the same Jul-18 compiler, libraries, tests, GPU, and ROCr revision, changing only the A0 rewrite-required policy. The failing COMGR return codes remained identical, but restoring optional fallback made the affected tests execute successfully. ## Technical Details - Keep legacy gfx1250 B0-to-A0 retargeting best-effort when COMGR cannot emit a rewrite. - Fall back to the original code object if an optional A0 rewrite fails or if its rewritten object fails to load. - Preserve fail-closed behavior for opt-in non-A0 strict-mode rewrites. - Update decision and runtime-load tests for both A0 failure paths. This restores the behavior before commit 8b9a6bf. Mandatory A0 enforcement can be reconsidered after COMGR supports the production code-object population or exposes a result that distinguishes a proven-safe no-rewrite case from an unsupported required rewrite. ## Issue Tracking JIRA ID: ROCM-28260 ## Test Plan - Build libhsa-runtime64 from current develop with this change. - Build and run hotswap_gfx_query and hotswap_rewrite. - Run the original affected quick-suite matrix on mi400 GPU 2 with AMD_COMGR_HOTSWAP_ENTRY_TRAMPOLINES=1. - Verify the exact production cases that previously terminated on COMGR rewrite failure. ## Test Result - ROCr HotSwap tests: 2/2 passed. - hiptensor: 108/108 CTest entries passed. - rocPRIM: 172/172 CTest entries passed. - rocThrust: 336/336 CTest entries passed. - hipSPARSE: 4/4 CTest entries passed. - rocSPARSE: three shorter CTest entries passed; the native quick executable passed 18,124/18,124 tests in 318.9 seconds. Its existing CTest entry has a 300-second timeout. - rocSOLVER: ffm-quick passed; the native quick executable passed 6,188/6,188 tests in 331.3 seconds. Its existing CTest entry has a 300-second timeout. - hipBLASLt: 1/1 quick entry passed, with zero out-of-range register diagnostics. - No GPU fault or hang occurred. The rocSPARSE and rocSOLVER timeout budgets are separate harness issues exposed because the formerly crashing cases now complete. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

github-actions Bot added the project: rocr-runtime label Jul 10, 2026

therock-pr-bot Bot added the Not ready to Review label Jul 10, 2026

harsh-amd force-pushed the users/harsh-amd/plat-204339-hotswap-strict-mode branch from e0b638c to 06b41ab Compare July 10, 2026 01:14

harsh-amd changed the title ~~ROCR: enable strict COMGR hotswap workarounds~~ ROCR: require strict COMGR hotswap workarounds Jul 10, 2026

systems-assistant Bot added the organization: ROCm label Jul 10, 2026

github-actions Bot added the documentation Improvements or additions to documentation label Jul 10, 2026

harsh-amd changed the title ~~ROCR: require strict COMGR hotswap workarounds~~ ROCR: add opt-in B0 strict COMGR hotswap mode Jul 10, 2026

harsh-amd changed the title ~~ROCR: add opt-in B0 strict COMGR hotswap mode~~ feat(rocr): add opt-in B0 strict COMGR hotswap mode Jul 10, 2026

harsh-amd force-pushed the users/harsh-amd/plat-204339-hotswap-strict-mode branch from e0dbc89 to 6102171 Compare July 10, 2026 18:33

harsh-amd marked this pull request as ready for review July 10, 2026 18:33

harsh-amd requested review from a team, atgutier, cfreeamd, dayatsin-amd, kentrussell and shwetagkhatri as code owners July 10, 2026 18:33

harsh-amd changed the title ~~feat(rocr): add opt-in B0 strict COMGR hotswap mode~~ feat(rocr): enforce required gfx1250 hotswap rewrites Jul 10, 2026

shwetagkhatri requested changes Jul 12, 2026

View reviewed changes

Comment thread projects/rocr-runtime/runtime/hsa-runtime/core/runtime/hotswap.cpp

Comment thread projects/rocr-runtime/runtime/hsa-runtime/core/runtime/hotswap.cpp Outdated

harsh-amd removed the Not ready to Review label Jul 12, 2026

shwetagkhatri self-requested a review July 12, 2026 22:13

shwetagkhatri approved these changes Jul 12, 2026

View reviewed changes

dayatsin-amd approved these changes Jul 13, 2026

View reviewed changes

harsh-amd added 2 commits July 13, 2026 06:39

ROCR: enable strict COMGR hotswap workarounds

360b0e8

ROCR: fail required hotswap rewrites instead of falling back

04d2c3d

harsh-amd added 6 commits July 13, 2026 06:39

ROCR: make hotswap strict mode opt-in for B0

b717a18

ROCR: probe COMGR strict hotswap support in tests

1383607

ROCR: keep non-A0 gfx1250 hotswap on B0 stepping

dfb7144

ROCR: require A0 hotswap rewrite success

8b9a6bf

Address hotswap review feedback

31d9663

Reconcile hotswap options with latest develop

16e4fdc

harsh-amd force-pushed the users/harsh-amd/plat-204339-hotswap-strict-mode branch from 2e2f418 to 16e4fdc Compare July 13, 2026 13:42

cfreeamd reviewed Jul 13, 2026

View reviewed changes

Comment thread projects/rocr-runtime/runtime/hsa-runtime/core/runtime/hotswap.cpp

harsh-amd mentioned this pull request Jul 13, 2026

ROCR: clean up hotswap test failure branch harsh-amd/rocm-systems#3

Closed

harsh-amd merged commit 1755df8 into ROCm:develop Jul 14, 2026
35 of 41 checks passed

This was referenced Jul 14, 2026

refactor(rocr): clean up hotswap test failure branch #8522

Closed

fix(rocr): restore optional A0 HotSwap fallback #8822

Merged

Navigation Menu

feat(rocr): enforce required gfx1250 hotswap rewrites - #8387

feat(rocr): enforce required gfx1250 hotswap rewrites#8387
harsh-amd merged 8 commits into
ROCm:developfrom
harsh-amd:users/harsh-amd/plat-204339-hotswap-strict-mode

harsh-amd commented Jul 10, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Jul 10, 2026

Uh oh!

shwetagkhatri left a comment

Uh oh!

Uh oh!

Uh oh!

powderluv commented Jul 12, 2026

Uh oh!

harsh-amd commented Jul 12, 2026

Uh oh!

harsh-amd commented Jul 12, 2026

Uh oh!

shwetagkhatri commented Jul 12, 2026

Uh oh!

shwetagkhatri left a comment

Uh oh!

cfreeamd left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

harsh-amd commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

JIRA ID

Testing

Uh oh!

therock-pr-bot Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Jul 10, 2026

Uh oh!

shwetagkhatri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

powderluv commented Jul 12, 2026

Uh oh!

harsh-amd commented Jul 12, 2026

Uh oh!

harsh-amd commented Jul 12, 2026

Uh oh!

shwetagkhatri commented Jul 12, 2026

Uh oh!

shwetagkhatri left a comment

Choose a reason for hiding this comment

Uh oh!

cfreeamd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

harsh-amd commented Jul 10, 2026 •

edited

Loading

therock-pr-bot Bot commented Jul 10, 2026 •

edited

Loading