Repository navigation
OneTrainer Windows-ROCm profile installs bitsandbytes 0.50.0.dev0, which OneTrainer doesn't support yet — bnb 8-bit optimizers crash #1708
Description
Activity
side question - do you have a working AMD install for OneTrainer?
we've been struggling with finding one: Nerogar/OneTrainer#1559if you have, we could "downstream" it so you don't have to manage your own requirements for it but it's part of OneTrainer
OneTrainerWindowsRocmProfile.cscurrently installs bitsandbytes from0xDELUXA/bitsandbytes_win_rocmrelease0.50.0.dev0-py3.12-rocm7.15-win_amd64_all.OneTrainer pins
bitsandbytes==0.49.1and doesn't support 0.50 yet: 0.50 removed theblock_wise=andpercentile_clipping=arguments from all optimizer constructors, which OneTrainer still passes. Any training run using a bitsandbytes optimizer (AdamW8bit etc.) fails at startup:File "...\modules\util\create.py", line 251, in create_optimizer optimizer = bnb.optim.AdamW8bit( TypeError: AdamW8bit.__init__() got an unexpected keyword argument 'percentile_clipping'Details on the API break and the planned OneTrainer-side upgrade are here: Nerogar/OneTrainer#1519 (comment)
This looks like it started with the July 15 wheel bump ("Update bitsandbytes wheel for ROCm 7.13/7.14/7.15 compatibility") — the previously used 0.49-based wheel wouldn't have hit this.
Until OneTrainer supports 0.50, could the profile pin a 0.49-based wheel instead? The same repo has
v0.49.2.dev0-py312-rocm7.12— though it's built against ROCm 7.12 rather than 7.15, so it may not fit the torch build the profile installs. If no compatible 0.49 wheel exists for ROCm-on-Windows, it might be worth noting in the profile that bnb optimizers are currently broken there; OneTrainer will catch up once the 0.50 migration lands.Drafted by Claude
Thanks for the report. There is no availability of a bitsandbytes wheel version .49 for the current production version of ROCm.
Going back to that version would involve reverting the ROCm install logic to ROCm 3.12 from the (soon to be deprecated and no longer updated) v2 repo as there is not any version lesser than 7.13 in the currentmulti-archrepo.Upstream b'n'b also has only added support starting with 7.14 with version .50 also. (which will switch to using when verified for windows compatibility).
Having a carve out for the OneTrainer package would require some modification of the ROCm helper to accommodate with a decent bit of logic specific for it for something that is only temporary.
Think best option would be just disabling the automatic install for bnb for the time being until the .50 pin update in upstream OneTrainer. Importing of the regular .49 bnb will fail at every package launch... but should be handled gracefully. Along with a note in the package card in the SM UI, or in the console output as a message at launch-time about it not working temporarily.
In regards to a "working AMD install", this is all facilitated by custom helper framework for ROCm on Windows done in-line additional to the base install of OneTrainer.
It is effectively the same process as what my GPU detect and ROCm install/launch/updating does in this fork, same fundamental process as I had proposed in the OneTrainer discord previously (except any baseline optimization/compatibility envioronment variables that SM passes to all ROCm helper compatible package launches)
Fixed in #1723 by pinning the
0.49.1wheel. I opened Nerogar/OneTrainer#1708 to migrate OneTrainer to0.50.2, but that is not on the roadmap right now, so the wheel moves to match the pin instead.
OneTrainerWindowsRocmProfile.cscurrently installs bitsandbytes from0xDELUXA/bitsandbytes_win_rocmrelease0.50.0.dev0-py3.12-rocm7.15-win_amd64_all.OneTrainer pins
bitsandbytes==0.49.1and doesn't support 0.50 yet: 0.50 removed theblock_wise=andpercentile_clipping=arguments from all optimizer constructors, which OneTrainer still passes. Any training run using a bitsandbytes optimizer (AdamW8bit etc.) fails at startup:Details on the API break and the planned OneTrainer-side upgrade are here:
Nerogar/OneTrainer#1519 (comment)
This looks like it started with the July 15 wheel bump ("Update bitsandbytes wheel for ROCm 7.13/7.14/7.15 compatibility") — the previously used 0.49-based wheel wouldn't have hit this.
Until OneTrainer supports 0.50, could the profile pin a 0.49-based wheel instead? The same repo has
v0.49.2.dev0-py312-rocm7.12— though it's built against ROCm 7.12 rather than 7.15, so it may not fit the torch build the profile installs. If no compatible 0.49 wheel exists for ROCm-on-Windows, it might be worth noting in the profile that bnb optimizers are currently broken there; OneTrainer will catch up once the 0.50 migration lands.Drafted by Claude