Run Qwen3-Coder-Next on 8Gb VRAM [DO NOT MERGE]

[nalexand]

No description provided.

[nalexand]

I am running large llms on my 8Gb laptop 3070ti. I have optimized: LTX-2, Wan2.2, HeartMula, ACE-STEP 1.5.

And now i abble to run 80b parameters model Qwen3-Coder-Next !!!

Instruction here: https://github.com/nalexand/Qwen3-Coder-OPTIMIZED

It is FP8 quant 80Gb in size, it is impossible to fit it on 8Gb VRAM + 32Gb RAM.

So first i tried offloading to disk with device="auto" using accelerate and i got 1 token per 255 second :(.

Than i found that most of large tensors is mlp experts and all other fit in 4.6Gb VRAM so i build custom lazy loading for experts with 2 layers caching VRAM + pinned RAM and got up to 85% cache hit rate and speed up to 1.2t/s it`s 300x speedup.

I wonder what speed will be on 4090 or 5090 desktop..

self.max_gpu_cache = 18 # TODO: calculate based on free ram and context window size
self.max_ram_cache = 100 # TODO: calculate based on available pinable memory or use unpinned (slow)

Tune this two parameters for your RAM/VRAM (each 18 it is about 3GB). For 5090 max_gpu_cache = 120 and it is >85% !! cache hit rate. Who can check speed?

Best for loading speed: PCE 5.0 Raid 0 up to 30Gb/s NVME SSD.

Available pinable ram (usualy 1/2 RAM) with DMA - much faster than RAM.

Hope 5090 will give > 20 t/s.. (i don`t have FP8 capable GPU so all calculations now in bfp16, TODO: find 4090 or 5090 and optimize for FP8)

Same lazy load and caching strategy with full controll on vram/ram usage can be used for all MoE models!!