Better strategy for GPU offload #520
Conversation
|
Looks good for setups like mine where PCIe bandwidth is low and prompt length is short. 128gb ddr4 3200 2ch For an existing context of 1400 + added prompt of 34 tokens, the difference was waiting a mere 3 seconds instead of 23 seconds until the first tokens: PR build: cdcb324 (3743):
Main, note the very low speeds for pp32 to pp256 build: 3f54b49 (3742):
|
|
Here the above data illustrated in a graph:
|
|
A also took the liberty to plot @quasar-of-mikus's data:
We see that in this case the performance at 512 tokens is better on the main branch. With the default value of |
|
On my setup and with this model, a lower value of
|
In a hybrid GPU/CPU situation, the decision if to offload model weights residing in RAM to the GPU to perform matrix multiplications is a tricky business. On the master branch (and also in mainline
llama.cpp) a simply heuristics is used: if the batch size is>= 32and the operation is supported, it is offloaded to the GPU. This heuristics comes from the experience with dense models (but even then, the correct decision will depend on the speed of the CPU, the GPU, and the PCI-E bandwidth).This heuristics is definitely not meaningful for MoE models. In a MoE model with$N_{\rm tot}$ total routed experts and $N_A$ active experts, the matrix multiplication for each expert will contain, on average, $N_A/N_{\rm tot} N_b$ tokens, where $N_b$ is the batch (or rather, u-batch, size). For a model such as DeepSeek-R1/V3 with $N_A = 8, N_{\rm tot} = 256$ , a batch size of 32 will result in a single token per expert on average, so offloading gigabytes of data to the GPU does not make sense at all.
This PR adds the above consideration. MoE matrix multiplications will only be offloaded if
where$N_{\rm min}$ is the minimum batch size for dense models (hard-coded to 32 on the main branch). To allow for setup/model specific adjustment, a compile time option is added that allows to change $N_{\rm min}$ via
The default value for$\ge 1024$ . For Qwen3-235B-A22B the minimumbtach size for offload becomes 512 tokens.
GGML_CUDA_MIN_BATCH_OFFLOADis left at 32. With this, MoE matrix multiplications will not get offloaded for DeepSeelk-R1/V3 unless the batch size isAs a reminder, in addition to this PR in
ik_llama.cppGPU offload can be disabled via-op 26,0,27,0,29,0.As a quick example, the following tables contain
llama-benchresults forPP-4096usingIQ4_KSquantized DeepSeek-Lite, with all experts left on the CPU.On the main branch we get this:
With this PR we get this:
We see massively better performance for small u-batch
sizes (important for a more fluid interaction with the LLM as not all prompts are so long). For this model offload kicks in at64/6*32 = 341` tokens, so for batch sizes of 512 and above the two results are the same.If I change
GGML_CUDA_MIN_BATCH_OFFLOADto 64, min batch size for offload becomes 682 tokens, and we get this result:We see that for my setup, even batches of 512 tokens are better left on the CPU (for this specific quantization type).
Please play with this PR and let me know if it is useful to get merged.