llama.cpp 0.7.0 is out with a GPU expert cache for mixture-of-experts models that do not fit entirely in graphics memory. Released on October 15, the update includes a way to keep selected experts close to the GPU while the larger collection remains in system RAM.
The official release notes highlight generation improvements of up to 2.2 times in the developer’s tests. That figure comes from a specific offloading setup, rather than a speed increase every model or GPU will receive.
Keeping Recently Used Experts on the GPU
A mixture-of-experts, or MoE, model activates only some of its expert components for each token. The full model still needs storage, which is why a small active parameter count does not remove its memory requirements. Our VRAM explainer covers the memory needed for weights and conversation context.
The original implementation adds a GPU cache for experts stored in system RAM. Recently used experts stay in that cache; missing experts are uploaded when needed, and their calculations run on the GPU.
The cache path handles batches of up to 32 tokens; larger batches use regular offloading. Faster generation does not establish faster prompt processing.
What the Developer Measured
The developer tested Qwen3.8-Flash-Next Q4_0 on an EPYC 7742 system using 16 CPU cores and PCIe 4.0 x16.
On an RTX 4090, automatic fitting produced 25 tokens per second without the cache. Adding a 6,544MiB cache increased that to 39.4 tokens per second, approximately 57% faster.
The largest reported gain used an RTX 5090 with all experts assigned to host memory and a 19,000MiB GPU cache. Generation reached 67.8 tokens per second, compared with 30.8 for the default-fit baseline.
These are developer measurements. MTP was disabled, so they measure a different approach from the speculative decoding covered in our llama.cpp MTP article.
Available Now, With a Memory Tradeoff
The release also includes multi-GPU cache support, which uses the existing tensor-split settings to distribute the cache budget.
Users enable the feature with --moe-cache-mib N, where N sets its size in mebibytes. That allocation occupies VRAM, so it competes with model layers, the context cache, and other working memory.
The release also adds model support, speculative decoding updates, and server improvements. One practical change is the server’s default port moving to 9931; Docker images retain port 8080 inside the container.
