Running local AI on a GPU with limited VRAM already involves compromises. You might need a smaller quantization, a shorter context window, or even move some model layers into system RAM. Speculative decoding can help with generation speed, but the traditional approach usually requires loading another model, which isn’t exactly ideal when you’re already running out of memory.
Multi-token prediction, or MTP, offers a different way to speed things up. With supported models, llama.cpp can generate tokens faster using a prediction head that’s already included in the model file. You don’t necessarily need to download and load a separate draft model, although MTP still requires some additional memory.
That makes it an interesting option for people with 8GB GPUs, but the actual improvement depends on the model, hardware, and what you’re generating.
How MTP Speeds Up Generation
Normally, a language model generates its response one token at a time. Each new token requires another decoding step, which involves repeatedly accessing the model’s weights in memory. For many local models, particularly larger ones, this process is limited by memory bandwidth. The GPU might have enough computing power to do more, but moving all that data takes time.
Speculative decoding tries to get around this by predicting several tokens ahead and letting the main model verify them together. Traditionally, this requires a smaller draft model to make those predictions. MTP takes a different approach by using a prediction head that was trained alongside the main model, so there’s no need for a completely separate language model.
The main model then checks the predicted tokens and accepts the ones that pass verification. This reduces the number of separate decoding steps needed to generate the same amount of text. Since the main model still verifies the predictions, properly implemented speculative decoding preserves the original model’s output distribution rather than trading accuracy for speed.
What Changed in llama.cpp?
On May 16, 2026, llama.cpp merged pull request #22673, bringing support for speculative decoding with MTP heads. This allowed compatible GGUF models to use their built-in prediction heads without requiring users to download an additional draft model.
Models from the Qwen3.5, Qwen3.6, and Qwen3.8 families are among the documented examples, although that doesn’t mean every version supports the feature. The GGUF needs to contain the necessary prediction tensors, and some quantized releases leave them out to save space.
Before trying MTP, it’s worth checking the model card to make sure those tensors are included. You’ll also need a recent llama.cpp build, since older versions may load the model normally without supporting its MTP functionality.
What the Benchmarks Actually Show
The interesting part is how differently MTP performs depending on the hardware. One example comes from the Qwen3.8-27B community benchmark repository, where contributors tested the feature across different GPUs and memory configurations.
One contributor tested Qwen3.8-27B on an RTX 5060 laptop GPU with 8GB of VRAM. The model was quantized to IQ4_XS, with some layers running on the GPU and the rest handled by the CPU. Without MTP, generation averaged around 4.2 tokens per second. Enabling it increased that to approximately 7.3 tokens per second, making generation about 74% faster.
That’s a pretty noticeable improvement for a 27B model running on hardware that can’t hold all its weights in VRAM. However, the faster generation came with additional memory usage. The reported GPU memory allocation increased from 6,796 MiB to 7,540 MiB, meaning MTP required another 744 MiB.
Another contributor tested a heavily quantized version of the same model on an RTX 3080 with 10GB of VRAM. This time, the model’s weights were kept entirely on the GPU. Generation improved from 45.1 to 64.4 tokens per second, roughly 43% faster, while memory usage increased by another 578 MiB.
Both examples suggest that MTP can provide meaningful speed improvements even on GPUs with relatively limited memory. However, they also show why the additional memory matters. An extra 500 to 700 MiB might not sound like much, but it becomes a bigger deal when your model already occupies nearly everything available.
These are also community benchmarks rather than tests we conducted ourselves, so the results shouldn’t be taken as something every GPU can reproduce.
Why Some Workloads Benefit More
MTP works best when its predictions are accepted frequently. If the prediction head guesses correctly, the main model can verify several tokens together instead of generating each one through a separate decoding step. This is especially useful for predictable output, which is one reason code generation can benefit more than ordinary conversation.
A Qwen3.6-35B-A3B MTP-enabled GGUF was tested on an AMD Radeon 780M integrated GPU. During a coding workload, generation improved from 27.07 to approximately 33.76 tokens per second, close to a 25% improvement. However, when the same model was tested with prose generation, performance actually dropped slightly from 26.82 to 26.36 tokens per second.
The difference was largely related to how often the predicted tokens were accepted. During code generation, the acceptance rate was around 89% to 91%, while prose had noticeably lower acceptance. Since programming languages follow relatively predictable structures, the prediction head had an easier time guessing what would come next.
This also explains why MTP isn’t guaranteed to make every response faster. If the model keeps rejecting predicted tokens, the extra work involved in generating and verifying them can outweigh the time saved.
The Problem With Limited VRAM
One of the biggest advantages of MTP is that you don’t have to load an entirely separate draft model. However, llama.cpp still needs additional memory for the prediction head and its runtime state, including the KV cache. That might become a problem if you’re already pushing your GPU close to its limit.
For example, suppose your 8GB GPU can barely fit a quantized model with its current context size. Enabling MTP introduces additional memory requirements, which could leave less room for other parts of the inference process. Depending on the configuration, you might need to reduce the context size, move more work into system RAM, or deal with an out-of-memory error.
There’s another limitation that doesn’t get as much attention. MTP mainly helps with generating new tokens, while processing the initial prompt is a different part of inference. The original llama.cpp implementation even mentions that MTP can negatively affect prompt-processing performance because of additional data transfers.
Someone generating hundreds of lines of code might benefit considerably from faster decoding, but someone who regularly submits long documents and only expects a short summary could see much smaller improvements. In some cases, the total response time could even become worse despite faster token generation.
This is why looking at generation speed alone doesn’t always tell the full story. The amount of available VRAM, prompt length, output length, and prediction acceptance all play a role in whether MTP is useful.
How to Try MTP Yourself
If you already have a compatible GGUF and a recent llama.cpp build, getting started is fairly straightforward. You can enable MTP with the following command, replacing the filename with your own model.
llama-server -m model-with-mtp.gguf --spec-type draft-mtp --spec-draft-n-max 2
The --spec-type draft-mtp option enables MTP-based speculative decoding, while --spec-draft-n-max 2 allows up to two draft tokens per step. You can experiment with three or four draft tokens as well, although increasing the number doesn’t necessarily improve performance. If the prediction head keeps guessing incorrectly, the additional work might actually slow generation down.
The easiest way to find out whether MTP helps is to compare the same model with and without it. Keep the context size, GPU offloading, and other settings unchanged, then run the same prompts while checking generation speed, draft acceptance, and memory usage.
It’s worth testing different workloads too. A coding prompt might produce a noticeable speed improvement, while a more open-ended writing prompt could show much smaller gains. You should also pay attention to prompt-processing time rather than focusing only on tokens per second during generation.
Is MTP Worth Enabling?
For people running local AI on limited hardware, MTP is definitely worth experimenting with if their model supports it. Some community benchmarks show impressive improvements, including a roughly 74% increase in generation speed on an 8GB laptop GPU. Other tests show smaller gains, and some workloads don’t benefit at all.
The memory requirement is probably the most important thing to consider. MTP doesn’t need an entirely separate draft model, but its prediction head and runtime state still occupy additional memory. If your model already uses nearly all your available VRAM, enabling it might create more problems than it solves.
Basically, MTP gives supported models another way to make better use of the hardware you already have. Just not magically. It can make generation considerably faster without loading a separate model, but whether it’s worth enabling depends on how much memory you have left and what you’re asking the model to generate.
