Large language models are getting incredibly powerful, but let’s be honest—their The inference speed Anyone trying to make use of them is going to have a lot of trouble. Google launched Multi-Token Prediction (MTP) drafters For the best, Gemma 4 model family. This is a specialized speculative decoding architecture Triple (3x) the speed of your car at Inference timeYou can have it all, without having to sacrifice anything. output quality You can also find out more about Justification accuracy. This release is just weeks after Gemma 4 reached 60 million downloads. It directly addresses one of the biggest pain points when deploying large languages models, the memory and bandwidth bottleneck which slows down token generation no matter what hardware capabilities are available.
What is the Slowness of LLM Inference??
The large language models of today operate auto-regressively. One token is produced at a given time. Each token is generated by loading millions of parameters into the compute unit from video RAM (VRAM). Memory-bandwidth is the term used to describe this process. This is because the CPU or GPU’s raw power cannot be compared to the transfer speed from memory into the compute units.
It is not uncommon for the latency to be a major bottleneck. The compute resources are underutilized, while data is being moved around. It is particularly inefficient that this model does the same amount computation on a trivially predictible token, such as predicting. “words” after “Actions speak louder than…” It is the same for generating complex logical conclusions. Standard autoregressive decoding does not have a mechanism to take advantage of how difficult or easy it is to guess the next token.
What is Speculative Decoding?
Gemma 4 MTP Drafters use a foundational decoding technique – speculative. Token generation is decoupled from verification with the help of two models, a light target and a lighter drafter.
The pipeline in action. The small, fast drafter model proposes several future tokens in rapid succession — a “draft” sequence — in less time than the large target model (e.g., Gemma 4 31B) takes to process even a single token. All of the tokens are then verified by the target model in a single, forward-looking pass. If the target model agrees with the draft, it accepts the entire sequence — and even generates one additional token of its own in the process. The application will then be able to output both the entire drafted sequence and an extra token within the time that it normally takes to create just one token.
The output of the Gemma 4 primary model is the same as what would be produced by the target model on its own. There is no quality tradeoff — it is a lossless speedup.
Gemma 4: New Features in Gemma 4 drafter architecture
Google’s architectural improvements have made Gemma 4 more efficient. Draft models use the activations of target models and can share their KV cache. It is an optimization that’s standard for transformer inference. The cache stores intermediate attention calculations so they do not have to be recalculated at each step. This cache allows the drafter to avoid wasting time on context computations that have already been processed by the target model.
Additionally, for the E2B and E4B edge models, the smallest Gemma 4 variants designed to run on mobile and edge devices — Google implemented an efficient clustering technique in the embedder layer. The final logit calculations, which map internal model representations into vocabulary probabilities, are a major bottleneck on edge hardware. This step is accelerated by the clustering method, which improves end-to-end production speed for hardware-limited devices.
Apple Silicon presents routing challenges unique to the Gemma 426B mix-of-experts model (MoE), for its hardware-specific performance. This is true even with batch sizes of just 1. A batch of 4 to 8 can unlock a speed-up up to 2.22x. NVIDIA hardware also shows similar gains based on batch size.
The Key Takeaways
- Google’s Multi-Token Prediction Drafters are now available for Gemma 4 models. They deliver 3x faster output speeds and higher reasoning accuracy without degrading the quality of their work.
- MTP drafters use a speculative decoding architecture that pairs a lightweight drafter model with a heavy target model — the drafter proposes several tokens at once, and the target model verifies them all in a single forward pass, breaking the one-token-at-a-time bottleneck.
- The draft models share the target model’s KV cache and activations, and for E2B and E4B edge models, an efficient clustering technique in the embedder addresses the final logit calculation bottleneck — enabling faster generation even on memory-constrained devices.
- MTP drafters, including Hugging Face and Kaggle with their model weights now come under Apache 2.0 licensing.
Check out the Model Weights The following are some examples of how to get started: Technical details. Also, feel free to follow us on Twitter Join our Facebook group! 130k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.
You can partner with us to promote your GitHub Repository OR Hugging Page OR New Product Launch OR Webinar, etc.? Connect with us
