Stanford and UC Berkeley researchers launched TokenGate, a module for long-context prefill. TokenGate realized token pruning lets an current LLM skim its enter. Solely the tokens that matter get full processing. The analysis crew experiences 1.4x to 11.3x faster time-to-first-token (TTFT) with no accuracy loss. Peak prefill reminiscence drops 1.5x to three.4x. Code and checkpoints are on GitHub below Apache-2.0.
TL;DR
- MLP gate: A small MLP gate and LoRA (r=16) added to Qwen3-1.7B or Qwen3.5-2B. Examined on contexts as much as 128k tokens.
- Runs on: Paper timings use 1 A100 at batch dimension 1. Coaching used 4 A100s.
- Efficiency: Pareto-optimal on 5 duties in opposition to 6 baselines.
- Greatest: NIAH 16k at 11.3x quicker TTFT and three.42x decrease reminiscence. TTFT achieve hits 13.0x at 128k.
- Worst: Speedup drops to 1.4x on some duties.
What’s TokenGate?
TokenGate is a drop-in part that provides trainable token pruning to pretrained LLMs. Transformers push each token by each layer. Consideration price grows quadratically with size. Coaching-free pruning is affordable however loses accuracy at excessive pruning charges. Architectural adjustments usually want pre-training from scratch.
How does TokenGate realized token pruning work?
The ahead cross splits into 3 phases:
- Customary ahead: layers under gate layer L course of all N tokens as traditional.
- Gate module: an MLP scores every token from its post-attention residual at layer L. A top-Okay selector retains the highest-scoring tokens.
- Pruned ahead: layers above L course of solely these Okay tokens and by no means create KV entries for pruned ones.
Solely the doc a part of the immediate is prunable. The query is rarely pruned.
What’s token-gated consideration?
Prime-Okay choice just isn’t differentiable, which blocks end-to-end coaching. TokenGate borrows the Combination-of-Consultants router trick. Every saved token’s rating turns into a bias on consideration logits above the gate. That bias offers gradients a path again to the scorer.
A learnable per-head achieve and a learnable temperature refine this. The temperature corrects for every question seeing fewer keys. Coaching makes use of plain next-token loss. Every step averages a full-context loss and a random-budget loss. So 1 skilled gate works throughout budgets.
window.addEventListener(“message”,perform(e){if(e.information&&e.information.mtpH){var f=doc.getElementById(“mtp-tg-frame”);if(f&&e.supply===f.contentWindow){f.type.top=e.information.mtpH+”px”;}}});
#mtp-tg-wrap hr,#mtp-tg-wrap p:empty,#mtp-tg-wrap del,#mtp-tg-wrap s{show:none!necessary}
How does TokenGate carry out on benchmarks?
The analysis crew examined Qwen3-1.7B on RepoBench-P, 2WikiMultihopQA, MultiNews, BioDEX and needle-in-a-haystack (NIAH). Baselines had been FastKV, SlimInfer, DASH, GemFilter, FlexPrefill and MInference, on the identical LoRA spine.
- Versus full prefill: 1.4x to 11.3x quicker TTFT and 1.5x to three.4x decrease reminiscence.
- Versus the next-best baseline: 1.2x to six.4x quicker TTFT and as much as 2.1x decrease reminiscence.
- 2WikiMultihopQA: at 95% of full-context accuracy, prefill runs 3.0x quicker with 2.7x much less reminiscence.
- Longer contexts: a gate skilled at 16k reaches 12.8x at 32k and 13.0x at 64k and 128k. Reminiscence financial savings keep close to 3.43x.
The realized selector beats FastKV’s heuristic on the identical layer. Even random-Okay choice scores greater on the TokenGate spine. The analysis crew credit spine co-adaptation.
Does it work on hybrid-attention fashions?
Sure, Qwen3.5-2B mixes Gated DeltaNet and full-attention layers. On NIAH 32k, TokenGate at layer 3 matched full-prefill accuracy. It ran 3.8x quicker than full prefill and a couple of.1x quicker than FastKV. Gating at layers 1 or 2 reached 7.8x, inside 6% accuracy. FastKV can solely prune consideration layers.
TokenGate vs FastKV, SlimInfer, GemFilter and DASH
| Function | TokenGate | FastKV | SlimInfer | GemFilter | DASH |
|---|---|---|---|---|---|
| Token choice | Discovered MLP gate | Consideration from final queries | Block consideration, multi-stage | Final-query consideration at filter layer | Hidden-state stability |
| Coaching wanted | Sure (gate + LoRA r=16) | No | No | No | No |
| Pruning layer in TokenGate’s Qwen3-1.7B runs | 1 | 13 (default) | 8, 17, 25 | 12 | 11 |
| Prunes at Gated DeltaNet layers | Sure | No (consideration layers solely) | Not disclosed | Not disclosed | Not disclosed |
| Greatest self-reported prefill achieve (personal setup) | 11.3x TTFT, 3.4x reminiscence | 1.82x prefill | 2.53x TTFT (LLaMA3.1-8B, RTX 4090) | 2.4x velocity, 30% much less GPU reminiscence vs SOTA | Not disclosed |
| Code | GitHub | GitHub | GitHub | GitHub | Announced |
Self-reported positive aspects use totally different setups, so they don’t seem to be instantly comparable. Pruning layers come from TokenGate’s Appendix C.2.
How do you deploy TokenGate?
The GitHub Repo wants Python 3.10+ and CUDA PyTorch 2.9+. An elective FlashAttention-2 key-bias kernel hurries up dense Qwen3. It prices 1.0x to 1.1x a plain name. With out it, inference falls again to FlexAttention.
A {state} placeholder marks the prunable doc. generate takes a max_kept_prefill_tokens funds. Checkpoints cowl all 5 duties on Qwen3-1.7B, plus a Qwen3.5-2B NIAH gate. Instance: liana313/tokengate-niah-qwen3-1.7b.
Coaching mirrors a TRL/PEFT recipe. Every step is 1.3x slower than plain LoRA. Runs converged quicker on 4 of 5 duties. Imply coaching time was 1.8 versus 2.2 A100 hours.
Key Takeaways
- TokenGate provides realized token pruning to current LLMs by way of a small gate and LoRA.
- Prefill runs 1.4x to 11.3x quicker with 1.5x to three.4x decrease peak reminiscence.
- Token-gated consideration makes top-Okay choice trainable, MoE-router type.
- It beats the next-best pruning baseline by as much as 6.4x on TTFT.
- It really works on hybrid Qwen3.5-2B, the place FastKV can’t prune DeltaNet layers.
Take a look at the Paper, GitHub repo and Hugging Face paper page. Additionally, be happy to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Have to accomplice with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so on.? Connect with us
The submit Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster appeared first on MarkTechPost.

