Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • Tesla Rolls Out FSD V15 with Actuality View and Imaginative and prescient Upgrades – Unite.AI
  • Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI
  • OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
AI-trends.todayAI-trends.today
Home»Tech»Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Consideration Makes Lengthy-Context Prefill As much as 11.3x Quicker

Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Consideration Makes Lengthy-Context Prefill As much as 11.3x Quicker

Tech By Gavin Wallace25/10/20265 Mins Read
Facebook Twitter LinkedIn Email
DeepSeek Releases R1-0528: An Open-Source Reasoning AI Model Delivering Enhanced
DeepSeek Releases R1-0528: An Open-Source Reasoning AI Model Delivering Enhanced
Share
Facebook Twitter LinkedIn Email

Stanford and UC Berkeley researchers launched TokenGate, a module for long-context prefill. TokenGate realized token pruning lets an current LLM skim its enter. Solely the tokens that matter get full processing. The analysis crew experiences 1.4x to 11.3x faster time-to-first-token (TTFT) with no accuracy loss. Peak prefill reminiscence drops 1.5x to three.4x. Code and checkpoints are on GitHub below Apache-2.0.

TL;DR

  • MLP gate: A small MLP gate and LoRA (r=16) added to Qwen3-1.7B or Qwen3.5-2B. Examined on contexts as much as 128k tokens.
  • Runs on: Paper timings use 1 A100 at batch dimension 1. Coaching used 4 A100s.
  • Efficiency: Pareto-optimal on 5 duties in opposition to 6 baselines.
  • Greatest: NIAH 16k at 11.3x quicker TTFT and three.42x decrease reminiscence. TTFT achieve hits 13.0x at 128k.
  • Worst: Speedup drops to 1.4x on some duties.

What’s TokenGate?

TokenGate is a drop-in part that provides trainable token pruning to pretrained LLMs. Transformers push each token by each layer. Consideration price grows quadratically with size. Coaching-free pruning is affordable however loses accuracy at excessive pruning charges. Architectural adjustments usually want pre-training from scratch.

How does TokenGate realized token pruning work?

The ahead cross splits into 3 phases:

  1. Customary ahead: layers under gate layer L course of all N tokens as traditional.
  2. Gate module: an MLP scores every token from its post-attention residual at layer L. A top-Okay selector retains the highest-scoring tokens.
  3. Pruned ahead: layers above L course of solely these Okay tokens and by no means create KV entries for pruned ones.

Solely the doc a part of the immediate is prunable. The query is rarely pruned.

What’s token-gated consideration?

Prime-Okay choice just isn’t differentiable, which blocks end-to-end coaching. TokenGate borrows the Combination-of-Consultants router trick. Every saved token’s rating turns into a bias on consideration logits above the gate. That bias offers gradients a path again to the scorer.

A learnable per-head achieve and a learnable temperature refine this. The temperature corrects for every question seeing fewer keys. Coaching makes use of plain next-token loss. Every step averages a full-context loss and a random-budget loss. So 1 skilled gate works throughout budgets.

window.addEventListener(“message”,perform(e){if(e.information&&e.information.mtpH){var f=doc.getElementById(“mtp-tg-frame”);if(f&&e.supply===f.contentWindow){f.type.top=e.information.mtpH+”px”;}}});

#mtp-tg-wrap hr,#mtp-tg-wrap p:empty,#mtp-tg-wrap del,#mtp-tg-wrap s{show:none!necessary}

How does TokenGate carry out on benchmarks?

The analysis crew examined Qwen3-1.7B on RepoBench-P, 2WikiMultihopQA, MultiNews, BioDEX and needle-in-a-haystack (NIAH). Baselines had been FastKV, SlimInfer, DASH, GemFilter, FlexPrefill and MInference, on the identical LoRA spine.

  • Versus full prefill: 1.4x to 11.3x quicker TTFT and 1.5x to three.4x decrease reminiscence.
  • Versus the next-best baseline: 1.2x to six.4x quicker TTFT and as much as 2.1x decrease reminiscence.
  • 2WikiMultihopQA: at 95% of full-context accuracy, prefill runs 3.0x quicker with 2.7x much less reminiscence.
  • Longer contexts: a gate skilled at 16k reaches 12.8x at 32k and 13.0x at 64k and 128k. Reminiscence financial savings keep close to 3.43x.

The realized selector beats FastKV’s heuristic on the identical layer. Even random-Okay choice scores greater on the TokenGate spine. The analysis crew credit spine co-adaptation.

Does it work on hybrid-attention fashions?

Sure, Qwen3.5-2B mixes Gated DeltaNet and full-attention layers. On NIAH 32k, TokenGate at layer 3 matched full-prefill accuracy. It ran 3.8x quicker than full prefill and a couple of.1x quicker than FastKV. Gating at layers 1 or 2 reached 7.8x, inside 6% accuracy. FastKV can solely prune consideration layers.

TokenGate vs FastKV, SlimInfer, GemFilter and DASH

Function TokenGate FastKV SlimInfer GemFilter DASH
Token choice Discovered MLP gate Consideration from final queries Block consideration, multi-stage Final-query consideration at filter layer Hidden-state stability
Coaching wanted Sure (gate + LoRA r=16) No No No No
Pruning layer in TokenGate’s Qwen3-1.7B runs 1 13 (default) 8, 17, 25 12 11
Prunes at Gated DeltaNet layers Sure No (consideration layers solely) Not disclosed Not disclosed Not disclosed
Greatest self-reported prefill achieve (personal setup) 11.3x TTFT, 3.4x reminiscence 1.82x prefill 2.53x TTFT (LLaMA3.1-8B, RTX 4090) 2.4x velocity, 30% much less GPU reminiscence vs SOTA Not disclosed
Code GitHub GitHub GitHub GitHub Announced

Self-reported positive aspects use totally different setups, so they don’t seem to be instantly comparable. Pruning layers come from TokenGate’s Appendix C.2.

How do you deploy TokenGate?

The GitHub Repo wants Python 3.10+ and CUDA PyTorch 2.9+. An elective FlashAttention-2 key-bias kernel hurries up dense Qwen3. It prices 1.0x to 1.1x a plain name. With out it, inference falls again to FlexAttention.

A {state} placeholder marks the prunable doc. generate takes a max_kept_prefill_tokens funds. Checkpoints cowl all 5 duties on Qwen3-1.7B, plus a Qwen3.5-2B NIAH gate. Instance: liana313/tokengate-niah-qwen3-1.7b.

Coaching mirrors a TRL/PEFT recipe. Every step is 1.3x slower than plain LoRA. Runs converged quicker on 4 of 5 duties. Imply coaching time was 1.8 versus 2.2 A100 hours.

Key Takeaways

  • TokenGate provides realized token pruning to current LLMs by way of a small gate and LoRA.
  • Prefill runs 1.4x to 11.3x quicker with 1.5x to three.4x decrease peak reminiscence.
  • Token-gated consideration makes top-Okay choice trainable, MoE-router type.
  • It beats the next-best pruning baseline by as much as 6.4x on TTFT.
  • It really works on hybrid Qwen3.5-2B, the place FastKV can’t prune DeltaNet layers.


Take a look at the Paper, GitHub repo and Hugging Face paper page. Additionally, be happy to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Have to accomplice with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and so on.? Connect with us

The submit Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster appeared first on MarkTechPost.

ar research search stanford x
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026
Top News

Katie Zacharia Was Linked to an AI Uncommon-Earths Startup. It’s Even Weirder Than It Sounds

CUDA Shows That Nvidia is a Software Company

Prompt Injection Attacks Are Thwarting AI Hacking Agents

OpenAI Says ChatGPT Runs a Fifth as Many Searches as Google

Why New Wearables Are Ignored

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

AWS Pizza Bot: A Open Source Inbox For Background AI Agents

13/09/2026

How to build a Netflix VOID video object removal and inpainting pipeline with CogVideoX and Custom Prompting.

06/04/2026
Latest News

Tesla Rolls Out FSD V15 with Actuality View and Imaginative and prescient Upgrades – Unite.AI

22/11/2026

Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.