Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
  • DailyAI Briefing: A Lean-checked Riemann proof, plus 4 extra
  • Anthropic Shares Drop 8.5% in Second Session Throughout AI Selloff – Unite.AI
AI-trends.todayAI-trends.today
Home»Tech»Google Cloud Ships llm-d 1.0 With Cache-Conscious Routing for TPU and GPU Fleets

Google Cloud Ships llm-d 1.0 With Cache-Conscious Routing for TPU and GPU Fleets

Tech By Gavin Wallace21/10/20269 Mins Read
Facebook Twitter LinkedIn Email
NVIDIA AI Introduces AceReason-Nemotron for Advancing Math and Code Reasoning
NVIDIA AI Introduces AceReason-Nemotron for Advancing Math and Code Reasoning
Share
Facebook Twitter LinkedIn Email

Google crew have released llm-d 1.0, the primary steady model of its distributed inference stack. This open-source, Kubernetes-native layer turns vLLM or SGLang mannequin servers into one coordinated pool. They launched it on the PyTorch Convention North America. It targets self-hosted agent and RL workloads, the place round-robin load balancing breaks down.

TL;DR

  • A serving stack: Google stories a printed benchmark on a 17,000-node multi-region GKE deployment.
  • Runs on: Kubernetes with NVIDIA, AMD or Intel GPUs or Google TPUs, over vLLM or SGLang.
  • Efficiency: Vendor-reported good points come from cache-aware routing, tiered KV cache offload and prefill/decode disaggregation.
  • Finest: Median TTFT reduce by greater than 85%, with 2.4x throughput, on an Ironwood TPU coding workload.
  • Worst: RL time-slicing lifts accelerator responsibility cycle solely to above 70%, so roughly 30% idle time stays.

What’s llm-d 1.0?

llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. Pink Hat, IBM and Google are the lead maintainers. The undertaking counts greater than 500 contributors from over 25 organizations. It’s a CNCF Sandbox project, accepted on March 12, 2026, and is licensed under Apache 2.0.

Prefix-aware routing, KV cache offloading, prefill/decode disaggregation, extensive skilled parallelism and batch inference already shipped. Model 1.0 is about working them reliably in manufacturing, which IBM calls day 2 operations.

What’s new in llm-d 1.0?

  • Multi-tenant high quality of service: Precedence and fairness-aware flow control stops bursty brokers from ravenous latency-sensitive requests.
  • Secure basis: v1 APIs and a hardened request path, validated on a number of clouds and accelerators.
  • Zero-downtime upgrades: Router excessive availability with sizzling standby, plus DisaggregatedSet, which rolls prefill and decode swimming pools collectively.
  • Agent-aware serving: Session-aware routing, turn-priority scheduling and native Responses API state.
  • Async inference: llm-d-async-processor pulls jobs from Google Cloud Pub/Sub or Redis into slack capability.
  • Expanded TPU assist: TorchTPU runs PyTorch fashions natively inside upstream vLLM and SGLang.

How does llm-d route requests?

The llm-d router builds on the Kubernetes Gateway API Inference Extension. It tracks which replicas maintain every cached prefix. It then weighs load and predicted latency earlier than choosing a duplicate. Brokers name one endpoint, and every step lands the place its context already lives.

When accelerator reminiscence fills, KV cache strikes to host reminiscence, then SSD or Managed Lustre. It makes use of vLLM’s OffloadingConnector, Mooncake or SGLang’s HiCache for this. Prefill and decode run as separate swimming pools, as a result of prefill is compute-heavy and decode is memory-bound.

<iframe id="mtp-llmd1-frame" title="llm-d 1.0 interactive explainer" srcdoc="<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta identify="viewport" content material="width=device-width,initial-scale=1">
<title>llm-d 1.0 Explainer</title>
<hyperlink href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600;800&display=swap" rel="stylesheet">
<fashion>
html,physique{margin:0!vital;padding:0!vital;background:clear!vital}
#mtp-llmd1{–bg:#0f1218;–card:#171b23;–line:#2a303c;–tx:#e8eaed;–mut:#9aa0a6;–blue:#4285F4;–red:#EA4335;–yel:#FBBC04;–grn:#34A853;–nv:#76B900;
max-width:880px;margin:0 auto;background:var(–bg)!vital;shade:var(–tx)!vital;border:1px stable var(–line)!vital;border-radius:16px;font-family:Inter,system-ui,sans-serif;overflow:hidden;box-sizing:border-box}
#mtp-llmd1 *{box-sizing:border-box}
#mtp-llmd1 hr,#mtp-llmd1 p:empty,#mtp-llmd1 del,#mtp-llmd1 s{show:none!vital}
#mtp-llmd1 .hd{padding:22px 24px 14px;border-bottom:1px stable var(–line)}
#mtp-llmd1 .tag{show:inline-flex;hole:6px;align-items:middle;font-size:11px;letter-spacing:.08em;text-transform:uppercase;shade:var(–mut)!vital}
#mtp-llmd1 .dots i{show:inline-block;width:8px;peak:8px;border-radius:50%;margin-right:3px}
#mtp-llmd1 h3{margin:8px 0 4px!vital;font-size:24px!vital;font-weight:800!vital;shade:var(–tx)!vital;line-height:1.2!vital}
#mtp-llmd1 .sub{margin:0!vital;shade:var(–mut)!vital;font-size:14px}
#mtp-llmd1 .tabs{show:flex;hole:4px;padding:10px 16px 0;border-bottom:1px stable var(–line);overflow-x:auto}
#mtp-llmd1 .tabs button{background:clear!vital;border:0!vital;border-bottom:3px stable clear!vital;shade:var(–mut)!vital;padding:10px 12px!vital;font:600 13px Inter,sans-serif;cursor:pointer;white-space:nowrap;border-radius:0!vital}
#mtp-llmd1 .tabs button.on{shade:var(–tx)!vital;border-bottom-color:var(–blue)!vital}
#mtp-llmd1 .pn{show:none;padding:20px 24px 24px;animation:mtpfade .35s ease}
#mtp-llmd1 .pn.on{show:block}
@keyframes mtpfade{from{opacity:0;rework:translateY(6px)}to{opacity:1;rework:none}}
#mtp-llmd1 .circulate{show:grid;grid-template-columns:repeat(4,1fr);hole:10px;margin-bottom:14px}
#mtp-llmd1 .node{background:var(–card)!vital;border:1px stable var(–line)!vital;border-radius:12px;padding:12px;text-align:middle;transition:all .3s;place:relative}
#mtp-llmd1 .node b{show:block;font-size:13px;shade:var(–tx)!vital}
#mtp-llmd1 .node small{shade:var(–mut)!vital;font-size:11px}
#mtp-llmd1 .node .ic{font-size:20px;show:block;margin-bottom:4px}
#mtp-llmd1 .node.act{border-color:var(–blue)!vital;box-shadow:0 0 0 3px rgba(66,133,244,.25);rework:translateY(-3px)}
#mtp-llmd1 .pkt{peak:6px;background:var(–line);border-radius:3px;place:relative;overflow:hidden;margin:4px 0 14px}
#mtp-llmd1 .pkt span{place:absolute;high:0;left:0;peak:100%;width:25%;background:linear-gradient(90deg,var(–blue),var(–grn));border-radius:3px;transition:left .5s ease}
#mtp-llmd1 .exp{background:var(–card)!vital;border-left:3px stable var(–blue)!vital;border-radius:8px;padding:12px 14px;font-size:14px;line-height:1.55;min-height:84px;shade:var(–tx)!vital}
#mtp-llmd1 .ctl{show:flex;hole:8px;margin-top:12px;align-items:middle}
#mtp-llmd1 .btn{background:var(–blue)!vital;shade:#fff!vital;border:0!vital;border-radius:8px;padding:8px 14px;font:600 13px Inter,sans-serif;cursor:pointer}
#mtp-llmd1 .btn.gh{background:clear!vital;border:1px stable var(–line)!vital;shade:var(–tx)!vital}
#mtp-llmd1 .ctl .st{margin-left:auto;font-size:12px;shade:var(–mut)!vital}
#mtp-llmd1 .tiers{show:flex;hole:6px;margin-top:12px;flex-wrap:wrap}
#mtp-llmd1 .tiers div{flex:1;min-width:110px;padding:8px;border-radius:8px;font-size:12px;text-align:middle;border:1px stable var(–line);shade:var(–tx)!vital;opacity:.35;transition:opacity .4s}
#mtp-llmd1 .tiers div.lit{opacity:1}
#mtp-llmd1 .bars .row{margin-bottom:12px}
#mtp-llmd1 .bars .lb{show:flex;justify-content:space-between;font-size:13px;margin-bottom:5px;shade:var(–tx)!vital}
#mtp-llmd1 .bars .lb em{font-style:regular;shade:var(–mut)!vital;font-size:11px}
#mtp-llmd1 .bars .tr{peak:12px;background:var(–card);border-radius:6px;overflow:hidden}
#mtp-llmd1 .bars .fl{peak:100%;width:0;border-radius:6px;transition:width 1.1s cubic-bezier(.2,.8,.2,1)}
#mtp-llmd1 .seg{show:flex;hole:6px;margin-bottom:14px}
#mtp-llmd1 .notice{font-size:11px;shade:var(–mut)!vital;margin-top:8px}
#mtp-llmd1 .grid{show:grid;grid-template-columns:repeat(3,1fr);hole:10px}
#mtp-llmd1 .card{background:var(–card)!vital;border:1px stable var(–line)!vital;border-radius:12px;padding:14px;cursor:pointer;transition:border-color .2s,rework .2s}
#mtp-llmd1 .card:hover{border-color:var(–blue)!vital;rework:translateY(-2px)}
#mtp-llmd1 .card b{font-size:14px;shade:var(–tx)!vital;show:block}
#mtp-llmd1 .card p{margin:6px 0 0!vital;font-size:12.5px;line-height:1.5;shade:var(–mut)!vital;max-height:0;overflow:hidden;transition:max-height .35s}
#mtp-llmd1 .card.open p{max-height:140px}
#mtp-llmd1 .card .okay{font-size:11px;shade:var(–blue)!vital}
#mtp-llmd1 desk{width:100%;border-collapse:collapse;font-size:13px}
#mtp-llmd1 th,#mtp-llmd1 td{padding:9px 8px;border-bottom:1px stable var(–line)!vital;text-align:left;shade:var(–tx)!vital;background:clear!vital;vertical-align:high}
#mtp-llmd1 th{shade:var(–mut)!vital;font-weight:600;font-size:12px}
#mtp-llmd1 th.us{shade:var(–blue)!vital}
#mtp-llmd1 .ft{show:flex;justify-content:space-between;hole:10px;padding:12px 24px;border-top:1px stable var(–line);font-size:11px;shade:var(–mut)!vital;flex-wrap:wrap}
#mtp-llmd1 .ft a{shade:#76B900!vital;text-decoration:none;font-weight:600}
@media (max-width:640px){
#mtp-llmd1 .circulate{grid-template-columns:repeat(2,1fr)}
#mtp-llmd1 .grid{grid-template-columns:1fr}
#mtp-llmd1 h3{font-size:20px!vital}
#mtp-llmd1 .pn{padding:16px}
#mtp-llmd1 .tw{overflow-x:auto}
#mtp-llmd1 desk{min-width:520px}
}
</fashion></head><physique>
<div id="mtp-llmd1">
<div class="hd">
<span class="tag"><span class="dots"><i fashion="background:#4285F4"></i><i fashion="background:#EA4335"></i><i fashion="background:#FBBC04"></i><i fashion="background:#34A853"></i></span>Google Cloud · Pink Hat · IBM · Oct 20, 2026</span>
<h3>How llm-d 1.0 serves brokers at fleet scale</h3>
<p class="sub">Open-source, Kubernetes-native distributed inference for vLLM and SGLang on TPUs and GPUs.</p>
</div>
<div class="tabs" function="tablist">
<button class="on" data-t="0">The way it works</button>
<button data-t="1">Manufacturing outcomes</button>
<button data-t="2">New in 1.0</button>
<button data-t="3">vs Dynamo and AIBrix</button>
</div>

<div class="pn on">
<div class="flow">
<div class="node act"><span class="ic"></span><b>Agent request</b><small>1 endpoint</small></div>
<div class="node"><span class="ic">🧭</span><b>llm-d router</b><small>prefix + load conscious</small></div>
<div class="node"><span class="ic">⚙</span><b>Prefill / decode</b><small>separate swimming pools</small></div>
<div class="node"><span class="ic">🗄</span><b>Tiered KV cache</b><small>HBM to Lustre</small></div>
</div>
<div class="pkt"><span></span></div>
<div class="exp" id="mtp-exp"></div>
<div class="tiers" id="mtp-tiers">
<div fashion="background:rgba(66,133,244,.15)">Accelerator HBM</div>
<div fashion="background:rgba(52,168,83,.15)">Host CPU RAM</div>
<div fashion="background:rgba(251,188,4,.15)">Native SSD</div>
<div fashion="background:rgba(234,67,53,.15)">Managed Lustre</div>
</div>
<div class="ctl">
<button class="btn gh" id="mtp-prev">Again</button>
<button class="btn" id="mtp-next">Subsequent step</button>
<button class="btn gh" id="mtp-play">Auto-play</button>
<span class="st" id="mtp-st">Step 1 of 4</span>
</div>
</div>

<div class="pn">
<div class="seg">
<button class="btn" data-g="lat">Latency and cache</button>
<button class="btn gh" data-g="thr">Throughput</button>
</div>
<div class="bars" id="mtp-bars"></div>
<div class="note">All figures self-reported by Google, Snap and Zenteiq. Supply: Google Cloud weblog, Oct 20, 2026.</div>
</div>

<div class="pn">
<div class="grid">
<div class="card"><span class="k">Multi-tenant QoS</span><b>Stream management</b><p>Precedence and fairness-aware routing retains bursty agent loops from ravenous interactive customers.</p></div>
<div class="card"><span class="k">Upgrades</span><b>DisaggregatedSet</b><p>Constructed on LeaderWorkerSet. Rolls prefill and decode swimming pools collectively on matching variations with out dropping periods.</p></div>
<div class="card"><span class="k">Reliability</span><b>Router HA</b><p>Sizzling standby and rolling updates preserve visitors flowing via upgrades and upkeep.</p></div>
<div class="card"><span class="k">Brokers</span><b>Session-aware routing</b><p>Flip-priority scheduling and native Responses API state assist brokers reuse context throughout turns.</p></div>
<div class="card"><span class="k">Background jobs</span><b>Async processor</b><p>Pulls requests from Google Cloud Pub/Sub or Redis and runs them in slack capability.</p></div>
<div class="card"><span class="k">TPU</span><b>TorchTPU time-slicing</b><p>Interleaves inference into idle RL cycles. Obligation cycle rises from about 40% to above 70%.</p></div>
</div>
<div class="note">Click on a card to disclose particulars. Secure v1 APIs ship throughout all parts.</div>
</div>

<div class="pn"><div class="tw">
<desk>
<tr><th></th><th class="us">llm-d 1.0</th><th>NVIDIA Dynamo</th><th>AIBrix</th></tr>
<tr><td>Model</td><td>1.0 (Oct 2026)</td><td>v1.5.1 (Oct 2026)</td><td>v0.7.0 (Jun 2026)</td></tr>
<tr><td>Governance</td><td>CNCF Sandbox</td><td>NVIDIA-led</td><td>vLLM undertaking org</td></tr>
<tr><td>Engines</td><td>vLLM, SGLang</td><td>SGLang, TRT-LLM, vLLM</td><td>vLLM, SGLang, TRT-LLM</td></tr>
<tr><td>Accelerators</td><td>NVIDIA, AMD, Intel, TPU</td><td>NVIDIA GPUs</td><td>Blended GPUs</td></tr>
<tr><td>KV tiers</td><td>HBM, CPU, SSD, Lustre</td><td>GPU, CPU, SSD, distant</td><td>Distributed KV cache</td></tr>
<tr><td>License</td><td>Apache 2.0</td><td>Apache 2.0</td><td>Apache 2.0</td></tr>
<tr><td>GitHub stars</td><td>4.8k</td><td>8.3k</td><td>5.1k</td></tr>
</desk></div>
<div class="note">Stars as of Oct 20, 2026. Sources: every undertaking’s GitHub repo and docs.</div>
</div>

<div class="ft"><span>Sources: Google Cloud, IBM Analysis, Pink Hat, GitHub</span><a href="https://www.marktechpost.com" goal="_blank" rel="noopener">© Marktechpost</a></div>
</div>
<script>
(perform(){
var R=doc.getElementById(‘mtp-llmd1’);
var tabs=R.querySelectorAll(‘.tabs button’),pns=R.querySelectorAll(‘.pn’);
tabs.forEach(perform(b){b.onclick=perform(){tabs.forEach(perform(x){x.classList.take away(‘on’)});pns.forEach(perform(x){x.classList.take away(‘on’)});b.classList.add(‘on’);pns[+b.dataset.t].classList.add(‘on’);if(+b.dataset.t===1)draw(cur);rs();};});
var steps=[
‘An agent sends one step of a multi-turn task to a single llm-d endpoint. TPU and GPU pools sit behind it, running vLLM or SGLang.’,
‘The router tracks which replicas already hold each cached prefix. It weighs load and predicted latency, then sends the step where its context lives.’,
‘Prefill (compute-heavy) and decode (memory-bound) run in separate pools. DisaggregatedSet upgrades both together without dropping sessions.’,
‘When accelerator memory fills, KV cache moves to host RAM, then SSD or Google Cloud Managed Lustre instead of being evicted.’];
var s=0,nodes=R.querySelectorAll(‘.node’),pk=R.querySelector(‘.pkt span’),ex=doc.getElementById(‘mtp-exp’),st=doc.getElementById(‘mtp-st’),ti=R.querySelectorAll(‘#mtp-tiers div’),timer=null;
perform present(){nodes.forEach(perform(n,i){n.classList.toggle(‘act’,i===s)});pk.fashion.left=(s*25)+’%’;ex.textContent=steps[s];st.textContent=”Step “+(s+1)+’ of 4′;
ti.forEach(perform(t,i){t.classList.take away(‘lit’);if(s===3)setTimeout(perform(){t.classList.add(‘lit’)},i*350);else if(i===0)t.classList.add(‘lit’)});}
doc.getElementById(‘mtp-next’).onclick=perform(){s=(s+1)%4;present()};
doc.getElementById(‘mtp-prev’).onclick=perform(){s=(s+3)%4;present()};
doc.getElementById(‘mtp-play’).onclick=perform(){var b=this;if(timer){clearInterval(timer);timer=null;b.textContent=”Auto-play”;return}b.textContent=”Pause”;timer=setInterval(perform(){s=(s+1)%4;present()},2600)};
present();
var knowledge={lat:[
[‘Ironwood TPU: median TTFT cut’,’>85%’,85,’#4285F4′,’agentic coding vs plain K8s Service’],
[‘Gemini: P95 TTFT cut (bursty chat)’,’>50%’,50,’#34A853′,’GKE Inference Gateway’],
[‘A3 Mega 70B: TTFT improvement’,’~40%’,40,’#FBBC04′,’Lustre + CPU offload, 50k-token prompts’],
[‘Gemini: TTFT cut (coding agents)’,’>35%’,35,’#EA4335′,’production traffic’],
[‘Gemini: prefix cache hit rate’,’35% to 70%’,70,’#4285F4′,’doubled’]],
thr:[
[‘Zenteiq: aggregate throughput’,’>4x’,4,’#34A853′,’1.47B to 5.97B tokens/day per 1,000 TPU v6e’],
[‘17,000-node GKE: throughput’,’2.9x’,2.9,’#4285F4′,’3 clusters, 99.9% success’],
[‘Ironwood TPU: total throughput’,’2.4x’,2.4,’#FBBC04′,’vs plain K8s Service’]]};
var cur=”lat”;
perform draw(g){cur=g;var field=doc.getElementById(‘mtp-bars’),mx=g===’thr’?4:100;field.innerHTML=”;
knowledge[g].forEach(perform(d){var r=doc.createElement(‘div’);r.className=”row”;r.innerHTML='<div class="lb"><span>’+d[0]+’ <em>’+d[4]+'</em></span><b>’+d[1]+'</b></div><div class="tr"><div class="fl" fashion="background:’+d[3]+’"></div></div>’;field.appendChild(r);
var f=r.querySelector(‘.fl’);setTimeout(perform(){f.fashion.width=(d[2]/mx*100)+’%’},60);});rs();}
R.querySelectorAll(‘[data-g]’).forEach(perform(b){b.onclick=perform(){R.querySelectorAll(‘[data-g]’).forEach(perform(x){x.classList.add(‘gh’)});b.classList.take away(‘gh’);draw(b.dataset.g)}});
R.querySelectorAll(‘.card’).forEach(perform(c){c.onclick=perform(){c.classList.toggle(‘open’);setTimeout(rs,380)}});
perform rs(){strive{father or mother.postMessage({mtpH:R.offsetHeight+40,id:’mtp-llmd1′},’*’)}catch(e){}}
window.addEventListener(‘load’,rs);window.addEventListener(‘resize’,rs);setTimeout(rs,300);
})();
</script>
</physique></html>
” style=”width:100%;max-width:880px;peak:600px;border:0;show:block;margin:0 auto;” loading=”lazy”>
window.addEventListener(‘message’,perform(e){if(e.knowledge&&e.knowledge.id===’mtp-llmd1’&&e.knowledge.mtpH){doc.getElementById(‘mtp-llmd1-frame’).fashion.peak=e.knowledge.mtpH+’px’;}});

What do the llm-d 1.0 benchmarks present?

All figures beneath are self-reported by Google and llm-d adopters.

  • Ironwood TPU (TPU 7x): prefix-aware routing plus host reminiscence offload reduce median TTFT by greater than 85%. Throughput rose 2.4x versus a plain Kubernetes Service.
  • Gemini platform: prefix cache hit fee doubled from 35% to 70%. TTFT fell greater than 35% for coding brokers. P95 TTFT fell more than 50% under bursty chat.
  • 17,000-node GKE benchmark: throughput scaled 2.9x throughout 3 clusters, with 99.9% success and under 1% overhead.
  • A3 Mega, 70B mannequin: Lustre plus CPU RAM offload improved TTFT by about 40% on 50k-token prompts. End-to-end latency fell 30% and GPU-hours fell nearly 60%.
  • RL time-slicing: accelerator responsibility cycle rose from about 40% to above 70%.

Adopters report related patterns. Snap stories prefix cache hit charges of as much as 75 to 80%. Zenteiq stories over 4x throughput, from 1.47 billion to five.97 billion day by day tokens per 1,000 TPU v6e chips. An IBM demo served 3,000 concurrent coding agents at 5 to 10 occasions beneath API pricing.

How does llm-d evaluate with NVIDIA Dynamo and AIBrix?

Characteristic llm-d 1.0 NVIDIA Dynamo AIBrix
Newest model 1.0 (Oct 20, 2026) v1.5.1 (Oct 6, 2026) v0.7.0 (Jun 16, 2026)
License Apache 2.0 Apache 2.0 Apache 2.0
Governance CNCF Sandbox NVIDIA-led (ai-dynamo org) vLLM undertaking org, ByteDance origin
Engines vLLM, SGLang SGLang, TensorRT-LLM, vLLM vLLM, SGLang, TensorRT-LLM
Accelerators NVIDIA, AMD, Intel GPUs, Google TPU NVIDIA GPUs (Blackwell, Hopper, Ada, Ampere) Blended GPUs (heterogeneous serving)
Cache-aware routing Sure, prefix and load-aware Sure, KV-aware Sure, gateway routing
KV cache tiers HBM, CPU, SSD, Managed Lustre GPU, CPU, SSD, distant storage (KVBM) Distributed KV cache
P/D disaggregation Sure, DisaggregatedSet Sure Sure
Headline consequence 2.9x throughput, 17,000 nodes (self-reported) 2x sooner TTFT, Qwen3-Coder 480B (Baseten) Not disclosed

These usually are not strict rivals. CoreWeave runs llm-d alongside Dynamo, and NVIDIA contributes Dynamo parts to llm-d.

How do you deploy llm-d 1.0?

On Google Cloud, GKE Inference Gateway runs the llm-d router as a managed service. It may additionally span clusters and regions with world failover. KEDA and HPA scale swimming pools on queue depth and predicted latency. Kueue enforces per-model quotas. Verified Helm charts deploy TPU and GPU swimming pools behind one endpoint.

Google crew suggests a staged path. Begin with prefix-aware routing in entrance of an present vLLM deployment. Then add circulate management, tiered KV offload and disaggregation. Revealed runs will be in contrast on Prism.

Key Takeaways

  • llm-d 1.0 is the primary steady launch, with v1 APIs.
  • New: circulate management, router HA, DisaggregatedSet and agent-aware routing.
  • Gemini doubled prefix cache hits from 35% to 70%.
  • A 17,000-node GKE benchmark scaled throughput 2.9x.


Try the Google Cloud announcement, the IBM Research post, the GitHub repo and the GKE docs. All credit score goes to the researcher of this undertaking. Additionally, be at liberty to comply with us on Twitter and don’t neglect to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

The put up Google Cloud Ships llm-d 1.0 With Cache-Aware Routing for TPU and GPU Fleets appeared first on MarkTechPost.

cloud Google
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026
Top News

Four indicted on suspicion of conspiring to smuggle Nvidia and supercomputers into China

AI Brokers Confirm the Longest Proof in Maths in 24 Days

A whole bunch of Mundane AI Apps Are Hiding Sexually Abusive Deepfake Options

Then, is it? “AI bubble” About to burst late in 2025 or even 2026?

Vibe Apps: Thousands of Vibe Coded Apps Can Expose Personal and Corporate Data to the Open Web

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

OpenViking is an open-source context database that brings filesystem-based memory and retrieval to AI agent systems like OpenClaw

15/03/2026

NVIDIA Releases Cosmos 3 Edge – A Model with 4B Parameters that Reasons on Devices, and Generating Robotic Actions

21/07/2026
Latest News

OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI

22/11/2026

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.