Amazon Net Companies introduced Amazon SageMaker HyperPod Inference Gateway on September 18, 2026, a Kubernetes-native, GPU-aware routing system for giant language mannequin inference that deploys as a single managed add-on for Amazon EKS on present HyperPod infrastructure. AWS stated the gateway can cut back first-token latency by as much as 82%.
The Routing Drawback Behind the Gateway
In line with AWS, default Kubernetes load-balancing algorithms corresponding to round-robin and least-connections don’t have any visibility into GPU state: which pods have saturated KV caches, that are mid-way by way of long-context generations, and which have already got the LoRA adapter a request wants loaded in reminiscence. The corporate stated requests pile up behind busy pods whereas idle capability stays unused, first-token latency spikes above 4 seconds throughout visitors bursts, utilization turns into uneven and unpredictable, and operators over-provision to compensate. AWS described a situation by which a chatbot consumer ready 4.4 seconds for a primary token as a substitute sees it in beneath 800 milliseconds.
Two-Tier Structure
The gateway makes use of a two-tier design constructed on Kubernetes-native primitives. AWS stated it makes use of real-time GPU indicators to position each inference request on the best-suited pod. Tier 1 installs straight on every HyperPod or EKS cluster because the amazon-sagemaker-hyperpod-inference add-on and consists of three elements, all constructed on the open-source Gateway API Inference Extension. Envoy Gateway, a layer-7 proxy, terminates incoming HTTPS visitors and exposes a single non-public endpoint per cluster. The Physique-Based mostly Router inspects every incoming OpenAI-compatible request physique, extracts the mannequin discipline, and routes the request to the right mannequin pool, so one gateway can serve a number of fashions.
The Endpoint Picker consumes real-time Prometheus metrics from each model-serving pod and applies a weighted scoring algorithm throughout scorers protecting KV cache utilization, queue depth, LoRA adapter residency, prefix cache hit charge, and operating requests. Every scorer carries a configurable weight, permitting routing conduct to be tuned for a particular workload, corresponding to latency-sensitive chat versus throughput-optimized batch.
Tier 2, the International Inference Router, is listed as coming quickly. AWS stated it can add fleet-wide coordination throughout a number of clusters and areas, with cross-cluster failover, international charge limiting, and cost-aware visitors shaping. Tier 2 builds on prime of Tier 1, whereas every cluster’s per-cluster gateway continues to deal with native routing.
Deployment, Failure Dealing with, and Observability
Deployment consists of a single aws eks create-addon command and one declarative InferenceGatewayConfig customized useful resource that defines fashions and routing conduct, with present mannequin server deployments found by way of pod labels. AWS stated the set up requires no sidecars, no service mesh, and no software code modifications. The gateway exposes a normal OpenAI-compatible endpoint over HTTP; in line with AWS, present shopper code works unchanged, with no SDK modifications and no SigV4 signing for inference visitors.
For workloads serving fine-tuned LoRA adapters on a shared base mannequin, the Endpoint Picker’s LoRA Affinity Scorer routes adapter requests to pods that have already got the requested adapter resident in GPU memory; if no pod has it loaded, the request goes to the pod with probably the most accessible capability. AWS stated this eliminates adapter swap latency.
Documented failure behaviors cowl pod failure, pool exhaustion, cluster failure, and regional failure. On pod failure, the Endpoint Picker excludes pods with stale metrics and routes to wholesome pods, recovering routinely when metrics resume. On pool exhaustion, the gateway returns HTTP 429 with a Retry-After header whereas autoscaling provides capability. On cluster failure, the International Inference Router detects a stale heartbeat and redirects visitors inside 35 seconds, with gradual ramp-up when the cluster is reintroduced. On regional failure, cross-region routing prompts routinely, which AWS stated carries increased latency however no availability influence.
The gateway emits metrics on the pod, pool, cluster, and fleet ranges: KV cache utilization, queue depth, operating requests, and adapter residency by way of Prometheus on the pod stage; request totals, length histograms, and token counts by way of Prometheus and Grafana on the pool stage; common KV cache, error charge, and P99 latency by way of Amazon CloudWatch on the cluster stage; and routing selections, failover occasions, and charge restrict hits by way of CloudWatch on the fleet stage.
AWS-Reported Benchmark Outcomes
AWS stated it benchmarked 4 fashions starting from 8B to 235B parameters on p5.48xlarge situations with H100 GPUs and g5 situations with A10G GPUs. All visitors was routed by way of inner Utility Load Balancers, matching the trail a manufacturing request travels, with a devoted shopper node group producing managed load and mannequin servers remoted on a separate server node group. Each outcome makes use of the gateway’s default routing configuration with no tuning and is measured in opposition to a Kubernetes round-robin baseline on the identical mannequin replicas, in line with AWS.
Within the reported outcomes, a mixed-generation GPU fleet lower time-to-first-token P95 and P99 latency by 97% every for Llama-3.1-8B, with an 8% throughput enhance, and by 98% and 97% for Qwen3-32B, with a 50% throughput enhance. Below bursty visitors, Llama-3.1-70B posted P95 and P99 reductions of 94% and 98% with 12% increased throughput, whereas Qwen3-235B confirmed comparable P95 latency and an 89% decrease P99. With shared immediate prefixes, Llama-3.1-8B P95 and P99 latency fell 26% and 43%.
AWS stated that on a completely uniform fleet beneath regular visitors the gateway performs on par with round-robin, and it outlined comparable outcomes as variations inside run-to-run variance. The corporate stated the enhancements are largest the place round-robin struggles most: blended {hardware}, bursty demand, and shared immediate prefixes.
Availability and Roadmap
AWS describes the gateway as conformant with the Kubernetes Gateway API and its Inference Extension, configured by way of a single customized useful resource definition, and appropriate with any OpenAI-compatible mannequin server, together with vLLM, SGLang, and TGI. Administration works by way of kubectl, GitOps, Helm, and ArgoCD, with set up, upgrades, and rollback dealt with by way of the EKS add-on lifecycle.
Tier 1 per-cluster routing is accessible as of September 18, 2026, in areas the place the inference add-on is accessible. Past the International Inference Router, AWS’s named roadmap gadgets embrace canary visitors splitting, which is able to route a share of visitors to new mannequin variations utilizing InferenceModelRewrite customized sources, and circulation management that classifies requests as Crucial, Commonplace, or Sheddable with per-band admission management.
