Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
  • DailyAI Briefing: A Lean-checked Riemann proof, plus 4 extra
  • Anthropic Shares Drop 8.5% in Second Session Throughout AI Selloff – Unite.AI
  • OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min
  • Copywriting: The Full Information | Copyblogger
  • AI Governance in 2027: Accountability Strikes Into the Workflow – Unite.AI
AI-trends.todayAI-trends.today
Home»Tech»Software Frameworks Optimized for GPUs in AI: CUDA, ROCm, Triton, TensorRT—Compiler Paths and Performance Implications

Software Frameworks Optimized for GPUs in AI: CUDA, ROCm, Triton, TensorRT—Compiler Paths and Performance Implications

Tech By Gavin Wallace14/09/20256 Mins Read
Facebook Twitter LinkedIn Email
Researchers at UT Austin Introduce Panda: A Foundation Model for
Researchers at UT Austin Introduce Panda: A Foundation Model for
Share
Facebook Twitter LinkedIn Email

How well a compiler stack can map tensor program execution to GPUs is critical for deep learning throughput. It includes things like thread/block schedules and memory allocation, as well as instruction selections (e.g. Tensor core MMA pipes). In this article we will focus on four dominant stacks—CUDA, ROCm, Triton, and TensorRT—from the compiler’s perspective and explains which optimizations move the needle in practice.

How does performance affect modern GPUs?

The same levers are used by all vendors:

  • Operator scheduling & fusion: reduce kernel launches and round-trips to HBM; expose longer producer→consumer chains for register/shared-memory reuse. TensorRT, cuDNN “runtime fusion engines” This is a good example for conv and attention blocks.
  • Tiling & data layout: To avoid memory conflicts or partitioning, match the tile shape to native fragment size of Tensor Core/WGMMA/WMMA. CUTLASS documents GEMM tiling on a warp-level for Tensor Cores, CUDA Cores.
  • Precision & quantization: Inference: INT8/INT4 for calibration or QAT. TensorRT automatically calibrates and selects kernels under these precisions.
  • Graph capture & runtime specialization: The graph is executed to reduce launch costs; the dynamic fusion (e.g. Attention) of subgraphs. Attention fusion engines now have graph support in cuDNN 9.
  • Autotuning: The search results include the size of tiles, their unrolling factor, and how deep they are per arch/SKU. TensorRT is able to select builder tactic at the same time as Triton or CUTLASS.

Here’s how to implement the lens in each of these stacks.

CUDA Graphs: cuDNN (CuDNN), CUTLASS and nvcc/ptxas

Compiler path. CUDA Code Compilation through The nvcc PTX is then a good option. Ptxas Reduces PTX by lowering it to SASS. For kernels, the key to controlling optimization is feeding flags both into host and device phase. -Xptxas. Many developers overlook this. -O3 The host code alone is the only thing that affects this.

Kernel generation & libraries.

  • CUTLASS provides parametric templates for GEMM/conv, implementing warp-level tiling, Tensor Core MMA pipelines, and smem iterators designed for conflict-free access—canonical references for writing peak kernels, including Hopper’s WGMMA path.
  • CuDNN 9, introduced runtime fusion engines (notably for attention blocks), native CUDA Graph integration for those engines, and updates for new compute capabilities—materially reducing dispatch overheads and improving memory locality in Transformer workloads.

Performance Impairments

  • The switch from PyTorch to CuDNN attention fusion reduces the number of kernel launch and global memory requests. CUDA GraphsIt reduces the CPU bottlenecks for short-sequence analysis.
  • CUTLASS Tutorials show how poor-sized tiles can reduce tensor core throughput.

CUDA: the perfect tool for your needs If you need to have maximum control of instruction selection, smem choreography, or occupancy while remaining on NVIDIA GPUs, then this is the right solution.

ROCm : toolchain HIP/Clang rocBLAS/MIOpen & 6.x series

Compiler path. ROCm is used Clang/LLVM Compose HIP (CUDA-like) into GCN/RDNA ISA. Release notes for the 6.x series include component level optimizations, HW/OS coverage and perf.

Library kernels and books

  • rocBLAS The following are some examples of how to get started: MIOpen Implement GEMM/conv primitives, with tiling and algorithms selection that are arch-aware. This is similar to cuBLAS/cuDNN. The changelog consolidates iterative improvements in performance across all libraries.
  • Recent ROCm workstream includes better Triton On AMD GPUs you can enable Python kernel development while still lowering the backends to AMD through LLVM.

Performance Impairments

  • For AMD GPUs the alignment of smem (shared memory bank) banks is equally important to AMD as it is for NVIDIA. Compiler assisted fusion (e.g. attention) and library autotuning for rocBLAS/MIOpen can close a significant portion of the gap between handwritten kernels depending on architecture and driver. Release documentation indicates continuous tuner improvements in 6.0–6.4.x.

The right tools for ROCm The AMD Accelerators must be optimized and natively supported, and the HIP kernels should have portability to existing CUDA-style Kernels. A clear LLVM toolset is also required.

Triton – a DSL/compiler for custom kernels

Compiler path. Triton, a Python embedded DSL lowers the via LLVMIt handles vectorization and memory coalescing while giving explicit control of block sizes and IDs. The build docs mention LLVM and the custom builds. NVIDIA developer materials cover Triton tunings for newer architectures, such as Blackwell with FP16/FP8 GEMM.

Optimizations.

  • Autotuning over tile sizes, num_warpsThe static and non-static stages of pipelining Masking For boundary conditions with scalar fallsbacks. shared-memory Overlapping global load with compute requires software pipelines and staging.
  • Triton’s design aims to Automatism This separation is outlined in the original announcement.

Performance Impairments

  • Triton shines when you need a fused, shape-specialized kernel outside library coverage (e.g., bespoke attention variants, normalization-activation-matmul chains). On modern NVIDIA parts, vendor collabs report architecture-specific improvements in the Triton backend, reducing the penalty versus CUTLASS-style kernels for common GEMMs.

Triton can be the perfect tool. Python first iteration and autotuning are important to you if you need near-CUDA-performance for your custom fused ops.

TensorRT: Builder-time graph-optimization for inference

Compiler path. TensorRT is a graph-based ingestor that combines ONNX and frameworks to produce a hardware specific output. The engine is a powerful and efficient way to drive.. It performs during the building. Fusion of layer/tensor, precision calibration (INT8,FP8/FP16), Selecting kernel tacticsThese phases are described in best-practice documents. This is extended with LLM runtime optimizations in TensorRT.

Optimizations.

  • Graph-level: constant folding, concat-slice canonicalization, conv-bias-activation fusion, attention fusion.
  • Precision: post-training calibration (entropy/percentile/mse) and per-tensor quantization, plus smooth-quant/QAT workflows in TensorRT-LLM.
  • Runtime: paged-KV cache, in-flight batching, and scheduling for multi-stream/multi-GPU deployments (TensorRT-LLM docs).

Performance Impairments

  • Most of the largest gains are typically made from end-toend INT8 (or FP8 On Hopper/Blackwell, where supported), eliminating framework overhead through a single engine and aggressive attention-fusion. TensorRT’s engine builder generates plans per arch to avoid generic kernels.

Why TensorRT can be the best tool for you Quantization and large graph fusion are available on NVIDIA GPUs for production inference.

Choosing and tuning your stack: Practical advice

  1. Training vs. inference.
    • Training/experimental kernels → CUDA + CUTLASS (NVIDIA) or ROCm + rocBLAS/MIOpen (AMD); Triton for custom fused ops.
    • Production inference on NVIDIA → TensorRT/TensorRT-LLM for global graph-level gains.
  2. Exploit architecture-native instructions.
    • On NVIDIA Hopper/Blackwell, ensure tiles map to WGMMA/WMMA The CUTLASS material shows the structure of GEMMs and smems at the warp-level.
    • Triton and ROCm autotuners can be used to perform shape-specialized operations.
  3. Quantize first and then fuse.
    • Quantization increases math density and reduces bandwidth. Kernel/graph Fusion reduces memory traffic. TensorRT’s builder time fusions and INT8/FP8 can often result in multiplicative benefits.
  4. Short sequences can be graphed.
    • CUDA Graphs combined with cuDNN attention fusions reduce launch costs in autoregressive inference.
  5. Consider compiler flags to be first class.
    • For CUDA, remember device-side flags: example, -Xptxas -O3,-v (and -Xptxas -O0 Diagnose the problem. Host-only -O3 isn’t sufficient.

References:


Michal Sutter, a data scientist with a master’s degree in data science from the University of Padova is an expert. Michal Sutter excels in converting complex datasets to actionable insights. He has a strong foundation in statistics, machine learning and data engineering.

AI ATH software work
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026

RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Mannequin Skilled With Reinforcement Studying Solely

20/11/2026
Top News

China Turns Legacy Chips Into a Trade Weapon

Trump and Energy Industry are Eager to Use Fossil Energy for AI

Meta is charging a subscription for smart glasses features. The New Era in Consumer Tech

The Peter Thiel-linked Dialog Club secretly ranks its members

The Social Society Serving to Ladies Be part of the AI Increase

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

NEURA and SECO Partner on Robot Compute Modules Built in Europe – Unite.AI

07/09/2026

The Physical AI of Next-Gen Embodied Systems: Combining Robotics, Materials Science and Artificial intelligence

19/09/2025
Latest News

Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains

22/11/2026

Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI

21/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.