Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI

Headline Performance Claim:

• The NVIDIA Rubin GPU, as part of the Vera Rubin platform, is designed to deliver up to 10x more agentic throughput per unit of energy compared to NVIDIA Blackwell. This claim is based on an internal 2-trillion parameter Mixture-of-Experts (MoE) workload.

Rubin GPU Core Specifications:

• Transistors: 336 billion, built on two reticle-limited compute dies connected by a high-speed NVIDIA High-Bandwidth Interface (NV-HBI).
• Compute Units: 224 Streaming Multiprocessors (SMs) and 896 Tensor Cores.
• Performance: Up to 50 petaflops of NVFP4 inference performance, enabled by a third-generation Transformer Engine.
• Memory: Up to 288 GB of HBM4 memory using 12-Hi stacks.
• Memory Bandwidth: Up to 22 TB/s, a 2.8x increase over Blackwell and Blackwell Ultra.

Connectivity and I/O:

• GPU-GPU (Scale-up): 3,600 GB/s of bandwidth via NVIDIA NVLink 6 to the NVLink Switch.
• CPU-GPU (Coherent): 1,800 GB/s via NVLink-C2C.
• Host Connectivity: Up to 256 GB/s via x16 PCIe Gen 6.
• Security: Features Confidential Computing with TEE-I/O to secure data at rest, in transit, and in use.

Architectural Improvements for Inference:

• MoE Scaling: The Tensor Memory Accelerator (TMA) is enhanced with inline descriptor updates. This allows a kernel to use a single unified descriptor for tensors with the same layout but different memory locations, overriding fields like memory pointers directly in the TMA instruction. This reduces metadata management overhead for models with many experts.
• Matrix Operations: Tensor Core throughput per clock is doubled by processing twice the amount of data along the K dimension. This reduces loop iterations in GEMM operations, improving efficiency for both context and decode phases at high tensor-parallel scales.
• Long-Context Attention: Rubin combines activation sparsity with adaptive compression. Intermediate attention scores (from QK^T) can be loaded into a structured 2:4 sparse compressed form. Subsequent operations, like softmax and the second attention GEMM, can then operate on less data, reducing compute and data movement without changing the model's dense output format.
• Kernel Execution Efficiency: Rubin introduces fine-grained, tile-level coordination between dependent kernels. This allows a consumer kernel to begin work as soon as its required input data is available from a producer, rather than waiting for a larger block of work to complete, reducing idle gaps in the GPU timeline.
• NVLink Communication: Introduces counted writes for device-initiated NVLink communication, which streamlines synchronization for GPU-to-GPU data transfers and reduces latency.

Exponential Throughput Improvements:

Vera Rubin NVL72 Rack-Scale System:

• Power Efficiency: Integrates Intelligent Power Smoothing with energy storage. This reduces average power consumption by ~10% and 50ms peak power by ~20% compared to previous techniques.
• Increased Density: The NVIDIA DSX MaxLPS system can enable operators to provision up to 40% more GPUs within the same power budget at energy-efficient operating points.
• System Design: Features a third-generation MGX cable-free architecture, 45°C liquid cooling, and hot-swappable NVLink switch trays for improved resiliency and scalability.