NVIDIA introduces per-task GPU memory limits in CUDA 13.4

NVIDIA has introduced new memory management capabilities in CUDA 13.4 via Multi-Process Service (MPS) V3. This update allows for more efficient GPU resource allocation in multi-tenant environments and distributed data centers by enabling per-task memory limits.

The system distinguishes between soft and hard limits. A soft limit acts as a target; workloads may exceed it, but the MPS may throttle or terminate them if memory competition occurs. A hard limit serves as a strict ceiling, where any allocation attempt exceeding the limit results in an out-of-memory error.

Users can configure memory in three ways: setting an exact limit, defining a flexible range consisting of a guaranteed minimum and a burst maximum, or restricting an application to see only a fraction of the total GPU memory.

To manage compute resources, NVIDIA offers two modes: time-slicing, which alternates GPU usage between tasks, and SM sharing, which allows simultaneous use of compute resources. While time-slicing provides better error isolation, SM sharing allows for concurrent execution. Implementation requires R615 or higher drivers and a Linux cgroup v2 environment. In Kubernetes clusters, the feature requires specific operators and schedulers, such as the KAI Scheduler, to manage GPU fractioning and resource requests.