Adobe optimizes AI training to reduce GPU idle time

At NVIDIA’s GTC conference, Adobe CTO Ely Greenfield detailed the challenges of maintaining GPU productivity during large-scale generative AI model training. Analysis revealed that approximately two-thirds of GPU time was wasted in an idle state, waiting for data. This inefficiency meant that roughly 60% of the budget allocated for GPU compute was lost to inactivity.

Adobe identified two primary causes for this waste. The first was a storage and retrieval bottleneck, where standard Ethernet networking could not handle the sustained, parallel, petabyte-scale movement of diverse datasets. Additionally, the process of checkpointing—periodically saving model copies to storage—consumed significant GPU time. To resolve this, Adobe transitioned to a high-performance network fabric designed for AI traffic and implemented a fragmented checkpointing system that saves model data in smaller pieces across multiple locations.

The second cause was uneven workload distribution, often referred to as a data loader problem. Even when data chunks were of equal size, varying file complexities caused some GPUs to finish tasks much faster than others, leading to idle wait times. Adobe addressed this by developing a proprietary balanced data loader to ensure tasks are distributed so that all GPUs complete their work nearly simultaneously. Following these optimizations, Adobe reported that GPU utilization increased to approximately 80%, which is near the practical limit when accounting for coordination overhead.