AI cost management becomes critical as GPU and token expenses rise

Organizations are facing significant challenges in managing the rising costs of artificial intelligence, particularly regarding GPU compute and unpredictable consumption units. According to the FinOps Foundation, 98% of organizations now formally manage AI costs, a rapid increase from 31% two years ago. Despite this, 41% of enterprises still waste more than 15% of their AI budgets, often due to idle hardware. For example, an audit of 50 AI teams found average GPU usage at only 22%.

Cloud providers present high costs for high-end hardware; an 8-GPU H100 node on AWS can cost between $55 and $60 per hour, while Azure prices can reach nearly $98 per hour.

Beyond hardware, forecasting remains difficult because the units of measurement are inconsistent. Companies like Ensono report that vendors often shift from token-based pricing to bundled credits, which obscures per-model pricing volatility and complicates reconciliation. Major firms including Uber, Accenture, and Walmart have already implemented various measures to cap or govern AI spending to prevent uncontrolled budget depletion.