Cloud providers are underutilizing their GPUs despite deploying thousands, reports the Register.
According to analyst firm TechInsights, in 2023 878,000 accelerators were responsible for turning out seven million GPU-hours of work, for an estimated $5.8 billion in revenue spending.
TechInsights, however, speculates that if the GPU clusters were operating close to capacity, the revenue figure would be much higher.
For example, AWS' UltraScale clusters are each comprised of 20,000 Nvidia H100 GPUs which can be rented for $98.32 per hour for a set of eight. Complete utilization would bring in revenue for Amazon alone of around $6.5bn.
TechInsight's analyst Owen Rogers said: "In fact, if each type of accelerator offered by AWS today were situated in a cluster of 20,000 in each region in which they are currently available and sold around the clock, they would generate 50 percent of AWS' revenue for 2023."
Some of this underutilization may be accounted for by internal workloads. There is also the fact that in order for cloud providers to enable the scalability they market themselves on, they must over-provision for actual expected demand.
Rogers noted that GPUs are typically passed through to a virtual machine, or an entire server is dedicated to a customer. With a large proportion of customers seeking to use GPUs looking to run large language models, it is also likely that they will be spinning up large clusters - and if customers all want access to thousands of GPUs simultaneously, some will have to wait.
With this in mind, both Google and Amazon have recently launched workload 'scheduling' services.
Rogers also pointed out that there are alternative methods for accessing GPUs both on the cloud and off. An example proffered is Amazon's SageMaker platform.
"Their [Amazon's] argument would be, if you can't get the capacity; if you're struggling to deal with when to use the capacity; or how to schedule, you could offload that to our platform, and we'll do it all for you," said Rogers.
He also pointed to offerings from colocation and bare metal-as-a-service providers, such as CoreWeave, which will lease H100s for as little as $2.23 an hour.
In December 2023, Yotta Data Services launched a GPU-based cloud offering in India based on Nvidia H100 GPUs. Nvidia H100 GPUs are also available as a cloud offering on Google Cloud Platform, AWS, Gcore, Oracle Cloud Infrastructure, and Voltage Park.
