The machines that wait

Modern police cars are packed with electronics, including a rugged laptop connected to dash-mounted cameras, automatic license plate readers, radios, positioning systems, and law-enforcement databases.

At the start of a shift, those computers can take several minutes to boot up — so they're not shut down until the end of the shift. 

It’s the main reason police officers keep their cars running, even when they’re not obviously doing anything. No one wants their first responders responding several minutes late because their laptop had to be restarted. 

This is how data centers work, too.

Even when idle, data centers leave their thousands of servers running — processors awake, memory live, network connections active — because waking them up to respond to your prompt takes too long. 

No one wants to wait several seconds to find out whether tomatoes are really a fruit.

As a result, data centers burn electricity just waiting for prompts to arrive. A lot of it.

Because they spend a lot of time waiting.

A recent study by Cast AI reports that the GPUs housed in data centers spend just 5% of their time doing useful work.

5%!!

Yes, these are the same data centers that hyperscalers can’t build fast enough, packed with the GPUs that Nvidia can’t make fast enough, delivering the AI people can’t seem to get enough of.

But most of them go unused, most of the time.

Generative AI is a bizarro market where hyperscalers and chip makers continuously raise prices for products and services their customers hardly ever use.

Cast AI blames this primarily on hoarding. 

High-end GPUs are scarce and their procurement has long lead times, their report explains. As a result, organizations are forced to guess at their future demand, signing contracts well in advance of delivery.

They generally overguess, preferring to pay extra rather than be caught short of compute.

Worse, even when they know they have too much of it, they keep it. With everyone else hoarding capacity, too, releasing a contracted GPU back to its owner carries its own risk: not being able to spin one back up in the moment it’s actually needed.

This is not a new phenomenon. A Microsoft study from way back in 2024 highlighted the problem of pervasively low GPU utilization. It said this could be fixed “with a small number of code/script modifications.”

And yet, the problem has only intensified since.

Underused data centers continue to be engineered for availability, not efficiency. And Cast AI says they’re getting less efficient as they scale — a remarkable inversion of how technology usually works.

"This is the third year we’ve published this report,” they note. “The numbers are worse."

Which means the problems of power consumption, water usage, and sky-high costs are all getting worse, too.

The servers remain powered up, the cooling systems keep running to cool them back down, and the hyperscalers keep billing their customers, no matter how little computation actually gets done.

Could crypto do something?

A research note from Messari suggests it might be able to. It highlights Dispersed, “a compute subnet of the Render Network ecosystem that turns a globally distributed set of GPUs into an on-demand marketplace for AI and general-purpose compute.”

This is the kind of coordination problem blockchains should be good at solving: matching intermittent demand with a pool of idle resources in an open, permissionless marketplace.

Dispersed aims to do exactly that, Messari explains, by pooling idle GPUs into a liquid marketplace — an Airbnb for compute, so to speak.

“The wager is that AI demand and idle supply will not reconcile inside the hyperscaler model,” the note adds.”

(Why would it, given the margins being made?)

“And that a marketplace settling in RENDER can clear the difference: operators monetize hardware that would otherwise sit at single-digit utilization, and consumers reach professional accelerators without the price or lock-in of incumbent clouds.”

If that seems like too obvious a solution to you, it’s probably because you’ve been paying attention to crypto for too long. There’s now a long history of centralized solutions, with all their low-minded flaws, beating out decentralized ones, despite all their high-minded ideals.

There’s reason for caution this time, too. Dispersed is a general-purpose compute subnet of Render, which was originally a marketplace for GPU capacity used to render graphics — an idea that has long made sense, but never really seemed to catch on. Six years after its launch, activity on the Render Network remains modest.

Still, Messari reports there’s “a long-standing waitlist of GPU operators hoping to join” Dispersed’s network — and that “early deployments already show the value proposition working in practice.”

One hopeful use case is evidence.guide, a nonprofit that uses AI to read scientific papers and predict whether their findings will replicate — a problem nearly as large as idling data centers, solved in a way that may help solve both.

This could, in theory, help break the cycle of scarcity and hoarding that’s made generative AI so insanely inefficient. If GPU capacity was pooled and available on-demand, fretful hoarders could release their excess capacity back into the wild, confident they could replace it when needed.

Police departments can't Airbnb their cruisers in between emergencies (sadly, because that would be fun). But GPU owners can.

They only have to trust they could get them back in a moment.

— Byron Gilliam