AI without the data center

Blockchains, often accused of being a solution in search of a problem, may have found a big problem to solve: the centralization of AI.

As problems go, it’s been a theoretical one so far. Consumers of AI are currently overrun with free chatbots and open-source models to choose from.

But as the topic of AI safety has now burst into the mainstream, it seems increasingly likely that the big labs that provide us with these chatbots and models will soon be regulated — perhaps in a way that leaves only a handful of government-approved providers able to build any new ones.

That dystopian prospect has inspired a call to action among techno-libertarians and AI accelerationists: “The only possible resistance is to build decentralized, permissionless, and sovereign AI,” the pseudonymous Beff Jezos said on X.

Replace “AI” with “money” and it’s exactly what crypto accelerationists have been working on for the last 15 years.

The cause of decentralized AI even has a crypto-esque meme to rally around: a mash-up of the gun-rights motto “Come and take it” and the “Don’t tread on me” snake, now snaking out of a GPU on a revolutionary flag.

Yes, it’s a little tinfoil-hat and backwoods-militia. The FBI is probably not going to seize the GPU out of your PlayStation or Xbox. 

But the feds did once make Americans surrender all their gold to the government, so who knows?

Less conspiratorially, the government might regulate model development in a way that would make AI the exclusive domain of a handful of giant corporations.

In the US, at least. Abroad, AI development would likely continue apace…and maybe underground, too, if it can be sufficiently decentralized.

Some of this is already happening. Blockchain-based marketplaces like Akash offer permissionless access to compute, for example. Subnets on the Bittensor blockchain have been serving and fine-tuning AI models for a couple of years now.

But the holy grail of decentralized AI is training frontier models — models that match the capabilities of Anthropic and OpenAI without the need for giant data centers.

Is this possible?

Most people who know about these things seem to say it isn’t, simply because of the amount of data that needs to be moved to train a model.

A new model begins as a blueprint filled with billions of random numbers (called parameters). It’s then distributed across thousands of GPUs (or competing chips like TPUs), which perform enormous amounts of matrix math to predict what token is likely to come next given the tokens that came before it. The GPUs compare those predictions with the actual next tokens in the training data, then calculate how the parameters should change to reduce the error. The GPUs communicate these error adjustments across the network, synchronizing their calculations to update the shared model.

And then they do it again, a gazillion times — make a prediction, measure the error, adjust billions of numbers, repeat.

This makes model training fundamentally a problem of reducing latency — which means packing enormous amounts of compute, memory, networking, and power into one giant data center.

Decentralized model training is the exact opposite: training a model across scattered, unrelated resources, wherever they happen to be.

A small fringe of researchers say it’s possible. If the bandwidth issue can be overcome.

The GPUs, CPUs, and memory inside data centers are connected by thousands of miles of specialized copper and fiber-optic cabling. 

Decentralized hardware, by contrast, can only connect over the internet, which is something like 1,000 times slower. 

By one estimate, pushing a single training prompt through a medium-sized language model being trained in a data center takes roughly 0.18 seconds. Over the internet, by comparison, it could take three minutes.

At that pace, it would take thousands of years to train our first permissionless frontier model, by which time most of us will be dead. (Depending on whether centralized AI decides to exterminate us or keep us alive forever.)

In July, though, a group of open-source researchers at Pluralis.ai did it in just five weeks.

Using a novel data-compression technique, Pluralis trained an 8.6-billion-parameter model on 330 geographically dispersed, consumer-grade computers connected by consumer-grade internet — in about the same time it takes Anthropic or OpenAI to train a model in a giant data center.

Problem solved!

In theory.

In practice, even the most optimistic experts say there are still a lot of real-world obstacles to overcome.

Responding to the Beff Jezos call to arms for sovereign AI, Pluralis founder Alexander Long explained that “we don't have it yet because it’s hugely technically challenging — it requires solving fundamental research problems around low-bandwidth training and a rewrite of most layers of the training stack.”

Beyond bandwidth, which Pluralis says is not yet entirely solved, there are also the problems of heterogeneous and unreliable hardware, adversarial participants, and attracting resources.

In other words, everything that crypto is good at!

Neither crypto nor blockchains was used by Pluralis in building its decentralized model, but scaling its proof-of-concept training methods to real-world relevance would seem to require it.

Crypto has spent 15 years studying the kinds of coordination problems and incentive structures that truly decentralized model training will inevitably encounter.

A decentralized data center, so to speak, isn't going to be filled with altruistically donated GPUs, CPUs, and memory. 

Instead, contributions will have to be incentivized, with money, pseudo-money, or some form of equity (perhaps on a blockchain). And the incentives will attract nodes that are dishonest, malicious, sabotaging, cheating, and unreliable (as every blockchain does).

Due to the technical complexity, all of these challenges will be even harder with decentralized AI than they have been for decentralized money.

But the upside might be even greater? Assuming you think it’s a good idea in the first place, that is.

Because the upside of decentralized training is also its downside: You can’t turn it off. 

— Byron Gilliam

_{Brought to you by:}

Crypto's premier institutional event is arriving in Asia this October 7.

Join us for a day of top-tier conversations and networking and hear CEO & founder voices from the top protocols in the Space.