Friday charts

When physicist Suvrat Raju saw the results of a new study he co-authored testing AI’s ability to solve physics problems, he was shocked: “I couldn’t sleep properly for a couple of nights,” he says.

Raju, known for his work on black holes, began by asking fellow physicists to develop unpublished problems from their particular sub-fields of expertise. He wanted to see how good standard LLMs like Claude and ChatGPT would be at solving them.

They were shockingly good.

“The best-performing agent, GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem.”

$3!

That is considerably less than the cost of hiring a human to solve the same problems.​​ The paper estimates it would take a physicist a full day to solve the same problems — and considerably longer if they don’t happen to be an expert in the relevant sub-field.

“Frontier agents can now solve verifiable problems in theoretical physics with great accuracy,” the study concluded. “Not only does their breadth exceed that of any individual, but their calculational abilities now match those of world experts even in specialized domains.”

Even the final grade of 98% understates the AI’s achievement. The remaining 2% came from a trick question, included to see if the machines would bend the rules they were given (they did not).

But that’s not the truly shocking part. 

Raju adds that the AIs initially appeared to have solved only 80% of the problems the physicists had challenged them with. As it turns out, there were problems with the problems:  

As we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98%.

In other words, the physicists who thought they were testing AI were in fact the ones being tested. By AI.

Economist Alex Tabbarok sees this as a “eureka” moment for scientists who did not expect the machines to catch up to them so quickly.

(Tabbarok uses a more emphatic term, but this is a family-friendly newsletter, of course.)

As illustrated in a classic post by Tim Urban on waitbutwhy.com, many scientists seem to have overestimated both how much smarter humans are than other animals and how much smarter the most intelligent humans are than an average one.

This may have given them a false sense of security:

What we now seem to be learning, however, is that 1) human intelligence is not as special as we like to think it is and 2) the difference between the smartest and dumbest humans is a lot smaller than we imagine.

As a result, machine intelligence has progressed from ant-level to Einstein-level far faster than scientists might have expected. 

“In the grand spectrum of intelligence, all humans, from the village idiot to Einstein, are within a very small range,” Urban wrote way back in 2015. “So just after hitting village idiot level and being declared to be AGI, it’ll suddenly be smarter than Einstein and we won’t know what hit us.”

A decade later, it may finally have hit us. Physicists are probably the smartest humans in the world and AI is suddenly correcting their mistakes.

This is excellent news, I think, because physics is how we understand our world, and the better we understand it, the better we can make it.

As with Raju, it might take physicists a few days to recover from the shock of seeing AI correcting their work. But once they do, they’ll be very busy.

“Our results show that modern agents have matured into powerful tools that professional physicists can use to expand the scale and scope of their work,” the paper concludes. “This is likely to accelerate scientific discovery in the near future and might lead to breakthroughs in longstanding problems.”

We might be about to make a few thousand years of progress.

Let’s check the charts.

Checking the mathematician’s math:

In a study of nearly 800 mathematics papers, an OpenAI LLM (GPT 6.1 Sol) found major errors in 6% of the mathematicians’ work and minor mistakes in another 30%. But AI appears to be fixing this. Only 1.5% of the papers written in the last two months were found to have errors (presumably because they were being checked with the latest LLMs).

The rent is pretty damn high:

Data from Compute Desk shows that the cost of renting Nvidia’s B300 GPU — the most recent model — has risen for something like 17 days in a row. 

The old GPUs:

Even more impressively, rental rates for Nvidia’s H100 GPU — now four years old — are rising, too. Compute Desk concludes that “Nvidia hardware is an appreciating asset.” It’s not supposed to be.

We’re gonna need a bigger chart:

Ark Invest calculates that only 32 companies in US history have added more than $30 billion in revenue (inflation adjusted) over 12 months. None have done it as quickly as Anthropic, which is the dot way off to the right above. By this measure, Anthropic has just had “the most remarkable 12 months in business history,” an Ark Invest analyst said.

The big four:

Ark Invest estimates that, combined, the big four providers of compute — OpenAI, Anthropic, Gemini, and xAI — have grown revenue 400% over the past year.

GLPs > GPTs:

As remarkable as the revenue at OpenAI and Anthropic has been, revenue from the two biggest weight-loss drugs is far bigger.

Peaked?

Ramp notes that, on a per-employee basis, spending on compute remains below what they were three months ago.

The tokens, though!

The tokens being consumed continue to skyrocket. 

 But prices down:

The cost of tokens is falling as users increasingly route their consumption to cheap open-source models.

Different this time:

In the dotcom bubble (left), share prices raced far ahead of company earnings. This time (right), earnings are rising faster than share prices.

Still, people are worried we’re in a bubble again — and we could well be. Earnings bubbles are a thing, too.

But if AI sneaks up on investors the same way it did on scientists, we might just be getting started.

Have a great weekend, scientific readers.

— Byron Gilliam