The new math of new news

In 1932, the statistician and economist Alfred Cowles proved that the stock market was almost perfectly efficient.

Over five years, Cowles and his research team collected thousands of stock recommendations from asset managers at 16 financial firms, thousands more real-world investments from 24 “fire insurance companies,” and hundreds of market forecasts published by 24 financial publications.

Everything was recorded manually. All the mathematics and statistical work was done with pencil and paper. "Proper corrections" were made to account for stock splits, dividends, brokerage charges, and interest expenses.

Heroic stuff, truly.

The results — the first rigorous, statistical study of whether professional investors could reliably predict stock prices — were published in a seminal paper titled Can Stock Market Forecasters Forecast?

Spoiler alert: they cannot.

The paper assessed the financial publications by comparing their results to the hypothetical returns generated by drawing investment advice from a shuffled deck of numbered cards.

On the whole, the published forecasters underperformed the deck of cards by four percentage points per annum. The best forecasters only managed to tie the deck of cards and the worst did significantly worse.

Similarly, the thousands of stock picks from the financial services firms and insurance companies underperformed the average stock by at least 1.2%.

This was a surprising result. Even in the wake of the 1929 crash, people assumed that professional investors and forecasters had some special insight into how markets worked.

Cowles proved they didn’t. Instead, markets were shown to be entirely unpredictable — and therefore efficient.

“For almost everybody,” Eugene Fama explains, “the market is efficient in the sense that they don’t have information that’s not already built into prices.”

Nearly 100 years later, a new study suggests markets remain unbeatably efficient for a different reason: we’re not using all the information we have.

The authors of The Inefficient Pricing of News — a team of finance and machine learning researchers — find that markets are surprisingly slow to incorporate news into prices.

“We uncover evidence that markets digest news media far less efficiently than suggested by previous studies,” they write. “The inefficient pricing of news shocks translates into an asset pricing anomaly that dwarfs other well known informational inefficiencies such as momentum, reversal, and post-earnings announcement drift.”

To me, this feels as unexpected as Cowles’ result must have felt in 1932.

We now have high-frequency trading, quantitative hedge funds, natural language processing, and an industry of people dedicated to exploiting every market inefficiency. And yet, the most basic mechanism of markets — reading the news — remains woefully ineffective.

Why this inefficiency persists may be explained by the arcane math required to discover it — “numerical embedding vectors” are a long way from pencil-and-paper finance.

Embeddings, the study explains, are “translations of human language into a numerical language suitable for statistical modeling.” An embedding vector is a long list of numbers that acts like a set of GPS coordinates for meaning: words and ideas are translated into a mathematical space where similar concepts end up near each other. The numerical representation of “cat” isn’t too far from “dog,” for example, and neither is anywhere near “tax return.”

This is an old concept that’s become newly powerful.

“With the advent of LLMs,” the authors explain, “modern embeddings are capable of expressing text meaning with extraordinary efficiency using embedding vectors of only a few hundred or few thousand dimensions.” 

For the study, the researchers had an LLM decompose every news article on major US stocks into tokens (each representing a word, word fragment, number, etc.). These tokens were then converted into a 4,096-number coordinate that captured its meaning and context (similar meanings produce similar numerical patterns).

To represent an entire article, the model then averaged all those token vectors together into a single 4,096-number embedding — essentially a compressed mathematical summary of the article’s meaning.

The summaries are meaningless to the human eye. But the LLMs used to read them are now so powerful they can accurately reverse-engineer the original news article — sometimes word for word — from just that sequence of 4,096 numbers.

The authors use some additional math magic to determine how much of a news article’s embedding (or content) could be predicted by the company’s fundamentals (its sector, profitability, growth, etc.). Whatever remained after subtracting out that predictable component — the unexplained residual — was treated as genuinely new information.

They then simulated a long-short portfolio that traded exclusively based on these residual embeddings, which they refer to as “news shocks.”

Amazingly, the strategy returns roughly 30% a year.

Most amazingly, it achieves that by trading only at the end of the month.

Today’s hyperactive market, it seems, is so maladroit at incorporating new information that an LLM can read the news once a month and outperform nearly everyone. 

Probably not for long, though.

Quantitative hedge funds will presumably start using LLMs to trade based on residual embeddings. They’ll make outsized returns until enough people are doing it that the news is efficiently priced.

I’m guessing nine months.

However long it takes, markets will be efficient again, in the sense of being near-impossible to beat.

They might even become informationally efficient — in the sense of quickly incorporating all available information.

It won’t prove the wisdom of crowds, though — only that we’ve moved a little further on from pencil and paper.

— Byron Gilliam