Thursday links: Could prediction markets do more than predict?

Kalshi: Markets beat models

A Kalshi research report reviewing the recent performance of prediction markets has determined they did a great job. In the midterms, Kalshi’s market-based probabilities were more accurate than either traditional polls or the forecasting models based on polls.

“Kalshi matched or exceeded polling averages at every horizon where they existed, and on the 9 am Election Day snapshot recorded the lowest Brier score of all forecasts compared,” the report concluded.

(In related news, I have reviewed my recent job performance and determined I deserve a raise.)

“Brier score” is a statistical method of judging the accuracy of probabilistic forecasts and lower scores are better. They’re calculated as the squared difference between the forecasted probability of an event occurring and the event’s actual outcome.

This penalizes overconfidence. For example, assigning a 90% probability to something that doesn’t happen gives you a high Brier score: (0.9 − 0)² = 0.81.  Assigning a 90% probability to something that does happen gives you a very low one (0.9 − 1)² = 0.01. 

Kalshi says its Brier score is the lowest:

A perfect Brier score is 0 (no difference between predictions and outcomes), so 0.014 is very, very good.

On that metric, at least, Kalshi’s markets beat Nate Silver’s models.

Silver: Models beat markets

Nate Silver thinks Kalshi is giving itself too much credit: “I’m finding myself a little annoyed if prediction markets are trying to take a disproportionate share of the credit as opposed to ‘the industry’ broadly.”

He has an idea about how the credit should be apportioned instead:

This ordering seems intuitively correct because polls are an input to forecasting models, and forecasting models are an input to prediction markets. But neither polls nor models consider prediction markets.

To put prediction markets first, as Kalshi implicitly does, you have to believe in an unquantifiable wisdom-of-crowds effect that might make markets more than the sum of their parts (polls and models).

It’s possible! Betting markets were weirdly accurate before polling was invented, for example, so perhaps there is some magic happening here.

But Silver saw no sign of it in the midterms. Instead, he saw prediction markets following his model.

“Models like FLIPR unambiguously had a better pre-election period,” Silver says. “Our model arrived at the right answer faster, leading the markets by days to weeks.”

His evidence is easier to understand than Brier scores:

“Prediction market prices converged toward FLIPR, and not the other way around,” Silver says — referring to both the overall Senate odds (shown above) and individual Senate and gubernatorial races.

Silver does give prediction markets credit for reacting to news so quickly on election night. But in the run-up to the election, they reacted to the polls more sluggishly. “The prices they wound up with on Tuesday morning, we often had days to weeks sooner,” Silver concludes.

“Days to weeks” is a lot!

Midterms pre-mortem

A pre-mortem on prediction markets’ midterms performance finds that — contra Silver — polls and models were not their primary driver: “Statistical tests showed that the prediction markets were driven primarily by their own internal trading dynamics rather than reacting directly to incoming daily polls or news headlines.”

The dynamics didn’t always make sense.

For example, the paper notes that prediction market prices for individual House seats implied a 97%+ chance of a Democrats winning control. But the single contract specifically tracking Democrats’ odds to win the House traded around 90–91%.

(The single-contract may have been anticipating a polling error, the author speculates.)

Nevertheless, prediction markets and polls ultimately ended up in about the same place. The paper calculates, for example, that polls implied a 8.3% margin for Democrats nationwide — almost exactly the 8.4% implied by state-level prediction markets.

Both were pretty far off, though. The actual margin was 13.6%.

The betting bankwagon

People once thought that betting markets could not just predict elections, but decide them.

Writing for the New York Times in 1944, famed political correspondent Arthur Krock reported widespread concern that betting markets might do more than just predict how people will vote. They thought they might decide how people vote.

A bandwagon effect!

People thought it might even decide an election.

Down to 2 to 1.

President Roosevelt was a 3 to 1 favorite at the time, which gave him a 75% chance to win re-election. But people worried that a decline to just 2 to 1 could set a dangerous bandwagon effect in motion.

It’s hard to imagine now. But politicians once took the possibility so seriously that losing candidates used campaign funds to try to kickstart the process.

Good luck building that into your model, Nate Silver.

— Byron Gilliam

_{Brought to you by:}

My OnChain Net Yield Fund (MONY) is J.P. Morgan Asset Management's first tokenized money market fund, built for a new era of liquidity.

MONY, our tokenized money market fund on a public blockchain, pairs MMF relative stability and yield with onchain speed and transparency for liquidity management.