by Gaia Dempsey, CEO at Metaculus
How it all began
Last fall, we introduced a question series on Metaculus focused on US Economic Indicators and began launching monthly (or in some cases quarterly) questions on six of the most cited macroeconomic indicators.
This was an experiment — led by Metaculus moderator and team member Alyssa Stevens — to find out how these forecasts would perform, build up a dataset for analysis, and see if we could ultimately learn from the experience to create a valuable set of tools for financial and economic forecasting. We also wanted to compare Metaculus’ performance directly to others forecasting the same metrics at the same time. This is much simpler to do in financial markets than in say, space exploration, since many people forecast market movements, and few forecast when we’ll land on Mars.
While there are Metaculus questions tracking the movement of the VIX and the average returns of the S&P 500, the platform has largely focused on topics other than markets.
With our cultural DNA so connected to science and technology forecasting, we weren’t sure how the Metaculus community would respond to Very Serious traditional economic indicator questions. So, we started slow and gauged the response. Over time, we found that within the diverse global community of Metaculus forecasters, there’s room to play in lots of different areas, including market movements and macroeconomics. It’s been exciting to watch our economic indicator forecast series connect with an enthusiastic base who actively participate and look forward to their regular release.
How did Metaculus perform?
The first indicators were for September 2020, and the performance wasn’t very good — the error rates were quite high. But if you’ll allow us to throw out the September numbers and chalk it up to a “practice round,” then the economic indicator forecast quality becomes quite reasonable from October on, and in some cases even impressive. Below you’ll find our September performance compared to that of four other forecasting sites, on the six economic indicators that made up this project’s focus:
- Non-Farm Payrolls
- US GDP
- Initial Jobless Claims
- Industrial Production Index
- Total Retail Sales
- EIA Petroleum Stock
Some context about this comparison: to calculate forecast error rates, we used Mean Absolute Percentage Error or MAPE, which is one of the most popular metrics for evaluating forecasting performance. You’ll note that here we are comparing the performance of the Metaculus Prediction, in terms of accuracy, with several other publicly accessible forecast sources for the same economic predictions: MQL5, Investing.com, FXStreet, and Markets Insider. Trading Economics is also included in later analyses.
Moving on to the more exciting results, here’s a summary of the Metaculus Prediction’s performance over the 7-month period from October 2020 to April 2021, compared with the MAPE of the five other forecasting sources listed above:
With over six months of data in hand, Alyssa’s analysis found that Metaculus consistently performed at or above the others. Despite average performance on some indicators, the Metaculus Prediction excelled in others. Intriguingly, Non-Farm Payrolls is the most accurately forecasted indicator, compared to other sources. Let’s take a closer look.
Across the indicators, Alyssa found that the aggregate Metaculus Prediction placed almost all of the correct resolution values within the interquartile ranges. In addition to accuracy, there is also the dimension of time to consider. Metaculus regularly provides forecasts on economic indicators over 30 days before the others in our comparison group.
Finally, there is the question of information density. The other publicly accessible forecast sources for these indicators do not provide full probability distributions. Rather, they deliver point estimates, which by definition contain much less information. This difference makes an apples to apples comparison with Metaculus full distributions challenging (if anyone has ideas here, we’d be very glad to hear them).
To learn more and access the details of the analysis and methodology, you can read Alyssa’s full report here.
What have we learned, and what’s next?
We’ve learned that Metaculus can perform reasonably well on economic forecasts, and we think that continuing the experiment to see how much better they can get is worthwhile. As a human judgment + algorithmic team, improvement can come from two sources: 1) improved performance of Metaculus forecasters, 2) improved performance of the Metaculus Prediction’s machine learning optimizations. And we know that Metaculus forecasters have gotten better at forecasting economic indicators since this project began.
In terms of what’s next, we’re planning an expansion of this work. Since April, we have added three more indicators to the economic forecasting mix to make the dataset more robust: New Building Permits, Durable Goods Orders, and Michigan CSI Prelim.
To continue this thread of research, on June 15th we’ll launch the Trade Signal Tournament with $1,500 in dedicated prizes for the top forecasts on the nine core economic indicators in our dataset for the months of July, August, and September 2021 (which will be the one-year anniversary of the start of this project). Plus, we’ll add in some more cocktail-party-friendly questions on the future of finance, just for fun.
And there’s an additional twist to this tournament. For Forecasting Cause tournaments, we introduced the concept of the Community Shaper, a dedicated, volunteer community moderator with ties and experience in the cause area of a given tournament.
For this tournament, we’ll instead introduce the concept of a Community Trader (which will also be a volunteer position). If the goal is to be able to use the Metaculus Prediction as a trading signal, why not start right away? The Community Trader will recommend options trades based on the information they’re receiving from the Metaculus forecasting community via the tournament questions, and we’ll take the $1,500 prize pool and use it to execute the trades on a retail trading platform (to be selected). The aim is to grow the size of the prize, but just in case things don’t go as planned, we will guarantee a minimum prize pool of $1000 should the Community Trader’s recommendations result in the loss of the original $1,500.
Who will the Community Trader be? Excellent question! As we’ve done in the past for community moderators, we’ll be running an election so that the Metaculus community can vote on their choice. If you’d like to be notified when applications are open and when the election starts, you can fill out this form.
We’re excited to hear what you think about the Trade Signal Tournament, and to explore the possibility of building a trading signal together with the Metaculus forecasting community.
