Announcing Metaculus’s Million Predictions Hackathon!

Metaculus recently achieved a big milestone: 1,000,000 predictions on over 7,000 questions! This unique dataset represents a rich, quantitative set of perspectives on the future of humanity, with particular focus on technological and scientific progress, and events with potential global impact. It combines the breadth of insights from tens of thousands of people globally, and the depth of calibrated judgments from domain experts and experienced forecasters.

To celebrate this milestone, we’re inviting the forecasting community to participate in The Million Predictions Hackathon from December 3rd through 11th. The hackathon will be virtual, supported by the Metaculus engineering team.

This is an open invitation to forecasters, data scientists, economists, engineers, and analysts to probe this rich dataset and distill insights that can advance the state of forecasting science and sharpen the accuracy of Metaculus’s aggregate forecasts.

We expect to accept the top 25 applications in order to maximize the support we can provide participants. You can apply as an individual or as a team here.

A $10,000+ prize pool will be awarded for the projects that contribute the most to increasing the strength, impact, and capabilities of the Metaculus forecasting ecosystem.

Categories

Participants may be individuals or teams pursuing projects in the following topic areas:

Aggregation and Accuracy — Projects in this category improve the precision and calibration of the Metaculus Prediction by conducting relevant analyses into questions such as whether the Metaculus Prediction is more or less accurate than average in certain categories, and how prediction accuracy evolves over time. We will advise participants on approaches to backtesting new aggregation methods as they seek novel approaches to improving predictive accuracy.

Evaluating Individual Forecasters — The 1M predictions dataset provides a rich testbed with information about the performance of individual forecasters. This category focuses on the identification of novel scoring systems that recognize and reward top forecasters. What are the forecasting patterns of the most skilled forecasters and how can we spot them? What are early indicators that a new forecaster might outperform their peers? Which additional metrics should be used to gauge forecasting skill? How does forecasting skill transfer across domains?

Open Inquiry — In this category, participants conduct open explorations into the Metaculus dataset and provide independent and innovative analysis. Are there new, valuable metrics that should be exposed to forecasters or the forecasting community? How should the Metaculus Track Record be improved? Such Open Inquiry projects could involve conducting analysis that generates new insights, building dashboards or visualizations, or answering key questions about the application of forecasting to solving problems that would help forecasters, decision-makers, and the public.

Projects will be evaluated by a panel of judges made up of Metaculus staff and advisors on the basis of their originality, creativity, quality, documentation, and utility. Prizes will be awarded to the top three projects in each category:

• First place: $2000 (3)
• Second place $1000 (3)
• Third place $500 (3)

The top projects will be featured on Metaculus and may be considered for incorporation into the forecasting platform as appropriate.

Logistics

Metaculus will provide extensive support to all hackathon individuals and teams, so we have an application process to make sure you’ll have what you need. Applications are due by November 23rd (though earlier applications are encouraged) and we will notify all applicants of their status by November 26th.

By applying for and participating in this event, you’ll become a trusted Metaculus Data Developer, with certain data access rights. You’ll be asked to sign a confidentiality agreement, along with a commitment to using Metaculus forecasting data solely for research purposes and work connected to the hackathon.

During the hackathon, we’ll support participating individuals and teams with a dedicated Discord server, a virtual gathering space, Metaculus team office hours, and daily presentations on relevant forecasting and data science topics.

Schedule (with more updates to come):

• Saturday, December 3: Kickoff and team creation (3 hours)
• Sunday, December 4: Hacking day
• Monday, December 5 — Friday, December 8: Optional daily talk and office hours (work on projects if you’re interested and able)
• Saturday, December 10: Hacking day
• Sunday, December 11: Final presentations and celebration (2 hours)

We expect to accept ~25 participants and will do our best to schedule events to accommodate as many time zones as possible. We encourage participants with a wide range of backgrounds to apply. Even if you’re unable to attend all sessions, we still invite you to apply and let us know your availability.

Next Steps

If you are interested in participating, please fill out this application form.

If you have questions, please comment below or message anastasia [at] metaculus.com.

We look forward to this collaboration with the forecasting community as together we advance the state of forecasting science!