March 4, 2021

The Forecast Forge Bet and Six Month Update

Earlier this week I tweeted the below as part of another conversation

My hypothesis with @ForecastForge is that people can make better forecasts using a fairly simple algorithm + their own domain knowledge rather than getting someone else who can make a fancy algorithm but doesn't know the domain

— Richard Fergie (@RichardFergie) March 2, 2021

I thought this would be a good opportunity to unpack my hypotheses about why Forecast Forge is a good/interesting idea. And also a good time to do a quick business update since it has been just over six months since launch.

Data Science and Domain Knowledge

Venn diagram of Data Science skills in 2017 from

Data Science has always been a combination of three things:

  1. Machine Learning
  2. Software Engineering
  3. Domain Knowledge

More recently, the growth and improvements in deep neural nets has increased the prominence of machine learning and decreased the prominence of domain specific knowledge. For example in late 2016 Google started to use a deep learning algorithm in Google Translate - work on this started in September 2016 and it was performing better than their old algorithm by November 2016. The old algorithm had been developed by language experts over the previous 10 years.

This could just be because of the Jeff Dean effect but other work on large models (e.g. GTP-2/3, Inception-V3, BERT) also shows large neural networks beating more specialised models even at tasks the specialised models were specifically designed for. Most work in this area seems to have been on NLP/text and image classification but it seems reasonable to assume that if Google/Facebook/Amazon were to turn their warehouses full of PhD’s and data centers full of GPUs onto other tasks that we might see similar results in other domains.

Part of the bet I’m making by putting time into Forecast Forge is that, for forecasting, these kind of results are a long way away.

The main reason I believe this is because looking only at the data from the metric you want to forecast is rarely enough to make a good forecast; you also need to think about context and what else is happening in the world.

This is a much harder data challenge than large language models because:

This means that it seems unlikely anyone will be able to combine data from lots of companies into the forecasting equivalent of Common Crawl or WebText2. And a large “big data” training set is an important pre-requisite for training this type of model.

I’m not saying that working hard to make a great forecasting model isn’t ever going to be worth it. I’m saying that I don’t think it is very likely that there will be a one-model-fits-all solution for this any time soon.

However, I’m pretty sure that engineers working on Google Translate thought the same thing in mid-2016 so this prediction could go wrong very quickly!

Assuming I’m right, a tool to help people make better forecasts needs to enable them to enrich their data before applying a forecasting algorithm. And the algorithm needs to be able to work with this extra information. For someone who isn’t a data scientist any tool to help them do this would end up looking a bit like a spreadsheet anyway so instead of making them work in an incomplete, half-working version of a spreadsheet I decided to go to where people are already working and put Forecast Forge in there.

This has the additional advantage that there are already lots of ways from getting data from wherever it is stored into a spreadsheet and from the spreadsheet into a presentation or Google Data Studio or wherever it goes to next in order to be acted upon.

So in summary:

  1. I think the best forecasts are not made through using better algorithms but through finding ways to give simpler algorithms more “context” around what else is happening in the world.
  2. For a non-coder the best and most flexible user interface for this will look a bit like a spreadsheet.

I’m not saying that Forecast Forge gets all of this right at the moment - but the product that does get these things right will look at bit like Forecast Forge. Hopefully that will be me!

Six Month Update

Forecast Forge launched in late August 2020. Now it is early March so I’ve been going slightly more than six months.

Firstly a few things that have gone well:

And of course, there are some things that I don’t think have gone so well.

A mixed bag of experiences over the first six months, but overall I have really enjoyed the work I’ve done on this and the interactions I’ve had with subscribers, blog readers (yes, including you!) and Twitter followers. This is not going to be my ticket to a massive yacht in the Caribbean any time soon but I am looking forward to seeing what the next six months will bring.

Never miss a post