The Anvil 
A place to hammer out ideas
A Forecasting Workflow
December 2, 2020Here is how I make forecasts using Forecast Forge. There are a lot of similarities with how forecasting is done by data scientists so some of this will be useful even if you aren’t a Forecast Forge subscriber.
1. Look at the data!
I cannot stress enough how important it is to plot the data! For forecasting problems it isn’t even that complicated; putting the date along the x-axis and the metrics you are interested in on the y-axis will be enough in most cases.
Read more
Forecasting the Effect of Lockdowns
November 23, 2020A Covid-19 vaccination is now clearly visible on the horizon but there are still significant uncertainties about when exactly this will become available and what will happen in the meantime.
In the UK I think there is likely to be a period of national lockdown in January. And maybe more in the Spring too. Other countries in Europe are in a similar situation and who knows what will be going on in the USA by then?
It is important for businesses to be able to make good estimates for what their demand will look like at these times. I will show you how to do this in Forecast Forge.
Read more
Cumulative Forecasts and Measuring Impact
November 3, 2020If you have a really good forecasting model that has, historically, produced good results then you can use it to estimate the effect of a change.
For example, physicists can use Newtonian Physics to forecast where planets will be in the sky and things like solar eclipses. If these predictions suddenly started being wrong (after they have been right for hundreds of years) people would conclude that something fairly drastic had changed.
Similarly, if you have made a good forecast in Forecast Forge but then it starts predicting badly this could be because something has changed. You can turn this reasoning around and also say “if something has changed then the forecast won’t work as well.”
You can watch a quick video of me demonstrating how you can do this in Forecast Forge. Or read on below for more details and commentary.
Look at this example of daily transaction data:
Read more
Extending Prophet for fun and profit - Part 2
October 27, 2020This post is part of a series on extending the Prophet forecasting model so that it can do other things. You should read part one on the default Prophet forecasting model first.
What Prophet Can’t Do
For me, one of the most useful features of Prophet is how easily you can add regression features to the model. This enables you to provide information to the machine learning model about things like the dates when email campaigns were sent or when your PPC budget increased; and you can use your marketing calendar to add information about when these things will happen in the future too.
Being able to do this in a spreadsheet is one of the main things that makes Forecast Forge great.
Read more
Extending Prophet for fun and profit - Part 1
October 19, 2020Understanding the original implementation
Prophet is a general purpose forecast algorithm developed by Facebook. It is the main algorithm used behind the scenes for Forecast Forge (I’ll go into why I picked it some other time).
At the core of the Prophet code is a model written in a programming language called Stan. I am lucky enough to have some familiarity with Stan from other projects.
This model is where the interesting stuff happens; the rest of the Prophet code (R or python) is just around scaling and preparing data for input into this model and extracting the results at the end.
I’m going to assume for this that you are familiar with the process of using Prophet in R or Python. If you aren’t then read the getting started guide and fit a few forecasts. After that this all should all make a bit more sense.
We need to talk about Stan
Suppose you have some process that you are interested in and that this process is complicated enough that you want to model it as something random or stochastic. Calculating how long a car traveling at 70 miles per hour takes to go between two towns doesn’t fit into this category (it is too easy to model without) but something like whether or not a user converts on a website is; it is just too much to take into account all the variables like their mental state, what they had for lunch, whether their neighbor is slowing down your site by watching Netflix at the same time as they surf and all the rest.
Stan let’s you describe how this random process works (called a “generative model”) and then does some clever stuff in the background to figure out the parameters of the model. For example, a simple model for website conversion might be “the user converts at random with probability p and does not convert with probability 1-p”. Then Stan does Bayesian inference with this model and your data to tell you what the likely values of p are.
You might think this is a lot of work for something like estimating a conversion rate and you’d be right. Stan is more useful when the model gets more complicated and I will show you an example of this in this post.
The Prophet model
In this post I will go through the existing Prophet Stan model. Then, in future posts I will discuss how it can be changed and modified for different things.
You can view the current model on github.
Stan is kind of an old fashioned language in that variables need to be declared in advance and there is very little flexibility in the order of code. Rather than start at the top and work down we will start at the end - at the absolute key part - and then work out.
The model
The most important part of the model is on line 136:
y ~ normal(
trend
.* (1 + X * (beta .* s_m))
+ X * (beta .* s_a),
sigma_obs
);This describes how the y values (i.e. the thing we are trying to forecast) vary depending on the things they depend on.
If I rewrite it like this it should be easier to see what is going on:
y ~ normal(
trend
.* multiplicative_regressors
+ additive_regressors,
sigma_obs
);So the y values are normally distributed around a trend, multiplied by some multiplicative_regressors and added to some additive_regressors. The variance is sigma_obs; a small value of sigma_obs will mean the model fits the y values used in training very closely so the predictive interval for the forecasted values will be narrower. sigma_obs is estimated from the data during model fitting.
Let’s unpack the regressors a bit more. Both of them have the form X * (beta .* s_?) where s_? is either s_a or s_m.
The s_? variables are provided by the user when they specify for each regressor whether it is multiplicative or additive. Each s_? is a vector with the same length as the number of regressors. For regressor number i, s_a[i] is 1 if it is an additive regressor and s_m[i] is 1 if it is a multiplicative regressor.
The beta values are regression coefficients. In Stan .* means elementwise multiplication so element i in the vector beta .* s_a is beta[i] * s_a[i].
Because of how we defined s_a earlier this means it is zero unless it is an additive regressor. So X * (beta .* s_a) means only the additive regressors are included in the additive component. The same goes for the multiplicative regressors as s_m.
X is a matrix containing the values of the regressor variables; again, the user provides this before fitting starts.
If this looks very similar to the maximum likelihood method for doing a regression that is because it is! One of the things that amazes me about Prophet is that it can work so well across so many different things with such a simple model.
I guess “just use regression!” is a meme for a reason.
Read more