Continuous improvement on the best-fit selection procedure of a statistical forecasting model

End of February 2023 six members of Asset | Econometrics in Tilburg started working on a case study for EyeOn for six hours a week. The goal of the case study was to investigate if the forecast model selection could be improved by applying representativeness in the best-fit procedure. The definition of representativeness was taken from the paper of Petropoulos et al. (2022). 

Representativeness

In the EyeOn best practice way of working, a rolling forecast using various models is simulated in the past – to prevent data leakage. These forecast models are then scored on their forecast error over a predefined number of periods, e.g. 12 months. To include representativeness, these future forecasts are laid back over past demand to determine whether a future forecast is also representative for demand further in the past. For each model, this representativeness score is added to the error. Thereafter, the best fit model is selected. 

On May 31st, the end results of the case study were presented. Unfortunately, on the current test dataset, the results were slightly underperforming the current way of working by a negligible difference in accuracy and bias. However, the case study nevertheless generated a lot of insights and ideas for future research.  

Insights and future research 

The most striking result was that a subgroup of product-region combinations were identified where including representativeness did result in higher forecast accuracy and lower bias. This subgroup was defined as forecasting combinations with: 

• A statistically significant trend according to current definition of EyeOn 
• A statistically significant individual and group level seasonal pattern detected, according to current definition of EyeOn 
• No intermittent item (defined as at most 6 zeroes in the last 12 months) 

Furthermore, there was a skewness to certain models when including representativeness, which were not selected as much in the current way of working.  

The next step is to test the proposed way of working by Petropoulos et al. (2022) on a different test dataset, both in the same industry as in others. Other future research includes testing adding representativeness to the selection procedure with a different model set and lastly testing with different parameter values.