This article explains forecast quality in the Demand Forecasting Machine Learning Solution Template in the AI & Analytics Engine.
What is Forecast Quality?
In the AI & Analytics Engine, forecast quality is a crucial metric used to evaluate the accuracy of predictions generated by a forecasting model. This metric is essential for businesses that rely on demand forecasting to make informed decisions about inventory, production, or resource allocation.
Forecast quality is expressed as a percentage, ranging from 0% to 100%. A higher percentage indicates better forecast accuracy, with 100% representing a perfect forecast where the predicted values match the actual values without any errors. This metric provides a straightforward way to assess how well a forecasting model performs.
How is Forecast Quality calculated?
The calculation of forecast quality in the Engine is derived from the Symmetric Mean Absolute Percentage Error (sMAPE). sMAPE is a measure that compares the absolute difference between actual and forecasted values to their total magnitude. The formula for sMAPE is as follows:
Once sMAPE is calculated, forecast quality can be determined using the following formula:
sMAPE provides a balanced view of forecast accuracy by better normalizing the error, as it takes into account both the actual and forecasted values. This makes it easier to compare across different scales and units, while being bounded between 0% and 100%. A lower sMAPE value indicates a smaller error, leading to a higher forecast quality percentage.
Forecast quality, derived from sMAPE, is a direct indicator of how reliable a forecasting model is. For instance, if the sMAPE value is 10%, the forecast quality would be 100% − 10% = 90%. High forecast quality shows that the model's predictions are closely aligned with actual outcomes, making the model reliable for future decision-making.
Forecast quality result in evaluation report
💡Forecast quality is used as the criterion to identify the best model in demand forecasting apps. For more information about how we evaluate forecasting models in the Engine, see What is backtesting
The Engine recommends the model with the highest forecast quality