Retail organizations today must strive to satisfy the unique demand for each of their customers. Gone are the good old days of the mass market where a single assortment standard pricing and a single “average location” forecast would satisfy consumer demand in all stores. Often, predicting demand for a single item in a single store can be a difficult proposition. Forecasting demand for all stock-keeping units (SKUs) across all stores and all geographies is a much greater challenge. Forecasts need to adjust to changing demand and quantity patterns, handling seasonality, including difficult-to-predict demand for slow-moving goods. To be able to do that retailers needs to store data for points-of-sale, events, promotions, price, weather and other effect parameters which can have an impact on demand. By applying big-data advanced analytics to determine the net effect of promotions and price changes on whole categories retailers can evaluate cross-effects between products, forecast new product sales and account for lost sales to generate a demand forecast. This helps a retailer improve their in-stocks and reduce out-of-stocks. Complex demand forecasting models can be created using a number of factors like sales figures, basic economic indicators, environmental conditions, etc. If correctly implemented, a data warehouse can significantly help in improving the retailer’s relations with suppliers and can complement the existing supply chain application.
Within an organization, generating forecasts is an important first step in many planning and decision-making processes. Typically, forecasting is performed on a regularly repeating basis for a wide variety of planning purposes. For example, future demand for products and services may be forecast in order to support production planning, marketing activities, resource scheduling and financial planning. Forecasters often follow the same iterative process with each forecasting and planning cycle.
The process usually involves generating updated forecasts based on the most recent data, reconciling forecasts in a hierarchical manner, identifying and remedying problematic forecasts, adding judgmental overrides to the forecasts based on business knowledge, and publishing the forecasts to other systems or as reports.
Upon completion of the forecasting process, the planning process determines what actions the organization will take in light of the forecasts. Planning processes not only take into account the forecasts but also the constraints upon the business and overall corporate goals. Much of an organization’s time and effort is spent in the planning process. Improving the reliability of statistical forecasts that feed these processes can result in huge rewards. Improved forecasting often leads to greater operational efficiency, reduced expenses and increased profits.
Although forecasting is an important function, many organizations rely on a relatively small number of forecasters to generate large numbers of statistical forecasts. Given the relatively small number of forecasters in most organizations, a large degree of automation is often required in order to complete the forecasting process in the time available each planning period. There are multiple planning functions within an organization that the forecasts can serve. Examples include forecasting future demand for production planning purposes, supporting marketing plans (promotion planning), resource planning, financial planning and reporting. Forecasting managers typically follow an iterative process for each forecasting and planning cycle. The steps usually include obtaining the latest data, generating updated forecasts, fixing problem forecasts, conferring with other internal or external parties about the forecasts, adding ad hoc adjustments based on business knowledge, obtaining executive review and change approval, and publishing the forecasts.
Forecasting managers may reside in a centralized department that serves other departments, or they may be embedded in the individual business units or departments (e.g., supply chain, finance, marketing or sales). As the forecasting function tries to address the more tactical/operational decision level (e.g., support daily and weekly planning processes), forecasters will need to rely on more automated forecasting functionality because of a lack of bandwidth (both time and analytic modeling expertise).
Although many people use the word “forecast” to imply only prediction, a forecast is not one number for each future time period. Forecasting looks at historic behavior. The basic assumption is that the future is a repetition of the past. Controllable historic events (like promotions) and uncontrollable historic events (like the SARS outbreak) can be taken into account during modeling.
However, when creating forecasts the uncontrollable future events are extremely challenging due to their nature (they are completely unknown to us). Measuring the impact of uncontrollable future events is the task of risk management. Techniques like Value at Risk are used to come up with an idea of how much risk we are willing to take in our decision process.
Forecasting situations vary widely in their time horizons, factors determining actual outcomes, types of data patterns, and many other aspects. To deal with such diverse applications, several techniques have been developed. According to Makridakis (1997) these fall into two major categories: quantitative and qualitative methods.
- Sufficient quantitative information is available
- Time series: predicting the continuation of historical patterns such as the growth in sales or gross national product (time series data)
- Explanatory: understanding how explanatory variables such as prices and advertising affect sales (cross-sectional data)
- Combination of both approaches. Models which involve both time series and explanatory features are for example dynamic regression models and state space models
- Can be applied when:
- Information about the past is available
- This information can be quantified in the form of numerical data
- It can be assumed that some aspects of the past pattern will continue into the future (assumption of continuity)
- Little or no quantitative information is available, but sufficient qualitative knowledge exists
- Predicting the speed of telecommunication around the year 2020.
- Predicting how a large increase in oil prices will affect the consumption of oil
- Examples are judgmental and consensus forecasts using techniques like the so-called Delphi approach
- No information is available:
- Predicting the effects of interplanetary travel
- Predicting the discovery of a new, very cheap form of energy that produces no pollution
- Underlying process is entirely random
- Predicting numbers in a lottery
There are special areas like new products forecasting (i.e. no historic data is available), which does not fit into this classification. There are special techniques available like diffusion models and clustering approaches for this kind of task. In general the classification holds, though.
Categories of forecasting methods
Qualitative vs. quantitative methods
Qualitative forecasting techniques are subjective, based on the opinion and judgment of consumers, experts; they are appropriate when past data are not available. They are usually applied to intermediate- or long-range decisions. Quantitative forecasting models are used to forecast future data as a function of past data; they are appropriate when past data are available.
Naïve forecasts are the most cost-effective forecasting model, and provide a benchmark against which more sophisticated models can be compared. For stationary time series data, this approach says that the forecast for any period equals the historical average. For time series data that are stationary in terms of first differences, the naïve forecast equals the previous period's actual value.
Time series methods
Time series methods use historical data as the basis of estimating future outcomes.
- Moving average
- Weighted moving average
- Kalman filtering
- Exponential smoothing
- Autoregressive moving average (ARMA)
- Autoregressive integrated moving average (ARIMA)
Causal / econometric forecasting methods
Some forecasting methods try to identify the underlying factors that might influence the variable that is being forecast. For example, including information about climate patterns might improve the ability of a model to predict umbrella sales. Forecasting models often take account of regular seasonal variations. In addition to climate, such variations can also be due to holidays and customs: for example, one might predict that sales of college football apparel will be higher during the football season than during the off season. Some forecasts take account of past relationships between variables: if one variable has, for example, been approximately linearly related to another for a long period of time, it may be appropriate to extrapolate such a relationship into the future, without necessarily understanding the reasons for the relationship. Causal methods include:
- Regression analysis includes a large group of methods for predicting future values of a variable using information about other variables. These methods include both parametric (linear or non-linear) and non-parametric techniques.
Quantitative forecasting models are often judged against each other by comparing their in-sample or out-of-sample mean square error, although some researchers have advised against this.
Judgmental forecasting methods incorporate intuitive judgments, opinions and subjective probability estimates. Methods used for these types of forecasts are:
- Composite forecasts
- Cooke's method
- Delphi method
- Forecast by analogy
- Scenario building
- Statistical surveys
While the area of decision making is very broad, the term is used for the specific process of exploring how different decision options will influence the plan. During the decision making process the aim is to select the most suitable single number from the forecasted values – taking risk measurements into account. For example: deciding to go for the values of the lower confidence limit instead of the prediction value (mean or median), is a part of the decision making process.
There are more advanced ways to come up with sensible decisions. In particular there are techniques available which involve specific algorithms:
- This technique is in particular valuable for defining fact based plans. It requires advanced forecasting models (like a dynamic regression model) which consist of both “inputs” (like business drivers and calendar events) and an “output” (which is the variable to be forecasted)
- Goal seeking algorithmically varies the future values of the “inputs” in order to determine the values that achieve a certain goal (profit, revenue, or cost goal) based on the forecasts
- This technique is potentially useful for understanding how inputs need to be modified in order to achieve a certain goal
- Example: “What are the combinations of sales price and advertising expenditures that achieve a specified sales target?”
Scenario analysis (What-If analysis):
- Again this technique requires advanced forecasting models (like a dynamic regression model) which consist of both “inputs” (like business drivers and calendar events) and an “output” (which is the variable to be forecasted)
- This time the aim is to figure out the impact of changes in the inputs on the output. In scenario analysis the user modifies the future values of the inputs to specific values and then evaluates the effect on the forecasts
- Example: “What happens to the predicted sales numbers if the organization increases the sales price and decreases the advertising expenditures?”
- This technique requires advanced forecasting models (like a dynamic regression model) which consist of both “inputs” (like business drivers and calendar events) and an “output” (which is the variable to be forecasted), also
- The aim is to come up with the best possible combination of inputs. Optimization algorithmically varies the future values of the inputs to find the optimum of an objective function (profit, revenue, or cost function) based on the forecast model
- Example: “What is the optimal sales price and advertising expenditure combination that maximizes profit?”
Other areas which need to be considered in decision making are techniques for market research, management science, and quality management. While all of these provide valuable insights to the decision making process they are not considered here. The decision making process requires human intervention, but it can be supported by software. Upon completion of the forecasting process, the planning process determines what actions the organization will take in light of the forecasts. Planning processes not only take into account the forecasts but also the constraints upon the business and overall corporate goals. Much of an organization’s time and effort is spent in the planning process. Improving the reliability of statistical forecasts that feed these processes can result in huge rewards. Improved forecasting often leads to greater operational efficiency, reduced expenses and increased profits.
There are multiple planning functions within an organization that the forecasts can serve. Examples include forecasting future demand for production planning purposes, supporting marketing plans (promotion planning), resource planning, financial planning and reporting. Forecasting managers typically follow an iterative process for each forecasting and planning cycle. The steps usually include obtaining the latest data, generating updated forecasts, fixing problem forecasts, conferring with other internal or external parties about the forecasts, adding ad hoc adjustments based on business knowledge, obtaining executive review and change approval, and publishing the forecasts.
Planning is a fairly diversified task, which happens on different levels: strategic planning, tactical planning and operational planning. In general it is concerned with sketching out a pathway for specific actions which a company or an individual are going to make. While forecasting is more concerned with what will happen, planning deals with what should happen, given the alignment of the resources to make it happen.
The relationship aligns different activities and how they interact with each other:
|Forecasting||An unbiased "best guess" of what will happen – provided that “history repeats itself”, together with some estimate of the uncertainty.|
|Risk management||Measuring the impact of uncontrollable future events|
|Decision Making||Exploration of how different decisions will impact the plan.The process of coming up with a decision.|
|Planning||What should happen, given the alignment of resources to make it happen|
Sometimes management disagree when the forecasts start to differ from their plans (especially in an unfavorable direction) but that is precisely the point of the forecast - to tell you when you are off plan. When you know a gap exists, you can at least take action (either adjust the plan to more accurately reflect the forecast, or else re-align your resources (advertise, change pricing, etc.) to cause the forecast to become closer to the plan). This leads to decision making activities in turn.
New product forecasting
Forecasting new products is a very difficult, but essential, task for any retailer. There are three issues with forecasting new products – response to initial price, predicting sales volume with no history and modeling cross-effects. Based on similar products within the share group, one can determine how the consumer responds to price and promotions for products in the share group, and it also knows how cross-effects with other products in the group affect it, so it will be able to generate a predicted sales curve for the new item.
Identify intermittent/slow-moving items
This is a feature of most retailers and is an important consideration when forecasting demand. Some Forecasting software has functionality to pool data across stores or up the product hierarchy in order to find sufficient and relevant data with which to generate a forecast.
Consumer response to price and promotion
In order to understand how consumers will respond to price changes, promotional and marketing activity, analytically-mature retailers can analyzes share groups defined by product and consumer attributes in order to calculate elasticity at a store level. When the offering has quantified how the consumer will respond, it can calculate the impact on future demand.
When you have in-depth analysis of past performance combined with plans and forecasts of future customer demand, you can more accurately allocate and restock merchandise across channels and stores. Truly understanding customer demand patterns, not just what was purchased, but what those patterns reveal about future potential, enables you to send the correct assortments, size and case-pack distributions to the correct stores. Daily price, promotion and markdown optimization ensures that items are priced for optimal profitability, both preseason and in-season. Space automation and optimization ensure that departmental sales and profit per square foot are maximized, and that products are given the correct inventory and space on the shelf. Optimized fulfillment ensures that products are allocated or replenished according to demand. Accurate analysis also results in a more efficient use of manpower in picking, packing and shipping the first wave of product while minimizing additional expenses. In-store and customer-facing activities rely on a multitude of support functions behind the scenes, all of which must also be optimized. For example, now that analytics have given you an accurate forecast of demand, by hour, by day, by location, by promotion and by price change. This knowledge must guide decisions for inventory replenishment, as well as for staffing on all store floors, catalog call centers and fleet crews delivering orders from distribution center to stores.