Multiple regression and forecasting

• yt is the variable we want to predict: the “response” variable

• Each xj,t is numerical and is called a “predictor”. They are usually assumed to be known for all past and future times.

• The coefficients β1, . . . , βk measure the effect of each predictor after taking account of the effect of all other predictors in the model. I.E., the coefficients measure the marginal effects

Some useful predictors for linear models

Dummy variables

Fourier series

 

We need fewer predictors than with dummy variables, especially when  is large.

• Particularly useful for weekly data, for example, where m≈52. For short seasonal periods (e.g., quarterly data), there is little advantage in using Fourier terms over seasonal dummy variables.

What is K

K is the parameter we set to specify how many pairs of sin and cos terms to include.

• , m = seasonal period

e.g., if we set model(TSLM(Beer ~ trend() + fourier(K = 2))) , then that means the model has 2 pairs of sin and cos

What if only the first two Fourier terms are used ( and )?

The seasonal pattern will follow a simple sine wave.

A regression model containing Fourier terms is often called a harmonic regression because the successive Fourier terms represent harmonics of the first two Fourier terms.

recent_production %>%
  model(
    f1 = TSLM(Beer ~ trend() + fourier(K = 1)),
    f2 = TSLM(Beer ~ trend() + fourier(K = 2)),
    season = TSLM(Beer ~ trend() + season())
  )

R interpretation

 

R Code for Fourier term

fit <- aus_cafe %>%
  model(
    K1 = TSLM(log(Turnover) ~ trend() + fourier(K = 1)),
    K2 = TSLM(log(Turnover) ~ trend() + fourier(K = 2)),
    K3 = TSLM(log(Turnover) ~ trend() + fourier(K = 3)),
    K4 = TSLM(log(Turnover) ~ trend() + fourier(K = 4)),
    K5 = TSLM(log(Turnover) ~ trend() + fourier(K = 5)),
    K6 = TSLM(log(Turnover) ~ trend() + fourier(K = 6))
  )

Interpretation of the plot

• When , it's only going to be 11 coefficients. This is because the sin and cos for the last coefficient is redundant. You've got the sine of pi times t is equal to 0.

• When , the seasonality is actually just a sine wave.

• When , it gets a smaller AICc, and my pattern is now starting to a bit more complicated by adding in one more harmonic.

• There is a tiny difference between and . The AICc of 6 is not as good as 5. So the best model is

• As we add extra terms, the shape of the seasonality gets more complicated. It tries to match what's going on in the data so that's the genius of fourier terms!

• WE can model any type of periodic pattern by having enough terms

Intervention variables

We regress sales on our usual variable ad expenditures and we created a dummy variable to capture whether the sales are taking place during the intervention or not.

We increased our ad expenditures during red framed period of time, for example start giving coupons.

Our dummy variable takes a value of 1 whenever the time period or sales are occurring in a period with our intervention.

The value of show us the effect of that particular intervention on the sales value as compared with the the omitted category that is without the intervention

Case 1: Spikes

We have the output variable against time.

• Our variable was having a slope of and then suddenly we intervene in the market in the mid-period where our ad expenditures went up. Suddenly, we saw that the value of the sales variable went up as well and then the increase in the ad expenditures last until our sales went back to our normal sales.

• When we say normal, here we assume

• The slope changes during the intervention with which the dummy variable = 1.

Case 2: Steps

Slicing our data into parts, we will denote the time period before the intervention by 0 and the period with the “permanent ” intervention denoted by 1.

Case 3: Change of slope

By increasing our ad expenditures, our happens before the intervention

After the intervention is much steeper .

1. Case 1 and case 2 are linear model, whereas case 3 is non-linear.

Distributed lags

Residual diagnostics

1. spot outliers.

2. decide whether the linear model was appropriate.

1. Scatterplot of residuals εt against each predictor xj,t



If a pattern is observed, there may be “heteroscedasticity” in the errors. This means that
1. the variance of the residuals may not be constant.
2. the relationship is nonlinear

2. Scatterplot residuals against the fitted values yˆt

If a plot of the residuals vs fitted values shows a pattern, then there is heteroscedasticity in the errors. (Could try a transformation.)

If this problem occurs, a transformation of the forecast variable such as a logarithm or square root may be required

Selecting predictors and forecast evaluation

ADJUSTED R^2

Akaike’s Information Criterion (AIC)

where L is the likelihood and k is the number of predictors in the model.

• AIC penalizes terms more heavily than R^2.

• Minimizing the AIC is asymptotically equivalent to minimizing MSE via leave-one-out cross-validation (for any linear regression).

AIC has a caveat because when T (# obs) is too low, the AIC tends to select too many predictors.

Bias-Corrected AIC (AICc)

Bayesian Information Criterion (SBIC / BIC / SC)

• We don’t use BIC because it's not optimizing any predictive property of the mode

Cross-validation

Traditionally you've got training sets and test sets time series cross validation is where you fit lots of training sets and you're predicting one step ahead.

Leave-one-out cross-validation

We use all of the data, except for one on one points; we predict that one point.

Then, we use all of the data except for a different point to predict that point and so on.

So the test set is always one observation in the data, and we got t possible test sets, where t is the length of the data set.

What are the step of LOOCV?

1. Remove observation t from the data set, and fit the model using the remaining data. Then compute the error (e∗t=yt−^yt) for the omitted observation. (This is not the same as the residual because the tth observation was not used in estimating the value of ^yt.)
t
et∗=yt−y^t
t
y^t

2. Repeat step 1 for t=1,…,T.
t=1,…,T

3. Compute the MSE from e∗1,…,e∗T. We shall call this the CV.
e1∗,…,eT∗

Choosing regression variables (

👐🏼

Subset selection)

Backward selection (or backward elimination),

Start with a model containing all variables.

Try subtracting one variable at a time.

Keep the model if it has lower CV or AICc. Iterate until no further improvement.

 

In another words, Starts with all predictors in the model (full model), iteratively removes the least contributive predictors and stops when you have a model where all predictors are statistically significant.

improvement is determined by metrics like RSS,CV or adjusted R square; ETC3550 uses CV or AICc.

Notes:

1. Computational power is very similar to forwarding Selection.

2. Stepwise regression is not guaranteed to lead to the best possible model.

3. Inference on coefficients of final model will be wrong.

Forecasting with regression

Scenario based forecasting

• When we don’t know the information that is available in advance, we assumes possible scenarios for the predictor variables known in advance.
• For example, a US policy maker may be interested in comparing the predicted change in consumption when there is a constant growth of 1% and 0.5% respectively for income and savings with no change in the employment rate, versus a respective decline of 1% and 0.5%, for each of the four quarters following the end of the sample.

• Prediction intervals for scenario based forecasts do NOT include the uncertainty associated with the future values of the predictor variables.
• The resulting forecasts are calculated below and shown in Figure 7.18.

#1. make future_scenarios 
future_scenarios <- scenarios(
  Increase = new_data(us_change, 4) %>% ## we set up 4 peroids ahead
# assuming income increase 1%, Savings by .5%, Unemployment by 0%, Production by 0%)
    mutate(Income = 1, Savings = 0.5, Unemployment = 0, Production = 0), 
  Decrease = new_data(us_change, 4) %>%
    mutate(Income = -1, Savings = -0.5, Unemployment = 0, Production = 0),
  names_to = "Scenario"
)
#2. Make predictions
fc <- forecast(fit_consBest, new_data = future_scenarios)

Building a predictive regression model

Correlation, causation and forecasting

Correlation is not causation

• When x is useful for predicting y, it is not necessarily causing y. e.g., predict number of drownings y using number of ice-creams sold x.

• Correlations are useful for forecasting, even when there is no causality.

• Better models usually involve causal relationships (e.g., temperature x and people z to predict drownings y)

Multicollinearity

FAQ

Math