Forecasting – Flat Line Forecasting with ARIMA in SAS: How to Identify and Solve the Issue

arimadata visualizationforecastingsas

I'm trying to forecast a large dataset using ARIMA (The data does not have seasonality), I ended up getting an ARIMA(2,1,2) model where the log of volume was taken due to increase in variance over time. The model appears to fit well and the plot of residuals look good. However upon forecasting the model (for 5 years on a monthly basis) the forecast flattens out. I was wondering if this is to be expected given the large volume of data?

The last plot shows the forecasted volume with the confidence intervals plotted. I also would like to know if there is a way to make the last plot look nicer? I have been unable to get a plot of the forecast at the end of the time series data that looks nice.

Here is my code and the excel file:

* Open the file;
data intel_stock;
infile 'path' dlm=',' firstobs=2;   
input Date anydtdte10. Volume;
format Date date10.;
Timeref=_n_;
logvolume = log(Volume);
run;

* Plot the data ;
proc sgplot data=intel_stock;
series x=Timeref y=Volume/markers;  
xaxis values=(1 to 5000 by 1);  
run;
* Variation seems to increase greatly over time, hence we take the log of volume ;

proc sgplot data=intel_stock;
series x=Timeref y=logvolume/markers;   
xaxis values=(1 to 5000 by 1);  
run;
* Plot shows a good amount of variance removed ;

* selecting an ARIMA model ;
proc arima data=intel_stock;
identify var= logvolume(1);
estimate p = 2 q = 2 ;
forecast lead=60 interval=month id=Date out=forecast;
run;

* Merge forecast and volume into the same dataset and remove log from forecast ;
data intel_forecast;
update intel_stock forecast;
by Date;
if Date <= '19APR2021'D  then forecast = "NULL";
else forecast = exp(forecast);
if Date <= '19APR2021'D  then L95 = "NULL";
else L95 = exp(L95);
if Date <= '19APR2021'D  then U95 = "NULL";
else U95 = exp(U95);
run;

* Plot the forecast ;
proc sgplot data=intel_forecast;
series x=Date y=Volume/markers;
series x=Date y=forecast/markers;
series x=Date y=L95;
series x=Date y=U95;    
xaxis values=(1 to 1000000 by 1);   
run;

https://www.kaggle.com/tosinabase/intel-stock-prices-historical-data-intc

Best Answer

ARIMA models usually flatten out, especially if:

you have a nonseasonal model (as here)
you have an $\text{ARIMA}(p,1,q)$ model without an intercept where the $\text{ARMA}(p,q)$ model on the increments (i.e., on the differenced series, since you have $d=1$) converges to zero - again, this seems to be the case here.

This is very often the best possible expectation forecast.

We have a number of similar threads that may be helpful.

Related Solutions

Solved – How to get the true mean forecast using the Arima package with a Box-Cox transformation

The approach adopted by the forecast package is to fit a model to the transformed data and to backtransform both the point forecasts and prediction intervals. This is not "wrong". It is a legitimate choice. This preserves the coverage of the prediction intervals, and the back-transformed point forecast can be considered the median of the forecast densities (assuming the forecast densities on the transformed scale are symmetric). For many purposes, this is acceptable, but occasionally the mean forecast is required. For example, with hierarchical forecasting the forecasts need to be aggregated, and medians do not aggregate but means do.

It is easy enough to derive the mean forecast using a Taylor series expansion. Suppose $f(x)$ represents the back-transformation function, $\mu$ is the mean on the transformed scale and $\sigma^2$ is the variance on the transformed scale. Then using the first three terms of a Taylor expansion around $\mu$, the mean on the original scale is given by $$f(\mu) + \frac{1}{2}\sigma^2f''(\mu).$$

For a Box-Cox transformation, $$f(x) = \left\{\begin{array}{ll} (\lambda x+1)^{1/\lambda} & \text{if $\lambda\ne0$;}\\ e^x & \text{if $\lambda=0$.}\end{array}\right. $$ So $$f''(x) = \left\{\begin{array}{ll} (1-\lambda)(\lambda x+1)^{1/\lambda-2} & \text{if $\lambda\ne0$;}\\ e^x & \text{if $\lambda=0$.}\end{array}\right. $$ and the backtransformed mean is given by $$\left\{\begin{array}{ll} (\lambda \mu+1)^{1/\lambda}\left[1 + \frac{\sigma^2(1-\lambda)}{2(\lambda \mu+1)^{2}}\right] & \text{if $\lambda\ne0$;}\\ e^\mu\left[1 + \frac{\sigma^2}{2}\right] & \text{if $\lambda=0$.}\end{array}\right. $$ Therefore, to adjust the back-transformed mean obtained by R, the following code can be used.

library(forecast)

fit <- auto.arima(AirPassengers, lambda=0)
fc <- forecast(fit, h=50, level=95)
fvar <- ((BoxCox(fc$upper,fit$lambda)-BoxCox(fc$lower,fit$lambda))/qnorm(0.975)/2)^2
plot(fc)
fc$mean <- fc$mean * (1 + 0.5*fvar)
lines(fc$mean,col='red')

fit <- auto.arima(AirPassengers, lambda=0.2)
fc <- forecast(fit, h=50, level=95)
fvar <- ((BoxCox(fc$upper,fit$lambda)-BoxCox(fc$lower,fit$lambda))/qnorm(0.975)/2)^2
plot(fc)
fc$mean <- fc$mean * (1 + 0.5*fvar*(1-fit$lambda)/(fc$mean)^(2*fit$lambda))
lines(fc$mean,col='red')

An extended version of this answer is given on my blog.

Solved – ARIMA forecast has all values as zero

You might want to look at Transforming data as it is very relevant to your problem and also https://stats.stackexchange.com/questions/249005/what-are-the-assumptions-for-the-residuals-of-arima-model/249106#249106. The fundamental problem is your identified model is flawed for a number of possible reasons as it incorrectly converts statistical symptoms to incorrect (in this case) statistical cures. The data is nonstationary (that is a symptom) , the cause is a shift in the mean (correct cure) at period 21 which is visually obvious from here whereas your model decided to unfortunately apply regular differencing (wrong cure) . There is no need for logs or any other power transform when you adjust for the clear anomalies. Box-Cox transform determination easily misreads untreated positive outliers (high values) as causing high variance (symptom) whereas once they are adjusted (correct cure) no evidence is found suggesting the need for a power transform.

Following is the ACF of the original series showing no significant seasonal structure whereas your model had a seasonal difference. Intervention Detection procedures suggested a day 5 effect which might have been the reason for the unwarranted seasonal differencing in your model.

The plot of the Actual/Fit and Forecast from the model suggested by AUTOBOX (which I have helped to develop) visually tells an interesting story. . Since values can never go below zero for your data simply truncate the lower confidence interval estimate to 0.0.

The ACF of the original series is here and ACF of the model's residuals are here with residual plot here .

The equation is here notice that the level shift variable was not the dominant player as it was obfuscated by the anomalies (just as my eyes were) . The details of the model are here in 3 parts . . Finally we show the plot of the forecasts .

It is interesting to see the actual and the cleansed data together as it shows what might have been

Hope this helps you and others to better understand time series methodology and practice.