Solved – Implementing Neural Network for time series

neural networksstationaritytime series

I am currently working on neural networks for time series forecasting. My doubt is: do we need to take into account issues like trend, non-stationarity and seasonality while using neural networks instead of the box jenkins methodology?

If yes, how do we do that? For instance, do we need to change the cost function or other parameters? I am currently using nnet and neuralnet packages in R but I do not find anything about these issues in the related documentation.

If not, then does it imply that the model learns the seasonality from data itself? For instance, suppose there is a time series with an upward trend, along with some variations. Could I assume that if we fit a neural network on such series then it might learn just an interpolation?

Best Answer

Some Googling for specifically neural networks and seasonality leads to this paper, Neural network forecasting for seasonal and trend time series, Zhang and Qi, European Journal of Operational Research, V.160, 2, 16 January 2005, 501–514. In this paper the authors sought to compare the Box-Jenkins approach with a neural network approach. From the abstract:

We find that neural networks are not able to capture seasonal or trend variations effectively with the unpreprocessed raw data and either detrending or deseasonalization can dramatically reduce forecasting errors. Moreover, a combined detrending and deseasonalization is found to be the most effective data preprocessing approach.

They conclude that accounting for trend and seasonality in your preprocessing steps is a good idea.

Related Solutions

Solved – Plotting predicted values in ARIMA time series in R

That is why you shouldn't do ARIMA or anything on non stationary data.

Answer to a question why ARIMA forecast is getting flat is pretty obvious after looking at ARIMA equation and one of assumptions. This is simplified explanation, do not treat it as a math proof.

Let's consider AR(1) model, but that is true for any ARIMA(p,d,q).
Equation of AR(1) is:
$$ y_t = \beta y_{t-1} + \alpha + \epsilon$$ and assumption about $ \beta $ is that $|\beta| \le 1$. With such a β every next point is closer to 0 than the previous until $ \beta y_{t-1} =0 $, and $y_t = const = \alpha $.

In that case, how to deal with such a data? You have to make it stationary by differentiation ($new.data=y_t-y_{t-1}$) or calculating % change ($new.data=y_t/y_{t-1} -1$). You are modeling differences, not a data itself. Differences are getting constant with time, that is your trend.

 require(tseries)
 require(forecast)
 require(astsa)
 dif<-diff(gtemp)
 fit = auto.arima(dif)
 pred = predict(fit, n.ahead = 50)
 ts.plot(dif, pred$pred, lty = c(1,3), col=c(5,2))
 gtemp_pred<-gtemp[length(gtemp)]
 for(i in 1:length(pred$pred)){
   gtemp_pred[i+1]<-gtemp_pred[i]+pred$pred[i]
 }
 plot(c(gtemp,gtemp_pred),type="l")

Solved – If an auto-regressive time series model is non-linear, does it still require stationarity

If the purpose of your model is prediction and forecasting, then the short answer is YES, but the stationarity doesn't need to be on levels.

I'll explain. If you boil down forecasting to its most basic form, it's going to be extraction of the invariant. Consider this: you cannot forecast what's changing. If I tell you tomorrow is going to be different than today in every imaginable aspect, you will not be able to produce any kind of forecast.

It's only when you're able to extend something from today to tomorrow, you can produce any kind of a prediction. I'll give you a few examples.

You know that the distribution of the tomorrow's average temperature is going to be about the same as today. In this case, you can take today's temperature as your prediction for tomorrow, the naive forecast $\hat x_{t+1}= x_t$
You observe a car at mile 10 on a road driving at the rate of speed $v=60 $ mph. In a minute it's probably going to be around mile 11 or 9. If you know that it's driving toward mile 11, then it's going to be around mile 11. Given that its speed and direction are constant. Note, that the location is not stationary here, only the rate of speed is. In this regard it's analogous to a difference model like ARIMA(p,1,q) or a constant trend model like $x_t\sim v t$
Your neighbor is drunk every Friday. Is he going to be drunk next Friday? Yes, as long as he doesn't change his behavior
and so on

In every case of a reasonable forecast, we first extract something that is constant from the process, and extend it to future. Hence, my answer: yes, the time series need to be stationary if variance and mean are the invariants that you are going to extend into the future from history. Moreover, you want the relationships to regressors to be stable too.

Simply identify what is an invariant in your model, whether it's a mean level, a rate of change or something else. These things need to stay the same in future if you want your model to have any forecasting power.

Holt Winters Example

Holt Winters filter was mentioned in the comments. It's a popular choice for smoothing and forecasting certain kinds of seasonal series, and it can deal with nonstationary series. Particularly, it can handle series where the mean level grows linearly with time. In other words where the slope is stable. In my terminology the slope is one of the invariants that this approach extracts from the series. Let's see how it fails when the slope is unstable.

In this plot I'm showing the deterministic series with exponential growth and additive seasonality. In other words, the slope keeps getting steeper with time:

You can see how filter seems to fit the data very well. The fitted line is red. However, if you attempt to predict with this filter, it fails miserably. The true line is black, and the red if fitted with blue confidence bounds on the next plot:

The reason why it fails is easy to see by examining Holt Winters model equations. It extracts the slope from past, and extends to future. This works very well when the slope is stable, but when it is consistently growing the filter can't keep up, it's one step behind and the effect accumulates into an increasing forecast error.

R code:

t=1:150
a = 0.04
x=ts(exp(a*t)+sin(t/5)*sin(t/2),deltat = 1/12,start=0)

xt = window(x,0,99/12)
plot(xt)
(m <- HoltWinters(xt))
plot(m)
plot(fitted(m))

xp = window(x,8.33)
p <- predict(m, 50, prediction.interval = TRUE)
plot(m, p)
lines(xp,col="black")

In this example you may be able to improve filter performance by simply taking a log of series. When you take a logarithm of exponentially growing series, you make its slope stable again, and give this filter a chance. Here's example: