Time-Series Prediction – ANN vs RNN: In-Depth Comparison and Analysis

forecastingneural networksrecurrent neural networktime series

The basic argument against ANN for time-series is that there is no conception of time for the inputs. However, it is obvious that one can feed multiple time-steps into an ANN by using various lag lengths where the features are just various lagged series. And "long term" memory one could include the feature of a moving average.

I understand RNN's have ability to work with variable length input. But given the case of an ANN with lookback lags and a fixed-window, what is the advantage of the RNN over the ANN?

Best Answer

The biggest practical difference from the prediction point of view is that ANN would use a fixed number of timepoints while RNN theoretically has no limit on how far it can look into history. Additionally, if you use ANN, the number of parameters would grow with the number of time-points you consider, while for RNN it would not. What follows, ANN would need more data, more computational resources, and would be more prone to overfit.

But if you look at the results of forecasting competitions you will notice that traditional or hybrid forecasting methods usually outperform neural networks, so in many cases, the answer would be that you should use neither.

Related Solutions

Solved – Truncated Back-propagation through time for RNNs

I had wondered the same thing, and it looks like this has been thoroughly researched in this nice blog post:

http://r2rt.com/styles-of-truncated-backpropagation.html (mirror)

The author found that the disjoint approach, which is what TensorFlow uses, works reasonably well. Using a sliding window every time step is difficult/expensive to calculate (especially with a framework like TensorFlow), and doesn't yield much benefit. The post cites a paper from the 1990s that found jumping forward h steps, then running BPTT back 2h steps was similar in outcome to just doing the disjoint approach of jumping h steps and running BPTT back also h steps.

If you're using LSTM's instead of simple RNN cells, then longer contexts do matter and it would probably be better not to truncate at all, or at least use a really high number of unroll steps. The author's results were for a simple RNN model.

Solved – Time steps in Keras LSTM

As described by Andrey Karpathy, the basic recurrent neural network cell is something like

$$ h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t) $$

so it takes previous hidden state $h_{t-1}$ and current input $x_t$, to produce hidden state $h_t$. Notice that $W_{hh}$ and $W_{xh}$ are not indexed by time $t$, we use the same weights for each timestep. In simplified python code, the forward pass is basically a for-loop:

for t in range(timesteps):
    h[t] = np.tanh(np.dot(Wxh, x[t]) + np.dot(Whh, h[t-1]))

So it doesn't matter how many timesteps there are, it is just a matter of how it is implemented. People often use fixed number of timesteps to simplify the code and work with simpler data structures.

In Keras, the RNN cells take as input tensors of shape (batch_size, timesteps, input_dim), but you can set them to None if you want to use varying sizes. For example, if you use (None, None, input_dim), then it will accept batches of any size and any number of timesteps, with input_dim number of features (this needs to be fixed). It is possible because this is a for-loop and we apply same function to every timestep. It would be more complicated in other cases, where varying sizes would need us to be using things like varying sizes for the vectors of parameters (say in densely-connected layer).

Best Answer

Related Solutions

Solved – Truncated Back-propagation through time for RNNs

Solved – Time steps in Keras LSTM

Related Question