Solved – How to make LSTM predict multiple time steps ahead

lstmrecurrent neural networktime series

I am trying to use a LSTM for time series prediction. The data streams in once per minute, but I would like to predict an hour ahead. There are two ways I can think of for going about this:

Squash the data into hourly data instead, taking the average over each 60 minute time period as one data point.
For each (X, y) training data pair, let X be the time series from t - 120 to t - 60, and let y be the time series from t - 60 to t. Force the LSTM to predict 60 timesteps ahead, and take y[-1] as the prediction.

Are there any best practices for going about this?

Best Answer

There are different approaches

Recursive strategy

one many-to-one model

prediction(t+1) = model(obs(t-1), obs(t-2), ..., obs(t-n))
prediction(t+2) = model(prediction(t+1), obs(t-1), ..., obs(t-n))

Direct strategy

multiple many-to-one models

prediction(t+1) = model1(obs(t-1), obs(t-2), ..., obs(t-n))
prediction(t+2) = model2(obs(t-2), obs(t-3), ..., obs(t-n))`

Multiple output strategy

one many-to-many model

prediction(t+1), prediction(t+2) = model(obs(t-1), obs(t-2), ..., obs(t-n))`

Hybrid Strategies
- combine two or more above strategies

Reference : Multi-Step Time Series Forecasting

Related Solutions

Solved – the purpose of unrolling an LSTM into multiple time steps if you can just use a stateful LSTM of 1 time step

An RNN and any of its more sophisticated versions has a hidden state. The hidden state at time $t$ is a function of the hidden state at time $t-1$ and the input at time $t$. This hidden state at time $0$ is typically initialized to $0$. The fundamental reason why RNNs are "unrolled" is because all previous inputs and hidden states are used in order to compute the gradients wrt the final output of the RNN.

One issue with this is that the memory required to hold all these activations and gradients is linear in the length of the sequence. Therefore, very long sequences require prohibitive amounts of memory. In order to get around this issue, long sequences are typically truncated. For example, if you have 99 words in a phrase and want to predict the 100th, you may only use the last 9.

The idea of stateful RNNs is that you can do better than simply initializing the hidden state at time $0$ to $0$. Indeed, if we are feeding in the 90th word at time $0$, then we should initialize the hidden state as computed by feeding in words $0$ through $89$ in an rolled fashion instead. So we're saying that we're only going to train with shorter sequences, but initialize the hidden state to the same value it would've been computed if we trained with longer sequences.

Obviously, we can't backpropagate through the initial value of the hidden state, since deliberately truncating the backpropagation was the whole point. So your stateful RNN with length 1 would not properly learn relationships across times.

Solved – How to design a many-to-many LSTM RNN in Keras

If you're still interested in this, I would check out this thread: https://github.com/keras-team/keras/issues/6063. A sample model might look like this:

model = Sequential()  
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))  
model.add(RepeatVector(X)) #X = number of time steps you want to forecast
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))  
model.add(TimeDistributed(Dense(1)))
model.add(Activation('linear'))   
model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])

I am also pretty interested in doing something similar and might end up coming back to it because it looks promising. I hope this helps!

Best Answer

Related Solutions

Solved – the purpose of unrolling an LSTM into multiple time steps if you can just use a stateful LSTM of 1 time step

Solved – How to design a many-to-many LSTM RNN in Keras

Related Question