Solved – How to design a many-to-many LSTM RNN in Keras

kerasmachine learningrecurrent neural networktime series

I have timeseries data with 1 minute cadence with 4 features, and I want to try to predict the time-evolution of 2 of these features using a RNN using LSTMs in Keras. My aim is to predict the e.g. next 10 minutes of data for these two features, based on the last hour of data of the 4 features.

I have previously constructed a RNN which can predict a single timestep for each 60 minute data input, using keras (many-to-one). Since I only predict a single timestep (minute), I need to predict all 4 features, to be able to use this model to predict up to 10 timesteps (minutes) iteratively. The model looks like this:

from keras.models import Sequential
from keras.layers import Dense, LSTM, Dropout, Activation, BatchNormalization

model = Sequential()
model.add(LSTM(32, return_sequences=True, input_shape=(shape,4)))
# ... more hidden LSTM layers
model.add(LSTM(32))
model.add(Dense(4))

My input data are timeseries of the features, data. I use MinMaxScaler in scikit-learn to normalize the features to (0,1). I then use TimeseriesGenerator from keras to generate the training data. I use a length of 60 to provide the RNN with 60 timesteps of data in the input.

from keras.preprocessing.sequence import TimeseriesGenerator

# data.shape is (n,4), n timesteps
tsgen = TimeseriesGenerator(data, data, length=60, batch_size=240)

I then fit the model, with checkpointing:

mcp = ModelCheckpoint("rnn_{epoch:03d}_{loss:.6f}.h5")
model.fit_generator(tsgen, epochs=30, callbacks=[mcp])

I would now like to a prediction "many-to-many", so that I do not need to predict all 4 features, but rather predict the next 10 timesteps of the 2 sought-after features.

I have looked at the TimeDistributed layer for keras, but according to the documentation that will only allow me to predict 60 values (i.e. the length of my input timestep shape). How do I use that to predict only the next 10 minutes of data, based on the last 60 minutes? How would I write that in keras? Do I need to use my own TimeseriesGenerator to account for the target vector? For example, is it sufficient to fill the last 50 minutes of the target vector with zeros, and just use the TimeDistributed layer?

Best Answer

If you're still interested in this, I would check out this thread: https://github.com/keras-team/keras/issues/6063. A sample model might look like this: 

model = Sequential()  
model.add(LSTM(input_dim=1, output_dim=hidden_neurons, return_sequences=False))  
model.add(RepeatVector(X)) #X = number of time steps you want to forecast
model.add(LSTM(output_dim=hidden_neurons, return_sequences=True))  
model.add(TimeDistributed(Dense(1)))
model.add(Activation('linear'))   
model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics=['accuracy'])

I am also pretty interested in doing something similar and might end up coming back to it because it looks promising. I hope this helps!

Related Question