Solved – Once trained, is it normal that LSTM-based neural nets output different values even though the input is the same

lstmneural networkstensorflow

I'm trying my hand at recurrent neural nets having successfully implemented a few simple feed-forward networks. I'm using tflearn and the following example code that performs a simple sentiment analysis using IMDB film reviews (see https://github.com/tflearn/tflearn/blob/master/examples/nlp/lstm.py):

from __future__ import division, print_function, absolute_import

import tflearn
from tflearn.data_utils import to_categorical, pad_sequences
from tflearn.datasets import imdb

# IMDB Dataset loading
train, test, _ = imdb.load_data(path='imdb.pkl', n_words=10000,
                                valid_portion=0.1)
trainX, trainY = train
testX, testY = test

# Data preprocessing
# Sequence padding
trainX = pad_sequences(trainX, maxlen=100, value=0.)
testX = pad_sequences(testX, maxlen=100, value=0.)
# Converting labels to binary vectors
trainY = to_categorical(trainY, nb_classes=2)
testY = to_categorical(testY, nb_classes=2)

# Network building
net = tflearn.input_data([None, 100])
net = tflearn.embedding(net, input_dim=10000, output_dim=128)
net = tflearn.lstm(net, 128, dropout=0.8)
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=0.001,
                         loss='categorical_crossentropy')

# Training
model = tflearn.DNN(net, tensorboard_verbose=0)
model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,
          batch_size=32)

# Predict
model.predict(testX)

I noticed that if I call model.predict(testX) more than once, the output is not the same despite the input vectors being identical. Is this expected? If so, why?

Sorry in advance for the novice-level question

Best Answer

The reason is that currently there is a bug using dropout in RNNs, for some reason it still apply dropout at prediction time. We are checking that issue, it will hopefully be solved soon. So for now you can just remove dropout parameter from your lstm layer.

Related Question