Solved – Neural Networks, what is the best way to deal with sequence when order does not matter

classificationneural networksrecurrent neural network

Right now I am working on a problem where I have sets of 5 profile pictures of people. I am trying to classify those pictures using a CNN which feeds into an RNN. As a whole the set of 5 images should fall into one class although there may be some outliers within the 5 (like someone who posts a picture of their dog rather than themselves.) Ideally the network should be able to learn to ignore these. Right now this CNN->RNN approach works decently well. However, in an RNN the order in which the images are presented seems to matter. If I shuffle the profile images and feed them to the network the prediction probability tends to vary by about 5%. I was wondering if there is an architecture other than RNNs where the whole sequence can be considered yet where the ordering of the sequence does not matter?

Best Answer

In case of RNN order matters. hence your architecture will not work well when you shuffle the images and then tries to predict a sample. For this problem, RNN is not required at all and they are not even a better choice for such kind of problems. The best architecture for your problem is the use of CNNs only. It will give you an exceptional accuracy if you implement it correctly. In order to get an overview how it works, you can take a look at the example of CIFAR10 small images classification: Convolutional Neural Network (CNN) with realtime data augmentation given here

Related Solutions

Solved – Structure of Recurrent Neural Network (LSTM, GRU)

A is, in fact, a full layer. The output of the layer is $h_t$, is in fact the neuron output, that can be plugged into a softmax layer (if you want a classification for the time step $t$, for instance) or anything else such as another LSTM layer if you want to go deeper. The input of this layer is what sets it apart from the regular feedforward network: it takes both the input $x_t$ and the full state of the network in the previous time step (both $h_{t-1}$ and the other variables from the LSTM cell).

Note that $h_t$ is a vector. So, if you want to make an analogy with a regular feedforward network with 1 hidden layer, then A could be thought as taking the place of all of these neurons in the hidden layer (plus the extra complexity of the recurring part).

Solved – Feed Forward Neural Network – How to Visualize the Weight Matrix

You're right, and there should be 50 images. You could easily verfiy this by:

[coef.shape for coef in mlp.coefs_[0]]

where mlp is the trained MLP classifer in the example.

So here are the two things caused confusion:

Clearly, the author of the example did not mention anything about why only 16 images
In Python when you zip two things in the for loop that don't have equal length (number of items) in this example zip(mlp.coefs_[0].T, axes.ravel()), Python will automatically ignore the extra items in the bigger lists (arrays, etc.). Here axes.ravel() has only 16 items, therefore, the loop iterates over first 16 vectors in mlp.coefs_[0]

Best Answer

Related Solutions

Solved – Structure of Recurrent Neural Network (LSTM, GRU)

Solved – Feed Forward Neural Network – How to Visualize the Weight Matrix

Related Question