Solved – Deep Belief Network (number of layers)

autoencodersdeep learningdeep-belief-networksrestricted-boltzmann-machine

So we have "several RBMs"

Deep Belief Network

A deep belief network is obtained by stacking several RBMs on top of each
other. The hidden layer of the RBM at layer `i` becomes the input of the
RBM at layer `i+1`. The first layer RBM gets as input the input of the
network, and the hidden layer of the last RBM represents the output. When
used for classification, the DBN is treated as a MLP, by adding a logistic
regression layer on top.

but why more RBM's don't means better? How to determine best number of layers?

Seems related:
Deep belief network performs worse than a simple MLP

Best Answer

I haven't worked the proof myself, but according to Hinton et al (2006) more RBMs should in fact always be better* as another RBM always increases the lower bound on the probability that the training data can be reproduced.

*The exception to this is when an RBM has as many or more visible units as the RBM before it has hidden layers, as then the chances of simply learning the identify function go up. For example, if you have an input of dimension 1000, then you should always get an increase in predictive performance if your next layers are 700, 500, 200, etc. up to the point at which the output of the last RBM matches the "true" dimensionality of variation in the data, which is a question of experimentation or expertise.

Related Question