deep learningmachine learning
I feel confused about how to design a Multilayer Perceptron(MLP). I have read a number of papers about how to design a deep CNN structure, like VGG or ResNet. However, for a Multilayer perceptron, a more basic ANN structure, should I follow this trend to make the MLP deeper? Actually, I try some trials on python that stacking more layers in one network, but the result showed it make no sense.
Therefore, I just wanna ask the question above.
can anybody give me some suggestions?
The fundamental problem is that RBFs are a) too nonlinear, b) do not do dimension reduction.
because of a) RBFs were always trained by k-means rather than gradient descent.
I would claim that the main success in Deep NNs is conv nets, where one of the key parts is dimension reduction: although working with say 128x128x3=50,000 inputs, each neuron has a restricted receptive field, and there are much fewer neurons in each layer.In a given layer in an MLP- each neuron represents a feature/dimension) so you are constantly reducing dimensionality (in going from layer to layer).
Although one could make the RBF covariance matrix adaptive and so do dimension reduction, this makes it even harder to train.
Typically, people suggest that the deeper the network the better
At some point, adding more layers hurt performances.
Example 1: Larsson, Gustav, Michael Maire, and Gregory Shakhnarovich. "FractalNet: Ultra-Deep Neural Networks without Residuals." arXiv preprint arXiv:1605.07648 (2016). https://arxiv.org/abs/1605.07648
Example 2: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. "Deep Residual Learning for Image Recognition". arXiv:1512.03385. https://arxiv.org/abs/1512.03385
Practitioners simply try and see (and in between buy more GPUs).
(Similar question has been asked before 2 years ago, and I think it is a good time to bring it up again: Guideline to select the hyperparameters in Deep Learning)
As far as I know, not much progress has been done since then. There have been some new papers, but using known hyperparameter optimization techniques, e.g.:
There is one original work though that one of my lab mate made me aware of:
Miconi, Thomas. "Neural networks with differentiable structure.", arXiv (2016): the author introduce the concept of differentiable network structure to include it in the objective function.
Best Answer
The optimal neural network architecture depends on your data set.
FYI: