Solved – ny method for choosing the number of layers and neurons

Question

I'm learning autoencoders applied to image classification. However, I'm in the beginning stage (training the autoencoder for feature extraction).

I was testing different topologies by changing the neurons number in each layer and adding more layers. But I don't have a reliable method to do that. So, I was changing the topology by adding neurons in each layer (always using sigmoidal function) and I have sometimes better results and in others worse. I want to know, how can I make this decision? For example, with 1000 neurons works fine and 1005 works fine as well, the same thing with 1010 and 1015.

Is there any methodology for changing the topology following any rule, concept, idea or technique? I used k-fold in neural networks to do that in simple MLPs. Nevertheless, with autoencoders I will need a lot of processing time to apply that in all architectures, when trying different topologies.

Answer 1

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches.

---

There exist more advanced techniques such as

1) Gaussian processes. Example:

• Franck Dernoncourt, Ji Young Lee Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.

2) Neuro-evolution. Examples:

• Zaremba, Wojciech. Ilya Sutskever. Rafal Jozefowicz "An empirical exploration of recurrent network architectures." (2015): used evolutionary computation to find optimal RNN structures.
• Franck Dernoncourt. "The medial Reticular Formation: a neural substrate for action selection? An evaluation via evolutionary computation.". Master's Thesis. École Normale Supérieure Ulm. 2011. Used evolutionary computation to find connections in the ANN.
• Bayer, Justin, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. "Evolving memory cell structures for sequence learning." In International Conference on Artificial Neural Networks, pp. 755-764. Springer Berlin Heidelberg, 2009.: used evolutionary computation to find optimal RNN structures.