autoencodersdeep learningdropoutkeras
I'm using Keras to implement a stacked autoencoder, and I think it may be overfitting. I wanted to include dropout, and keep reading about the use of dropout in autoencoders, but I cannot find any examples of dropout being practically implemented into a stacked autoencoder.
Where would the dropout layer(s) go, between every layer, only after the input layer, is anyone able to let me know/provide some resource implementing this?
The most obvious issue that I see is the size of your hidden layers. They're generally trained on a GPU and the training can take hours, days, and sometimes weeks.
The "capacity"of your network is not large enough to fully represent the underlying variability in the data (MNIST). Try increasing the size of your hidden layers.
I guess it's the problem with the data: the background is the same, and the ball is small.
As shown in the blog you referenced, one application of autoencoders is image denoising.
When you use the denoising autoencoder you actually add noise to the input images on purpose, so from your results it seems that the autoencoder only learns the background and the ball is treated as noise.
Maybe we could try the followings to see if we can get better.
1) I don't know if you've subtracted the mean (background) from the image already, if not we can try that so that the data will be almost zero everywhere except the ball.
2) Instead of using fully-connected autoencoders we can try the convolutional autoencoders, which works better with image data and does not really rely on the denoising part. There's example code in that blog for this as well.
Best Answer
We have tried adding it in few different ways:
However, we could not eliminate the overfitting completely. We have also tried adding noise to input data (Denoising AE), adding regularization (Sparse AE) to encoding layers. But still our hand made features performed better than AE created features.