When people use autoencoders, they usually normalize the data such that the values are normalized to the range [0,1]. Why is that? Why not use zero-mean unit variance normalization for example? I read on a Quora answer that this range gives you more choice of loss functions, but I don't really understand why. Any ideas?
Solved – Why normalize data to the range [0,1] in autoencoders
autoencodersnormalization
Related Solutions
I hope to replace this with a full answer once we have sorted out what's going on.
In trying to show you what's going on with the dependence in large vs small values, I see another problem:
The y-values have been monotonically transformed (log(y-2.e6)) for clarity.
The green box shows how the large values always occur together.
But they're also astonishingly regular. It looks like the large ones are every 20th value.
The red ovals show another problem. Notice patches of almost pure-black followed by almost pure-blue in the red ovals? There's something weird going on. Why would the middling-size values alternate, with a patch from column 1 then a patch from column 2?
You have neither independence within columns nor within pairs, but I am not sure what is causing that particular alternation in the red ovals.
The part about normalizing across rows pops out at me. It's usual to normalize a feature (column) so that, having done this for each feature, the features will be on more comparable scales. Normalizing across rows probably won't make any physical sense, and I'm not sure I can see any situation where it would be justified. (Imagine mashing a person's height, weight, and blood pressure together.)
Even if you normalize only the columns, note: if you normalize all of your data and then split it into train/test, you will get unrealistically better test results than you should. Your training data represents the data you have before you deploy your model and your test data represents the data that comes in after deployment. By normalizing across this boundary, you are allowing data from the future (test set) to leak into the present (training set). This can't and won't happen in the real world.
Best Answer
In general, the exactly normalization of data isn't super important in neural networks as long as the inputs are at some reasonable scale. As Alex mentioned, with images, normalization to 0 and 1 happens to be very convenient.
The fact that normalization doesn't matter much is only made stronger by use of batch-normalization, which is a function/layer frequently used in neural networks which renormalizes the activations halfway through the network to zero mean and unit variance. And the authors of the paper you linked did use batch normalization, which means however the data was normalized before, it was renormalized a bunch of times inside the network anyway.
Furthermore, reading their code, which is on github, they actually did preprocess the data two ways -- with zero mean unit variance normalization, and also with min 0 max 1 normalization. They didn't explain why they chose one preprocessed dataset over the other, but I suspect they either just arbitrarily to use min 0 max 1 normalization, or some preliminary hyperparameter searches showed that one worked better than the other for whatever reason.