I have a simple question. I am training a neural network feeding it with normalized data patterns using Gaussian normalization. My question arises when I see that some people use the mean and standard deviation of the training data to normalize the training data (which is logical) and the mean and standard deviation of the validation data to normalize the validation data (which is not so logical).
The validation data is supposed to be used to test the accuracy of the network in a real application, but in many applications you do not know beforehand what data you are going to predict/classify. For example: if you trained your NN to approximate a mathematical function "$f$" such that $y=f(x)$ and you want to know the output for a given value $x=i$, you only have that value and it cannot be normalized without other data.
So, I think that the $x=i$ value should be normalized using the mean and sd values of the training data, which makes more sense to me. If you were using [Max- min] standardization you would use the maximum and minimum values of your training data as well.
What do you think?
Best Answer
There's a situation that using the mean and std from the test data might make sense.
Say I want to predict whether a student gets a scholarship or not based on his/her test results.
If my training data are students from school A, my test data are students from school B, and A and B use two different grading systems.
Then using the mean and std of school B would be better.