I was reading about when to use standardization vs normalization and what I could understand was that standardization should be used when the model in use makes some assumptions about the data (I don't know why this assumption). Also I have read that standardization retains the original distribution of data only if it was Gaussian.
Many people on the internet seem to believe that neural nets don't make any assumptions.
Now, I do not see how neural networks don't make any assumptions because-
Few months back I had gone through the derivation of mean squared error function and how for normally distributed error term, MSE is the maximum likelihood estimator. This assumption for error term is stated as assumption for linear regression. So, the assumptions depend on the choice of our cost function, I believe? Wouldn't that mean neural network also make assumptions based on choice of cost function, otherwise the estimates wouldn't be that good?
Also, does it mean that I should standardize when data is normally distributed or distribution doesn't matter?
Best Answer
They don’t. Moreover, normality is not among core assumptions of linear regression either. It is true that minimizing squared error is equivalent to maximizing Gaussian likelihood, but this doesn’t mean that you need to make such assumption when minimizing squared errors. You can use linear regression when the assumption is broken. For linear regression we need the assumption to hold mostly for hypothesis testing and confidence intervals, both are not used, and would be hard to do, in case of neutral networks.