MATLAB: How to check and remove outliers when it is Non-normal distribution

Question

I found that many people say z-score and mapstd standardization is good to detect outlier. But z-score is useful when only it is normal distribution. When I found my data doesn't follow normal distribution. What should I do? (1)Should i transform my data(boxcox,Johnson transformation) into normal distribution and use z-score to detect outlier? (2)After transformation and remove the outliers, should I use my transformed data or original data(outliers removed in both data) to be the input of neural network? I found that if I input my transformed data(Johnson transformation) into neural network, it works worse than the original data.How come is it?

Can anybody help.Thanks a lot.

Answer 1

The z-score is frequently used because according to the Central Limit Theorem, when the data are sufficiently numerous, the tend to be normally distributed regardless of the underlying distribution. (There is more to it that this simple statement, but that is the most basic explanation.)

If you know how your data are distributed, you can get the ‘critical values’ of the 0.025 and 0.975 probabilities for it and use them as your decision criteria to reject outliers. Again, outlier detection and rejection is another topic that goes beyond this simple explanation, and I encourage you to explore it on your own. How you decide to implement it with your data is something you will have to experiment with.