i know the difference between noise and outlier but i want to know the relationships of them and the effects of noises … are they cause outliers? and if you know the link of data-sets with outliers please give me the links. thank you
MATLAB: What is the relationship between noise and outliers ?
anomaly detectionnoiseoutlier
Related Solutions
Well no and yes. Maybe not theoretically the same, however you know, or you should know, that Poisson noise for anything over an expected value of about 8-10 looks virtually identical to Gaussian noise. Just prove it to yourself by doing a Poisson curve for, say 100 and then check the differences between that and a Gaussian fitted to it. The differences will be very small.
And you pretty much never have just 10 photons per pixel unless you are doing very low exposure photon noise limited experiments like in astronomy or radiography. So for most intents and purposes, Poisson noise manifests itself as Gaussian noise, and so the Gaussian noise you have can be considered Poisson noise. The bottom line is how the image looks and if it improves it enough for you to get the required measurements out of it. And it sounds like it's working for you, so go with it.
As dpb said, it is impossible to know if some arbitrary value for RMSE is good or bad. Only you know if it is good, because only you can know how much noise you would expect in the data.
The point is, when you use a model on some data that generates an RMSE, there are TWO components to the error, noise and lack of fit.
Any model will not be able to predict random noise in the data, so the predictive capability of the model can be no better than that noise. If it is better, then you are overfitting the noise. This is a bad thing to do.
At the same time, a model is just a model. It is an approximation to the real thing, with some aspects of reality that are left out. In some cases, a model is just a metaphor for the real system under study, so your model arises from some completely different system, but one that you hope behaves similarly to the real world case under study. In either case, a model will typically have some degree of lack of fit, since it is never a perfect model of the process. There is always something you have not included.
So we think of the residuals for any model to a physical system as something composed of both noise and lack of fit. These two components mix together. If the model is a good one, and the lack of fit is very small, then all we will see after the model is removed is noise. So our hope is the RMSE will be roughly the same as the underlying noise standard deviation, something that you SHOULD have some feel for if you are modeling any process.
So if the RMSE is significantly very small, more so than we would expect, then you are overfitting the problem. That is bad, since your model is now compensating for (chasing after) random noise.
If the RMSE is significantly larger than you would have expected, then this is again a problem, since it indicates there is a potential problem in the model. The model is inadequate to represent the system under study, or you did a poor job of estimating anything that must be estimated in the model.
But the point is, RMSE means NOTHING unless you have some a-priori knowledge about the process. You must have some expectation as to what is acceptable for the problem you are studying. IF you have no clue, it is impossible for someone else to know, since did not develop the model, we did not generate the data under study.
Best Answer