MATLAB: Problem having isoutlier detecting anything

isoutlierMATLAB

I have what looks like a very easy problem but I cannot seem to solve it. I have a dataset (attached) that has some obvious (to the human eye) outliers.

I cannot get isoutlier to detect it in any way. My attempt is essentially this:

idx = isoutlier(x(:,2),'movmedian',w);

I have put the code in a for loop, spanning all possible values of w, I get at most 3 outliers detected when the window size is 3 and those detected are not actually outliers.

Using movmean instead movmedian detects no outliers for any value of w. I have also played with the threshold factor, but without luck. This seemed to me like a straightforward application for the outlier detection. What am I missing?

Best Answer

One possible way to detect this type of outlier would be like this:

load('out1.mat');
% Assuming that data has 2nd order polynomial curve trend
p = polyfit(x(:,1),x(:,2),2);
y = polyval(p,x(:,1));
% Detect outlier in de-trend data
idx = isoutlier(x(:,2)-y);
% Show the result
plot(x(:,1),x(:,2))
hold on
plot(x(idx,1),x(idx,2),'ro')
legend({'Original data','Detected outlier'},'FontSize',14)

Related Solutions

MATLAB: How to make array from characters from variables

Create a cell array out of the character array:

c = cellstr(val);

Then, each cell element is a string that can be used:

c{1}

for example, is the first string. (Notice the use of curly brackets there.) You can read more about the use of cell array in the documentation.

MATLAB: Smoothing of noisy data

You forgot to attach your data. But you can try smoothdata(). It looks like it in turn calls movmean(), movmedian(), sgolayfilt(), or whatever smoothing method you want:

Smoothing method, specified as one of the following:

'movmean' — Moving average over each window of A. This method is useful for reducing periodic trends in data.
'movmedian' — Moving median over each window of A. This method is useful for reducing periodic trends in data when outliers are present.
'gaussian' — Gaussian-weighted moving average over each window of A.
'lowess' — Linear regression over each window of A. This method can be computationally expensive, but results in fewer discontinuities.
'loess' — Quadratic regression over each window of A. This method is slightly more computationally expensive than 'lowess'.
'rlowess' — Robust linear regression over each window of A. This method is a more computationally expensive version of the method 'lowess', but it is more robust to outliers.
'rloess' — Robust quadratic regression over each window of A. This method is a more computationally expensive version of the method 'loess', but it is more robust to outliers.
'sgolay' — Savitzky-Golay filter, which smooths according to a quadratic polynomial that is fitted over each window of A. This method can be more effective than other methods when the data varies rapidly.

Best Answer

Related Solutions

MATLAB: How to make array from characters from variables

MATLAB: Smoothing of noisy data

Related Question