MATLAB: How to replace outliers with 2 standard deviation from the mean

MATLABoutliers

How can I replace outliers with 2 standard deviation from the mean?

This script is to replace with mean, but not 2std from the mean, anyone could help to modify this?

A=filloutliers(A,'center','mean','ThresholdFactor', 2);% replace 2 std away from mean with mean

Best Answer

Can't do it with filloutliers alone; it for some reason doesn't have the facility to use a function handle as a fill option...

One way amongst many but one that keeps filloutliers in the code--

A=filloutliers(A,'center',nan,'ThresholdFactor', 2);    % step 1: replace 2 std away from mean with NaN
A(isnan(A))=2*std(A);                                   % step 2: replace NaN w/ 2*std()

Does seem like a reasonable enhancement request to be able to use a function handle in filloutliers

Alternatively,

mnA=mean(A); sdA=std(A);
Z=(A-mnA)/sdA;
isOut=(abs(Z)>=2);
A(isOut)=mnA+sign(Z(isOut))*2*sdA;

ADDENDUM:

To consolidate in one place, with the added caveat the data are 2D by column, the above must be extended as follows:

Since mnA and sdA are now row vectors of column statistics, Z needs the "dot" division operator to be element-wise:

Z=(A-mnA)./sdA;

and have to apply the outlier calculation by column as well. It's probably just as quick here to write the explicit loop as:

for i=1:size(A,2)
  A(isOut(:,i))= mnA(i)+sign(Z(isOut(:,i)))*2*sdA(i);
end

This way each column is a vector so the size of the logical elements selected will match and the mean and std dev are constants for the column instead of arrays.

Related Solutions

MATLAB: Repeat for-loop from the beginning after recalculating values

Directly addressing the Q?, here's a basic function to do the removal--

function x=thompsontau(x)
% Return array x w/ outliers replaced w/ NaN from input x
% Each column in x is assumed to be variable, rows are observations
%
% dpbozarth -- 5Sep2018
    [r,c]=size(x);            % row, col dimensions
    cols=1:c;                 % later lookup for columns if outliers exist
    mn=mean(x,'omitnan');     % mean by column
    sd=std(x,'omitnan');      % std deviation
    n=sum(isfinite(x));       % finite observations
    [mx,ix]=max(abs(x-mn));   % value, location (row) of max deviation
    isOut=mx>fntau(n).*sd;    % column locations are outliers if T
    if any(isOut)
      indxOut=sub2ind([r c],ix(isOut),cols(isOut));  % locations in x of outliers
      x(indxOut)=nan;         % mark as outlier
    end
  end
    function tau = fntau(n)
    % returns critical Thompson tau for n observations
      tcrit=tinv(1-0.05/2,n-2);    % critical t value
      tau=tcrit./sqrt(n)./sqrt(n-2+tcrit.*tcrit).*(n-1);
    end

This vectorizes your loop approach for the first case; your mission, should you choose to accept, is to now encapsulate this to do the subsequent tests until any(isOut) is false.

This could be done by calling the above in a loop or calling it recursively inside this function; will leave that as "exercise for the student" (which seems only appropriate for a t-test based algorithm :) )

NB: This operates on the full input array; if there's reason to not operate on some columns of an array, simply pass the subset. Or, as in this case, the first two columns will never have an outlier show up so it doesn't matter much, but for some datasets obviously that might not be so.

But, don't build such "magic constants" into functions that makes them unsuitable for other use as a general practice.

PS: HINT The recursive solution is by far the neater here and only takes inserting three additional code lines, all of which are consecutive, and two of which are Matlab branching keywords. :) (And, actually, the latter two aren't absolutely mandatory but I think it makes the code more legible to include them).

NB2: fntau uses tinv which requires Statistics TB. Lacking that, will have to revert to using a lookup table or finding another replacement for the t-value.

MATLAB: Axes component name as an argument to a function

Hi Jason,

You can pass the 'axes' object to a function and plot on that specific axes

function [high,low]=Autoscaleimage(ax,Image,n)
%ax is the handle to 'axes' object
    mn=mean(Image(:));
    sd=std2(Image(:));
    low=mn-sd;
    high=mn+n*sd;
    axes(ax);
    imshow(Image,[low,high]);
end

Regards,

Rohit

Best Answer

Related Solutions

MATLAB: Repeat for-loop from the beginning after recalculating values

MATLAB: Axes component name as an argument to a function

Related Question