MATLAB: How to isolate data from a large input file

isolatelarge data sets

I have a number of large data files with approximately 8,000,000 rows and 10 columns. The data is taken from a train and monitors various inputs over a number of days. The 10th column indicates direction of the train with 1 and -1 for differing direction and 0 for when the train is at a standstill.

Each time the train changes direction I would like to be able to create a new variable that stores all the following data until the next direction change.

I am able to do this manually, by examining the data and finding the index where a direction change is indicated, i.e. 1 becomes -1. I would like to make a process that could automate this.

Any help would be greatly appreciated.

Best Answer

I suggest not using a new variable but indexing into the one.

A very useful coding scheme easy to deal with.

To find the direction changes, use

ixdir=find(abs(diff(x(:,10))==2))+1;  % all the points of direction change

The first direction section is from 1:ixdir(1); second is then ixdir(1):ixdir(2), etc., ... Processing those in sequence is quite easy with the indices w/o different variables.

Related Solutions

MATLAB: Delete rows with bad data and surrounding rows

Here's another approach

% script to clean data
B = [0 1 0 0 1 0 1; 
    0 0 0 0 0 0 0;
    0 1 0 0 1 0 1; 
    0 1 0 0 0 1 0;
    0 0 0 0 0 0 0;
    0 1 0 1 1 0 1;
    0 0 0 0 0 0 0; 
    0 0 0 0 0 0 0;
    0 1 0 0 0 1 0; 
    0 1 0 0 0 1 0;
    0 0 0 0 0 0 0;
    0 0 0 0 0 0 0;
    0 0 0 0 0 0 0;  
    0 1 0 0 1 1 0;
    0 0 0 0 0 0 0; 
    0 0 0 0 0 0 0; 
    0 0 0 0 0 0 0;
    1 0 0 0 1 0 0; 
    1 0 1 1 1 0 1];
D = rand(size(B)); % data matrix to be cleaned
% assign parameters
minRun = 3; % minimum number of adjacent rows to be considered good data
% make vector with ones for the good rows (rows with only zeros)
iGood = ~any(B,2);
% now mark the locations where the beginning and end of each run of ones
% starts and ends
% use diff to create jumps at transitions, pad with 1's to ensure jump at start and end
isJump = [1; diff(iGood(:))~=0; 1]; 
% find location of jumps
jmpIdx = find(isJump);
% find run lengths of zeros, and ones
n = diff(jmpIdx);
% n has the lengths of runs of zeros, and runs of ones interleaved, but we
% need to find out whether it starts with the zeros, or starts with the
% ones
if iGood(1) == 1
    % starts with ones
    offset = 0;
    
else
    % starts with zeros
    offset = 1;
end
% in preparation for using repelem, build a vector with alternating 
% values of zero and run lengths
run = zeros(size(n)); % initalize and preallocate
iStart = 1 + offset; % element where first run of ones starts
run(iStart:2:end) = n(iStart:2:end);
% assign the run lengths corresponding to each row
runLength = repelem(run,n);
% only keep rows in B that are members of sufficiently wide (run length) peaks
idxClean = 1:size(B,1);
idxClean = idxClean(runLength >= minRun);
Bclean = B(idxClean,:);
% also probably want to clean some other matrix based upon status of B
Dclean = D(idxClean,:)

MATLAB: Vectorization instead of for loop

Preallocate your data array before the loop.

data = zeros(l,9);

Then populate it directly in the for-loop. The preallocation will be what speeds this up, not vectorization.

[data(indx,1),data(indx,2),data(indx,3),FIXME data(indx,etc)]=myGaussianFit(double(xdata),double(ydata), b0,c0);

Best Answer

Related Solutions

MATLAB: Delete rows with bad data and surrounding rows

MATLAB: Vectorization instead of for loop

Related Question