MATLAB: Reading data from a specific CSV file

marker trajectoriesread csvregexptextscan

Hi all, I need to write a program for reading csv files, produced while recording trajectories of reflective markers (points) with a 3D camera. The first 44 lines of the csv file do not contain important data of the tracked markers. All the next lines contain comma separated data taken with a frequency of 120 Hz. So the 45th line is taken at time 0 and so on. These lines include x,y and z coordinates of all the tracked markers at this time. It is not necessary that the same markers were tracked in all the lines, therefore markers with new id numbers appear (Marker-idNumber).
I would like to store the x,y,z data for each marker in a cell array with marker names, which contain the marker's coordinates through time. For beginning we could assume that same markers appear in each line!- I guess this is a much easier task and I have almost done it (look ahead).
I found an existing program, which performs a similar job. I adapted the program for my needs, but due to my lack of programming knowledge I failed to succeed. If anyone knows what I am doing wrong I would appreciate your help.
Here is my code up to now (example data file attached):
clear all
openedFile = fopen('ReducedData.csv','r');
currentLineNo = 0;
%we go through the first lines which dont contain data of interest
for lineIndex = 1:43
    lineContent = fgetl(openedFile);
    currentLineNo = currentLineNo +1;
end
commaSeperatedValues = regexp(lineContent, ',', 'split');
numberOfFrames = str2num(commaSeperatedValues{3});
%We go to frame 0
for lineIndex = 44 : 45
    lineContent = fgetl(openedFile);
    currentLineNo = currentLineNo + 1;
end
frameIndex = 0;
for lineIndex = 1:numberOfFrames
      currentLineNo = currentLineNo + 1;
      frameIndex = frameIndex+1;
      frameLine = fgetl(openedFile);
      commaSeperatedValues = regexp(frameLine, ',', 'split');
      frameIndexFromFile = str2num(commaSeperatedValues{2});
      timestamp = str2num(commaSeperatedValues{3});
      numberOFTrackedMarkersInThisFrame = str2num(commaSeperatedValues{5});
       % Get marker coordinates
       idx = find(~cellfun(@isempty, strfind(commaSeperatedValues,'Marker-'))); %searches for locations of word 'Marker'
       for c2 = 1 : length(idx)
       coor = cellfun(@str2double, commaSeperatedValues(idx(c2)-4:idx(c2)-2)); %reads the coordinates of each Marker
       nums = regexp(commaSeperatedValues{idx(c2)},'\d*','match'); %reads the identification numbers of markers
       MarkerCoordinates(str2double(nums{c2})).(sprintf('Marker%s',nums{c2}))(frameIndex,:) = coor; %stores data into a CellArray??
       end
  end
The exemplar data contains rows of different markers. But we can check if the code works for lines with same markers in all lines by changing the perturbation field of the second last for loop ( "for lineIndex = 1:numberOfFrames" – instead of numberOfFrames = 1). When I do this I get error: "Index exceeds matrix dimensions." So how can I make the program work for lines with same markers in each line, and how could I upgrade the program to work for various markers in lines? Thank you.
Regards,
Jurij Hladnik

Best Answer

This code is based on your description of the problem, rather than your code. I choose to store the data in a structure, rather than a in cell array. I'm not sure, I exactly understand the format. Why the integer in the position before Marker in ...,15238,Marker-15238,...? I have skipped it. And why do the rows end with a marker-Id, e.g. 15251,Marker-15251. I skipped that too. However, try
>> M = OptiTrack( 'ReducedData.csv' );
>> M
M = 
    RowHead: [506x7 double]
     m01003: [506x4 double]
     m01004: [506x4 double]
     m01005: [506x4 double]
     m01006: [506x4 double]
     m15232: [3x4 double]
     m15229: [3x4 double]
     ....
>> plot( M.m15339(:,1), M.m15339(:,2:4) )
>> plot( M.m15330(:,1), M.m15330(:,2:4) )
(I failed to include an image. It displays in the Preview, but disappears when I Submit.)
where
function M = OptiTrack( filespec )
    fid = fopen( filespec );
    cac = textscan( fid, '%s', 'Headerlines',44, 'Delimiter','\n' );
    fclose( fid );
    cac = cac{1};
    M   = struct( 'RowHead', zeros(0,7) );
    len = length( cac );
    for jj = 1 : len  % loop over all rows
        row = regexp( cac{jj}, 'Marker-', 'split' );
        num = cell2mat( textscan( row{1}, '%*s%f%f%f%f%f%f%f%*f', 'Delimiter',',' ) );
        M.RowHead = cat( 1, M.RowHead, num ); 
        sec = num(2);
        for kk = 2 : length(row) - 1 % loop over all markers of one row
            mmm = textscan( row{kk}, '%d%f%f%f%*f', 'Delimiter',',', 'CollectOutput',true );
            str = sprintf( 'm%05d', mmm{1} );
            if ismember( str, fields(M) )
                M.(str) = cat( 1, M.(str), [ sec, mmm{2} ] );
            else
                M.(str) = [ sec, mmm{2} ];
            end
        end
    end
end
Yes, the function isn't fast, but I think it does the job.
Replacing
if ismember( str, fields(M) )
by
if any( strcmp( str, fields(M) ) )
improves the speed of the function by 25% (on R2013b and with the csv-file already in the cache).
&nbsp
OptiTrack version 2.0
Assumptions
  • all data rows are trackable rows containing marker data (the value of the first column is 'frame')
  • the number embedded in the marker-name is equal to the marker-id number
function M = OptiTrack( filespec )
    fid = fopen( filespec );
    cac = textscan( fid, '%s', 'Headerlines',44, 'Delimiter','\n' );
    fclose( fid );
    cac = cac{1};
    M   = struct( 'RowHead', zeros(0,4) );
    len = length( cac );
    for jj = 1 : len
        row = regexp( cac{jj}, ',Marker-\d+,', 'split' );
        buf = textscan( row{1},'%*s%f%f%f%f%[^\n]','Delimiter',','  ...
                    ,   'CollectOutput',true                        );
        %        
        M.RowHead = cat( 1, M.RowHead, buf{1} ); 
        sec = buf{1}(2);
        %


        row(1) = buf{2};
        %                
        for kk = 1 : length(row)
            mmm = textscan( row{kk},'%f%f%f%d','Delimiter',',' ...
                        ,   'CollectOutput',true                  );
            str = sprintf( 'm%05d', mmm{2} );
            if any( strcmp( str, fields(M) ) )
                M.(str) = cat( 1, M.(str), [ sec, mmm{1} ] );
            else
                M.(str) = [ sec, mmm{1} ];
            end
        end
    end
end
Here is a new version to test. The format of the data row is a bit tricky, since there is no explicit delimiter between the "row header" and the first marker data. It was easier to make and test the required changes than to explain :).
&nbsp
OptiTrack version 3.0
function M = OptiTrack( filespec )      %
    fid = fopen( filespec );
    cac = textscan( fid, '%s', 'Headerlines',44, 'Delimiter','\n' );
    fclose( fid );
    rows = cac{1};
    M = struct( 'RowHead', zeros(0,4) );
    for jj = 1 : length( rows )    % loop over all data rows
        %
        %   Strip off the row header
        cac = textscan( rows{jj}, '%s%f%f%f%f%[^\n]'    ...
                ,   'Delimiter',',', 'CollectOutput',true );
          M.RowHead = cat( 1, M.RowHead, cac{2} ); 
          sec = cac{2}(2);
          %   split the rest of the row into cells of single marker 
          marker_data = regexp( char( cac{3} )                ...     
                  ,   '([\d\.\-]+,){4}Marker\-\d+,?', 'match' );
          for kk = 1 : length( marker_data )  % loop over all markers 
              % split into numerical data and marker name
              marker  = textscan( marker_data{kk} ,'%f%f%f%f%s' ...
                              ,   'Delimiter'     , ','         ...
                              ,   'CollectOutput' , true        );
              % make a short marker name, which is legal as a Matlab name 
              mmm = textscan( char(marker{2}), '%*s%d', 'Delimiter','-' );
              str = sprintf( 'm%05d', mmm{1} );
              if any( strcmp( str, fields(M) ) )
                  M.(str) = cat( 1, M.(str), [ sec, marker{1}(1:3) ] );
              else
                  M.(str) = [ sec, marker{1}(1:3) ];
              end
          end
      end
  end
Related Question