MATLAB: Accessing data within a structure

accessdataMATLABstructures

I have this structure:

veh(x).meas(y).data.fileName

So i have a certain number x of vehicles (veh) and a certain number y of measurements (meas).

how can i get the fileName for every measurement as a list? (efficient)

Would like ot avoid a construct of numerous "numel" and "for" commands.

thanks!

Best Answer

The chosen data structure demands for loops. There is no "efficient" solution, which extracts the fields in a vectorized way. You can at least avoid some pitfalls:

fileList = {};
nVeh     = numel(veh);
for k = 1:nVeh
    aVeh  = veh(k);
    nMeas = numel(aVeh.meas);
    for kk = 1:nMeas
       fileList{end+1} = aVeh.meas(kk).data.fileName;
    end    
end

Here the output grows iteratively. If you know a maximum number of elements, you can and should pre-allocate the output:

nMax     = 5000;
fileList = cell(1, nMax);
iList    = 0;
nVeh     = numel(veh);
for k = 1:nVeh
    aVeh  = veh(k);
    nMeas = numel(aVeh.meas);
    for kk = 1:nMeas
       iList           = iList + 1;
       fileList{iList} = aVeh.meas(kk).data.fileName;
    end    
end
fileList = fileList(1:iList);   % Crop ununsed elements

Choosing nMax too large is very cheap, while a too small value is extremely expensive. Even 1e6 does not require a remarkable amount of time.

numel is safer than length. The temporary variable aVeh safes the time for locating veh(k) in the inner loop repeatedly.

If you do not have any information about the maximum number of output elements, collect the data in pieces at first:

nVeh  = numel(veh);
List1 = cell(1, nMax);
for k = 1:nVeh
    aVeh  = veh(k);
    nMeas = numel(aVeh.meas);
    List2 = cell(1, nMeas);
    for kk = 1:nMeas
       List2{kk} = aVeh.meas(kk).data.fileName;
    end
    List1{k} = List2;
end
fileList = cat(2, List1{:});

Related Solutions

MATLAB: Make code faster for import and array creation

Hi Jenna, I bet this is a pre-allocation issue.

When you build a big matrix one column at a time (such as is being done for mydata_X and mydata_Y), MATLAB needs to do some juggling of memory to make the variable a little bit bigger on each iteration. If you know how big the end result will be before you start building your matrix, you can initialise the matrix to this size just once, and MATLAB no longer has the overhead of juggling memory space inside the loop.

That said, there might even be a sneakier way of avoiding the loops you're using. Let's say that after you read all the files:

%bring the data in from CSV format into a cell-array
csvFiles = dir('*.csv');         %works for the current directory only
numfiles = length(csvFiles);     %try to keep only the data you want in the dir
mydata = cell(1,numfiles);       %finds the total number of files in the dir
%save each XY data file into a cell of a cell-array called 'mydata'
for k = 1:numfiles
    mydata{k} = importdata(csvFiles(k).name); 
end

At this point, is the matrix of every cell in mydata the same size? It seems that it should be, based on the rest of your code... If it is, you can make a 3D matrix with one "sheet" for each cell in mydata like this:

myData3d = cat(3, mydata{:});

And then you can get mydata_X like this:

mydata_X = myData3d(:,1,:);

Now, mydata_X here will be an N-by-1-by-NFILES matrix, whereas I think from your code that your original mydata_X ended up as a N-by-NFILES matrix. If you want to get the same as your original, you can just use reshape or permute:

mydata_X = permute(myData3d(:,1,:), [1 3 2]);

The same can be done for mydata_Y.

Did this get you going? If it works, it will be much faster that the loop that builds matrices one column at a time.

And also, the last loop you have:

for i=1:numfiles
    arrayOfZ = 1:numfiles
    mydata_Z(i,:) = arrayOfZ(1,:) % here I assign 1 through 'numfiles' ...
end

Can be replaced by:

mydata_Z = repmat(1:numfiles, numfiles, 1);

Although I'm not quite sure what purpose this variable serves... it will be of a different size to mydata_X and mydata_Y ...

MATLAB: Importing data files with the same name in different directories (seeds) , setting them equal to “loop able” variable

"I will say that I've written a verbose form of this code that sets variables for every import. I have 25 simulations with 20 seeds each to compare and its very error prone."

That is not a surprise, because your approach is one way that beginners force themselves into writing slow, complex, buggy code. The approach of "..creating some variable name that corresponds seed numbers...": magically creating/accessing variable numbers is the problem. Read this to know why:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

Forcing meta-data into variable names is bad way to write code. Meta-data, like the sequential folder names, is data, so it should be stored as data and not forced awkwardly into variable names.

You should simply store the data in one array (e.g. a structure, or cell array, or an ND array) which is then trivial to access using efficient indexing. Here is something to get you started:

D = 'path to where the PATH* directories are located';
P = 'path1';
S = dir(fullfile(D,P,'seed*));
for jj = 1:numel(S)
    F = fullfile(D,P,S(jj).name,'Ca_cyt.dat');
    S(jj).data = csvread(F); % use whatever function imports your data files
end

And that is all. The data for the files are all stored in the structure S, along with the folder names. So you can access them trivially using indexing, e.g. the second folder's name and data:

S(2).name
S(2).data

It also means that you can use a comma-separated list to perform actions on all of the imported data or filenames. For example, to put all of the filenames into one cell array:

{S.name}

or to vertically concatenate all of the imported data:

vertcat(S.data)

or you can access it in other ways, depending on the size and classes of your data.

Read this to know more about how comma separated lists work:

https://www.mathworks.com/matlabcentral/answers/320713-how-to-operate-on-comma-separated-lists

You might also want to look at the examples in the MATLAB documentation:

https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html

Because you did not upload any sample files or explain anything about them I have no idea how your data files are formatted, so you will have to pick the file importing function yourself. I just used csvread as an example. If you upload a sample data file then I could help you with picking a suitable data importing function.

"Lastly, I would like to compare the averages of different situations.... path1 ... path2 ... path3 ..."

Then you will probably have to put the entirety of my code into another loop, basically like this:

D = 'path to where the PATH* directories are located';
N = 3;
C = cell(1,N);
for ii = 1:N
    P = sprintf('path%d',ii);
    C{ii} = dir(fullfile(D,P,'seed*));
    for jj = 1:numel(C{ii})
        F = fullfile(D,P,C{ii}(jj).name,'Ca_cyt.dat');
        C{ii}(jj).data = .... use whatever function imports your data files
    end
end

And then access the structures inside the cell array C. You could generate one structure array from that cell array, which might make accessing the data easier:

S = [C{:}]

Best Answer

Related Solutions

MATLAB: Make code faster for import and array creation

MATLAB: Importing data files with the same name in different directories (seeds) , setting them equal to “loop able” variable

Related Question