MATLAB: Importing data files with the same name in different directories (seeds) , setting them equal to “loop able” variable

average data

Hello world!
I am a noob and wondering if anyone can help me with this problem.
I have multiple seeds in a simulation that output the same data filenames.
I would like to import all the data files from a structure like this:
path1/ seed_0001/Ca_cyt.dat
path1/ seed_0002/Ca_cyt.dat
path1/ seed_0003/Ca_cyt.dat
path1/ seed_000n/Ca_cyt.dat
I'm not sure if it is possible to loop over the entries, creating some variable name that corresponds seed numbers so I can individually use the data files if I need to (ex. seed 1 vs. seed 2)
The real issue is that the data overwrites itself when you import something with the same name.
I would like to set up a calculation to average the values of the data file over a time series (average of time-series of calcium from seeds 1,2,3…n)
Lastly, I would like to compare the averages of different situations.
Like:
path1/seed_0001/Ca_cyt.dat
path2/seed_0001/Ca_cyt.dat
path3/seed_0001/Ca_cyt.dat
My final issue: the length of some of the matrices are not the same. I've tried using something like (length(x)) to define the calculation and am having horrible luck getting it to plot them comparatively using the same X value of time.

Best Answer

"I will say that I've written a verbose form of this code that sets variables for every import. I have 25 simulations with 20 seeds each to compare and its very error prone."
That is not a surprise, because your approach is one way that beginners force themselves into writing slow, complex, buggy code. The approach of "..creating some variable name that corresponds seed numbers...": magically creating/accessing variable numbers is the problem. Read this to know why:
https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval
Forcing meta-data into variable names is bad way to write code. Meta-data, like the sequential folder names, is data, so it should be stored as data and not forced awkwardly into variable names.
You should simply store the data in one array (e.g. a structure, or cell array, or an ND array) which is then trivial to access using efficient indexing. Here is something to get you started:
D = 'path to where the PATH* directories are located';
P = 'path1';
S = dir(fullfile(D,P,'seed*));
for jj = 1:numel(S)
    F = fullfile(D,P,S(jj).name,'Ca_cyt.dat');
    S(jj).data = csvread(F); % use whatever function imports your data files
end
And that is all. The data for the files are all stored in the structure S, along with the folder names. So you can access them trivially using indexing, e.g. the second folder's name and data:
S(2).name
S(2).data
It also means that you can use a comma-separated list to perform actions on all of the imported data or filenames. For example, to put all of the filenames into one cell array:
{S.name}
or to vertically concatenate all of the imported data:
vertcat(S.data)
or you can access it in other ways, depending on the size and classes of your data.
Read this to know more about how comma separated lists work:
https://www.mathworks.com/matlabcentral/answers/320713-how-to-operate-on-comma-separated-lists
You might also want to look at the examples in the MATLAB documentation:
https://www.mathworks.com/help/matlab/import_export/process-a-sequence-of-files.html
Because you did not upload any sample files or explain anything about them I have no idea how your data files are formatted, so you will have to pick the file importing function yourself. I just used csvread as an example. If you upload a sample data file then I could help you with picking a suitable data importing function.
"Lastly, I would like to compare the averages of different situations.... path1 ... path2 ... path3 ..."
Then you will probably have to put the entirety of my code into another loop, basically like this:
D = 'path to where the PATH* directories are located';
N = 3;
C = cell(1,N);
for ii = 1:N
    P = sprintf('path%d',ii);
    C{ii} = dir(fullfile(D,P,'seed*));
    for jj = 1:numel(C{ii})
        F = fullfile(D,P,C{ii}(jj).name,'Ca_cyt.dat');
        C{ii}(jj).data = .... use whatever function imports your data files
    end
end
And then access the structures inside the cell array C. You could generate one structure array from that cell array, which might make accessing the data easier:
S = [C{:}]
Related Question