MATLAB: How to load data from csv-files with “dot”-structured variable names and convert to struct/table

csvimportdataMATLABstruct

Importing data is not the biggest problem, however autogenterate variable names from the headers from the csv-file has been a nightmare.

For example, in a csv-file i can have the table like this(note i can't change the structure of the data sorce)

in.sensors.temp.x in.sensors.temp.y

2 3

1 4

Now what i want to do, is to load in the data and create a struct with the same structure as readings header and assign the data to it.

% what i have done so far:

I import the data by using the functionen "importdata". I have tried csvread, xlsread and readtable , but they have either not worked or created corupt data.

After that I get a struct containg data and datatext. The data text is type char characters.

How do I take create a struct by reading a cell containing charcaraters with dots?

I have tried something like this:

in = createStruct(var_name(5), data(:,5))
function temp = createStruct(fullString, data)
    fullString = cell2mat(fullString);
    dotFind = strfind(fullString,'.');
    temp = data;
    while ~isempty(dotFind)
      %temp = temp.(convertCharsToStrings(fullString(1:dotFind(1)-1)));
      struct(fullString(1:dotFind(1)-1),0);
      %fullString = fullString(dotFind(1)+1:end);
      %dotFind = strfind(fullString,'.');
    end
end

but did not perform very well

Best Answer

Better than evil EVAL is to use more robust SETFIELD and a comma-separated list:

str = 'in.sensors.temp.x';
val = pi;
tmp = regexp(str,'\.','split'); % or SPLIT
S = struct();
S = setfield(S,tmp{:},val)
S = struct with fields:
    in: [1×1 struct]

Checking:

S.in.sensors.temp.x
ans = 3.1416

Loop as required. Read more:

https://www.mathworks.com/help/matlab/matlab_prog/comma-separated-lists.html

https://www.mathworks.com/matlabcentral/answers/320713-how-to-operate-on-comma-separated-lists

Related Solutions

MATLAB: How to call data from files named with dynamic index

"select a great number of files for importing (best from a Windows Explorer "open a file" dialog)"

If you really have a "great number" of files your user is going to dislike this way of selecting them. Most likely simply selecting the folder and using dir with a filename filter would be simpler and less fiddly to use. But each to their own...

[C,P] = uigetfile(..,'MultiSelect','on');
N = numel(C);
D = cell(1,N);
for k = 1:N
    F = fullfile(P,C{k});
    M = .. whatever file importing function you want to use
    .. whatever processing you want to do
    D{k} = M;
end

You can access the filenames and filedata using basic indexing, e.g. for the third file:

C{3} % filename
D{3} % imported data

https://www.mathworks.com/help/matlab/matlab_prog/access-data-in-a-cell-array.html

A simple alternative is to import the data into the same non-scalar structure as dir returns, e.g.:

P = 'absolute/relative path to the folder where the files are saved';
S = dir(fullfile(P,'sample_*.txt'));
for k = 1:Nnumel(S)
    F = fullfile(P,S(k).name);
    M = .. whatever file importing function you want to use
    .. whatever processing you want to do
    S(k).data = M;
end

MATLAB: Alternatives for using EVAL to access data in multi-layered struct

One simple solution is to use getfield and setfield to access nested structures. Instead of returning the structure location as one character vector like this:

S = 'data.xlsx.file1.tab1'

you should return it in a cell array of char vectors, like this:

C = {'xlsx','file1','tab1'}

(this will require only a simple change to the recursive function). Then you can trivially access the data using getfield:

getfield(data,C{:})

and that is all! Here is a simple working demonstration:

>> data.xlsx.file1.tab = 1;
>> data.xlsx.file2.tab = 2;
>> data.csv.file2 = 3;
>> C = {'xlsx','file2','tab'};
>> getfield(data,C{:})
ans =  2

No ugly loops, no evil eval, no problems!

"But then again, maybe my whole approach for using such a multi-layered struct is already poor to begin with..."

Personally I am not a big fan of nested structures, and I notice that they tend to be overused by beginners wanting to reflect the minutae of how they see their data-organization. One of the main risks (which you are doing) is encoding meta-data like filenames and tab names into the code (as filednames). This is a bad way to write code: it make code complex and makes accessing that meta-data slow and buggy. Your approach is very fragile, e.g. because there are many filenames that are not valid fieldnames: what would your code do with the filename a-1.csv ? Or a.2.csv? The approach of mixing meta-data (like filenames and tab names) into data is simply flawed, and should be avoided. Meta-data is data, and it should be stored as data in it own right. Consider those example filenames: if we put them into a structure field named filename, then the code will never break depending on the name itself:

S.filename = 'a-1.2.3-4.csv'

You should consider that a table is a very powerful option and has many advantages for processing groups of data.

Personally I would probably use a single non-scalar structure, where the meta-data are simply encoded as data in fields:

data(1).type = 'xlsx'
data(1).name = 'file1'
data(1).tab  = 'tab1'
data(1).data = ...
data(2).type = 'csv'
data(2).name = 'file2'
data(2).tab  = [];
data(2).data = ...

This would make accessing and processing the data quite simple, and has some neat syntaxes that you will find very handy:

https://www.mathworks.com/help/matlab/matlab_prog/access-multiple-elements-of-a-nonscalar-struct-array.html

Best Answer

Related Solutions

MATLAB: How to call data from files named with dynamic index

MATLAB: Alternatives for using EVAL to access data in multi-layered struct

Related Question