MATLAB: How to read text files form sub-sub folders

Database Toolboxmachine learningstatisticsStatistics and Machine Learning Toolbox

    Hi,
    I want to read text files from sub-sub folders:
    Architecture:
    Mainfolder
      Tool1
       sub-subFolder1
       sub-subFolder2
       .....
       .....
      Tool2
       sub-subFolder1
       sub-subFolder2
       .....
       .....
    ......
1. Read text files by each sub-folder(i.e, Tool1, Tool2, etc)
2. Output 
   Tool1.xlsx, Tool2.xlsx

I use the following code, but I can access sub-sub folders.

% - Define output header.
 header = {'RainFallID', 'IINT', 'Rain Result', 'Start Time', 'Param1.pipe', ...
    '10 Un Para2.pipe', 'Verti 2 mixing.dis', 'Rate.alarm times'} ;
Mainfolder='Mainfolder';
outLocatorFolder='OutputFolder';
 nHeaderCols = numel( header ) ;
 % - Build listing sub-folders of main folder.
%  D_main = dir( 'D:\Mekala_Backupdata\Matlab2010\Mainfolder' ) ;
 D_main = dir(Mainfolder ) ;
 D_main = D_main(3:end) ;             % Eliminate "." and ".."
 % - Iterate through sub-folders and process.
 for dId = 1 : numel( D_main )
    % - Build listing files of sub-folder.
    D_sub = dir( fullfile(Mainfolder, D_main(dId).name, '*.txt' )) ;
    nFiles = numel( D_sub ) ;
    keyboard
    % - Prealloc output cell array.
    data = cell( nFiles, nHeaderCols ) ;
    % - Iterate through files and process.
    for fId = 1 : nFiles
        % - Read input text file.
        inLocator = fullfile(Mainfolder, D_main(dId).name, D_sub(fId).name ) ;
        content = fileread( inLocator ) ;
        % - Extract relevant data.
        rainfallId = str2double( regexp( content, '(?<=RainFallID\s+:\s*)\d+', 'match', 'once' )) ;
        iint       = regexp( content, '(?<=IINT\s+:\s*)\S+', 'match', 'once' ) ;
        rainResult = regexp( content, '(?<=Rain Result\s+:\s*)\S+', 'match', 'once' ) ;
        startTime  = strtrim( regexp( content, '(?<=Start Time\s+:\s*).*?(?= -)', 'match', 'once' )) ;
        param1Pipe = str2double( regexp( content, '(?<=Param1.pipe\s+[\d\.]+\s+\w+\s+)[\d\.]+', 'match', 'once' )) ;
        tenUn      = str2double( regexp( content, '(?<=10 Un Para2.pipe\s+[\d\.]+\s+\w+\s+)[\d\.]+', 'match', 'once' )) ;
        verti2     = regexp( content, '(?<=Verti 2 mixing.dis\s+\S+\s%\s+)\S+', 'match', 'once' ) ;
        rateAlarm  = strtrim( regexp( content, '(?<=Rate.alarm times\s+\S+\s+)[^\r\n]+', 'match', 'once' )) ;
        % - Populate data cell array.
        data(fId,:) = {rainfallId, iint, rainResult, startTime, ...
            param1Pipe, tenUn, verti2, rateAlarm} ;
    end
    % - Output to XLSX.
%     outLocator = fullfile( 'D:\Mekala_Backupdata\Matlab2010\OutputFolder', sprintf( '%s.xlsx', D_main(dId).name )) ;
    outLocator = fullfile(outLocatorFolder, sprintf( '%s.xlsx', D_main(dId).name )) ;
    fprintf( 'Output XLSX: %s ..\n', outLocator ) ;
    xlswrite( outLocator, [header; data] ) ;
 end

many thanks in advance,

Best Answer

You need to use in dir() instead of *. See attached demo.

Related Solutions

MATLAB: How to read text files from each subfolder

Try something along this line:

 % - Define output header.
 header = {'RainFallID', 'IINT', 'Rain Result', 'Start Time', 'Param1.pipe', ...
    '10 Un Para2.pipe', 'Verti 2 mixing.dis', 'Rate.alarm times'} ;
 nHeaderCols = numel( header ) ;
 % - Build listing sub-folders of main folder.
 D_main = dir( 'Mainfolder' ) ;
 D_main = D_main(3:end) ;             % Eliminate "." and ".."
 % - Iterate through sub-folders and process.
 for dId = 1 : numel( D_main )
    % - Build listing files of sub-folder.
    D_sub = dir( fullfile( 'Mainfolder', D_main(dId).name, '*.txt' )) ;
    nFiles = numel( D_sub ) ;
    % - Prealloc output cell array.
    data = cell( nFiles, nHeaderCols ) ;
    % - Iterate through files and process.
    for fId = 1 : nFiles
        % - Read input text file.
        inLocator = fullfile( 'Mainfolder', D_main(dId).name, D_sub(fId).name ) ;
        content = fileread( inLocator ) ;
        % - Extract relevant data.
        rainfallId = str2double( regexp( content, '(?<=RainFallID\s+:\s*)\d+', 'match', 'once' )) ;
        iint       = regexp( content, '(?<=IINT\s+:\s*)\S+', 'match', 'once' ) ;
        rainResult = regexp( content, '(?<=Rain Result\s+:\s*)\S+', 'match', 'once' ) ;
        startTime  = strtrim( regexp( content, '(?<=Start Time\s+:\s*).*?(?= -)', 'match', 'once' )) ;
        param1Pipe = str2double( regexp( content, '(?<=Param1.pipe\s+[\d\.]+\s+\w+\s+)[\d\.]+', 'match', 'once' )) ;
        tenUn      = str2double( regexp( content, '(?<=10 Un Para2.pipe\s+[\d\.]+\s+\w+\s+)[\d\.]+', 'match', 'once' )) ;
        verti2     = regexp( content, '(?<=Verti 2 mixing.dis\s+\S+\s%\s+)\S+', 'match', 'once' ) ;
        rateAlarm  = strtrim( regexp( content, '(?<=Rate.alarm times\s+\S+\s+)[^\r\n]+', 'match', 'once' )) ;
        % - Populate data cell array.
        data(fId,:) = {rainfallId, iint, rainResult, startTime, ...
            param1Pipe, tenUn, verti2, rateAlarm} ;
    end
    % - Output to XLSX.
    outLocator = fullfile( 'OutputFolder', sprintf( '%s.xlsx', D_main(dId).name )) ;
    fprintf( 'Output XLSX: %s ..\n', outLocator ) ;
    xlswrite( outLocator, [header; data] ) ;
 end

Note that if you have a recent version of MATLAB, you can use the `folder` field of the struct outputed by DIR, and simplify most FULLFILE calls.

EDIT 4:09pm

Just a few extra comments. While it may look complicated, you should be fine with most of the code here. The general approach is

 Iterate through sub folders of 'Mainfolder'
      Iterate through files of sub folder
           Extract data from file and store in data array
      Export data array to relevant Excel file

The part that will likely be the most complex for you is the data extraction. One quick option for this is pattern matching using regular expressions. You can see a series of calls to REGEXP:

 .. = regexp( content, pattern, option1, option2, .. )

This extract from content a string that matches the pattern. When you need to export a number we convert it to double using STR2DOUBLE. When it may capture extra white spaces we trim it using STRTRIM.

Regular expressions are a big topic, so it is normal if you don't really understand the patterns. In short,

 aAb,123 etc : are literals; they are simply matched and they don't have
               any special meaning 
 \s, \S, \d  : match a single white-space, non white-space, numeric digit respectively
 *, +        : mean zeros or more, and one or more respectively times the pattern that precedes
               \d+ hence means one or more numeric digit
 [..], [^..] : defines a set of characters (-sets) to match or not to match respectively
               [\d\s]+ hence means one or more element of either \d or \s
 (?<=..)     : defines a look behind
               (?<=hello )world matches 'world' when it is preceded by 'hello '
 (?=..)      : defines a look forward
               hello(?= world) matches 'hello' when it is followed by ' world'
 .           : matches any character. To match a the character '.', it has to be escaped with \
               .[\d\.]+ matches any character followed by one or more characters that
               are either a numeric digit or a '.'

Given this information, you can understand the pattern for extracting e.g. the value of IINT:

 '(?<=IINT\s+:\s*)\S+'

which is, match

 (?<=..)\S+    : one or more non white-space preceded by something

and the something is

 IINT\s+:\s*   : the literal 'IINT' followed by one or more white-spaces,
                 followed by the literal ':', followed by zero or more white-spaces

Cheers,

Cedric

MATLAB: How to use regexp to split the filepath

Use the fileparts() function, not regexp():

[folder, baseFileName, extension] = fileparts(yourFilePath);

Best Answer

Related Solutions

MATLAB: How to read text files from each subfolder

MATLAB: How to use regexp to split the filepath

Related Question