MATLAB: Textscan with very large .dat files, Matlab keeps crashing

crashloadreadtextscan

Hey everyone,
I am using R2013b to read in some very large files, 35 of them but I am running into memory problems and matlab usually crashes before loading the files in. I am using textscan but I was hoping someone could help me edit the code so that it will load in the data a block at a time or at least make it less memory intensive. I need all the years in one large cell array.
Any ideas?
Many thanks!
tic;
  HWFiles = {'midas_wind_197901-197912.txt','midas_wind_198001-198012.txt', ..........(up to 2013)};
  HWData = cell(1,numel(HWFiles));
    for i=1:numel(HWFiles);                                                                                                                             
        fid = fopen(HWFiles{i}, 'r');                                                                                                                   
        tmp = textscan(fid,'%s %*s %*f %*f %s %*f %f %*f %f %f %f %f %f %*f %*f %*f %*f %*f %*s %*f %*f %*s %*f %*f', 'Delimiter',',');                
        HWData{i} = tmp ;                                                                                                                               
        fclose(fid);
    end
    toc;

Best Answer

Run through the files, reading them with textscan(), but instead of storing the results all in memory, use the matFile class to append the new data to the end of a variable in a .mat file.
Once that is done, you can start a new MATLAB session and load() the .mat file to get the combined cell array.
Related Question