MATLAB: Large data file with mixed character strings and numerical formats

large dataMATLAB

I have a large data text file with 2,000,000 rows and 100 columns. Some columns have numerical values and some are character strings with variable length. I don't need all the data at once but I need to be able to import different columns (character columns and numerical columns) for my analysis. How should I do that?
The issue is with the size of the file rather than the mixed formats. MATLAB datastore function only reads 20,000 rows at a time and I don't know if converting the data into a SQL datatable would help.
Thanks in advance

Best Answer

Do not use fgetl or fgets: on such a large file as this would be very slow. Use textscan, exactly as the MATLAB documentation recommends:
http://www.mathworks.com/help/matlab/import_export/import-large-text-files.html
The third textscan input lets you specify a block size, which sets a limit to how many lines to read. So to read your required data into MATLAB without reading all of the data at once, you need to do the following in a loop (pseudocode, see link above):
out = {};
k = 0 ;
while ~feof(fileID)
  k = k+1;
  C = textscan(fileID,formatSpec,N);
  out{end+1} = the columns you need
end
This reads each block, you extract and store the columns that you need, and then the rest of the data is discarded. In this way, all of the data is read into MATLAB, just not simultaneously! This is really quite fast :)
Related Question