MATLAB: How to import big data CSV files

csvdata importtextscan

Hi,
Similar questions have been already asked but I wanted to know if there is an alternative to importing. I have a csv file with 3.5 million rows and 56 columns. At present while importing, I have to select the range I need. For instance I am selecting only 36 columns and barring few rows almost all. I tried textscan() but unable to achieve what I want which is without importing my code should select those rows and columns needed and also my main aim is to save time of importing and since textscan() is pretty fast I was using that. My column values are numerics.
Please let me know if I need to update with something here. I have a 8GB RAM so I think it should be easily feasible.
Thanks in advance!text

Best Answer

These days, often the most convenient way is to use detectImportOptions, and set the SelectedVariableNames property of that to choose particular columns, and the readtable() -- or as of R2019a, readmatrix() if you are pure numeric.
textscan() is also a possibility. The easiest way to use that might be something like,
numcol = 56;
wanted_cols = [5 17:33 44:47];   %adjust to suit
fmt_cell = repmat('%*s', 1, numcol);
fmt_cell(wanted_cols) = {'%f'};
fmt = [fmt_cell{:}];
data = cell2mat( textscan(fid, fmt, 'Delimiter', ',', 'CollectOutput', 1) );
Related Question