MATLAB: Does FOPEN fail when trying to open very large files from MATLAB

bigenormousfilefopenhugeintlargelongMATLABpointer

Why does FOPEN fail when trying to open very large files from MATLAB?

How can I read/write a file whose size is with greater than 2,147,483,647 bytes (=2^31-1) in MATLAB using FOPEN and FREAD/FWRITE? FOPEN always returns -1, which indicates an invalid file id (FID).

Best Answer

This is a limitation in the FOPEN routines in MATLAB, due to the underlying Standard C Library implementation of FOPEN. This implementation only guarantees the ability to address memory in files of less than 2^31 - 1 bytes. The reason behind this limitation is that a 32-bit integer is used to represent the position in the file. For more details, see page 230 of "The Standard C Library" by P.J. Plauger (<http://www.amazon.com/exec/obidos/tg/detail/-/0131315099/103-3235037-9951021?v=glance>).

You can workaround this C limitation by breaking up your file into smaller sections, preferably on boundaries which are logical divisions of your data. It may be also be possible to use non-Standard library functions to address larger files. You would need to create a C MEX-file to use this second option.

Related Solutions

MATLAB: Do I get an OutOfMemory error when using FREAD and FPRINTF with large amounts of text (ASCII) data

The data type "char" is represented internally in MATLAB as Unicode, which means that 2 Byte of memory are needed per character in order to represent the data. To work around this issue, you will either need to have a largest contiguous free block that is at least twice the size of the data that you are loading, or you could read the data as an unsigned char, as follows:

fId = fopen('largefile.txt', 'rt');
y = fread(fId, 'uchar=>uchar')';
fclose(fId);

The function FPRINTF in MATLAB is vectorized, unlike the FPRINTF function in the C programming language. Because of this, there is some extra overhead in parsing the MATLAB data and preparing it before it can be passed to the low-level C-FPRINTF function. This overhead may or may not be reflected in the output of FEATURE MEMSTATS. So in the end, more memory is needed to write a file using FPRINTF than the amount of memory that is used by the data passed to it.

MATLAB is a higher programming language, because of this; it needs to convert the data to a low-level format first, before passing it to a low-level function.

MATLAB: Is MATLAB unable to read an ASCII file containing a large amount of data in a single line using the CSVREAD function

This bug has been fixed in Release 14 Service Pack 2 (R14SP2). For previous releases, please read below for any possible workarounds:

This has been verified as a bug in MATLAB in the way that the CSVREAD function handles large data files.

Currently, as a workaround, try the following code:

fid = fopen('filename'); 
a = textscan(fid,'%n','delimiter',',');

This reads the file rather quickly and creates an array of doubles. The difference is that when CVSREAD reads the same file, it tries to have a format string with thousands of fields, one for each line of data.

Best Answer

Related Solutions

MATLAB: Do I get an OutOfMemory error when using FREAD and FPRINTF with large amounts of text (ASCII) data

MATLAB: Is MATLAB unable to read an ASCII file containing a large amount of data in a single line using the CSVREAD function

Related Question