MATLAB: Fopen fails to open file at certain number of iterations

Question

Hello all,

I have a GUIDE program that is using a loop to read data within .csv files that are contained within a folder. The file names are numbers i.e. 1.csv, 2.csv, 3.csv etc. Each file is about 5-6 MB and contains columns of numerical data.

At some point in the code there is an fopen to open and read in the relevant data from the files and this is assigned a file ID:

fID = fopen(strcat(fullfile( file location, file name),'.csv'),'r')

At a further point within the code, the file is closed and when the loop executes again the next file will be opened.

When i'm running the program on, for example, 100 files within the folder, whether they are 0.csv to 99.csv or 3400.csv to 3500.csv, the program executes fine with no issues, processes the files and outputs the desired information. However, when attempting to run the program with say 4000 files in the folder, the program halts but no error is shown in the command window.

Running

echo on all

allowed me to see that at some point in the code it flagged the error ' Error while evaluating uicontrol Callback'.

I was able to find that this error was occuring at the fopen line (as above).

The program was also stopping at the same file every time it was being run at 510.csv. To eliminate the cause of the issue being the source files, i deleted 50 files before 510.csv and ran it again. 510.csv processed fine but then it stopped at 560.csv, 50 files later.

I had a look at the fID values for the original run and it appears that the previous file 509.csv had a fID = 511 then for 510.csv fID = -1 (could not open file). Similarly when I deleted 50 files and it stopped at 560.csv the previous file 559.csv had fID = 511 and then for 560.csv fID = -1.

So what I gathered from of all of this was that the problem was not with any of the source files but that it was also stopping at the same location after fID = 511.

It's bizarre to me as the program works perfectly when I don't have as many files to process. It's also not a CPU/memory issue as I have been looking at the resource monitor when it halts and nothing is alarming.

Does anyone have any idea as to what is causing fopen to not find the next file when processing a large number of files?

Any help would be greatly appreciated.

Cheers.

Answer 1

If you were fclose()'ing the file properly, the file ID would not continuing increasing. So some code execution path is failing to fclose() the file after use.