MATLAB: Store all iteration loop outputs in a file

loopmatrixsave

Hi, I am writing a program with a loop where the output from every iteration is a matrix with the same size (the size of the matrix is very large). However, I want to save the output from each iteration in a file (and not to create another variable (matrix or a vector)), as the number of iterations is more than 100 000 (needed for Gibbs sampling). I need to save the output from each iteration in a file (ex. .mat or txt) as storing it in a new variable might consume more RAM than my computer has available. I need to read the results from the file and do further calculation. I was recommended to use 'save' but I don't know how to save the output from all iterations as the name of the matrix (b) is the same and it is overwritten after each iteration, if I use the '-append'.
Simple example of the program is:
a=[1 1 1; 1 1 1; 1 1 1];
b=[];
for i=1:6
    b=a.*i
    save('test.mat','b', '-append')
end
The output should be:
[1 1 1
1 1 1
1 1 1 
2 2 2
2 2 2
2 2 2
3 3 3
3 3 3
3 3 3
4 4 4
4 4 4
4 4 4
5 5 5
5 5 5
5 5 5
6 6 6
6 6 6
6 6 6]

Best Answer

If the file is very large then you could use memmapfile:
https://www.mathworks.com/help/matlab/import_export/overview-of-memory-mapping.html
Or one of the other large file accessing tools:
https://www.mathworks.com/help/matlab/large-files-and-big-data.html
Or you can simply create a new file on each iteration:
 save(sprintf('name%04d.mat',k),'X')
Note: the -append option should really be named the -replace option, as the save documentation makes clear: " save(filename,variables,'-append') adds new variables to an existing file. If a variable already exists in a MAT-file, then save overwrites it. The variables argument is optional."
Related Question