MATLAB: How to free up memory for writing a large number (almost a million) of large (15 KB) text files

memory

% Get a cube of horizontal winds centred on an observing station
% for a range of altitudes and a range of latitudes and longitudes
%--------------------------------------------------------------
%Initialize variables
% z direction
max_alt=100000; %in metres

vertical_res=100; %in metres
num_vertical_pts=max_alt/vertical_res+1; %+1 for surface
alt=-vertical_res; %initial altitude

% x-y direction
horizontal_res=.1; %in degrees
horizontal_extent=5; %extent in lat and long directions, in degrees
num_horizontal_pts=(horizontal_extent/horizontal_res)^2;
%Get all latitudes and longitudes in grid
central_lat=14.5; 
central_long=79.2;
start_lat=central_lat-floor(horizontal_extent/2);
start_long=central_long-floor(horizontal_extent/2);
lat=[];
long=[];
for i=1:(horizontal_extent/horizontal_res)
        lat=[lat start_lat]; %set of latitudes
        long=[long start_long]; %set of longitudes
        start_lat=start_lat+horizontal_res;
        start_long=start_long+horizontal_res;
end
% time direction
daysofyear=[1:1:366];
% output
wind_all=cell(num_vertical_pts,1);
% merid_all=cell(num_vertical_pts,1);
% zonal_all=cell(num_vertical_pts,1);
altitude_all=cell(num_vertical_pts,1);
%------------------------------------------------------------------------
%Get data
for i=1:(horizontal_extent/horizontal_res) %lat
    for j=1:(horizontal_extent/horizontal_res) %long        
        day=0;
          for k=1:366
              day=day+1;
              alt=-vertical_res; %initial altitude
              wind=[];
              for l=1:num_vertical_pts
                  alt=alt+vertical_res;                                
                  %Get merid and zonal wind each altitude
                  wind= [wind atmoshwm07(lat(i),long(j),alt,'day',day,'seconds',0)];
              end
              %Arrange data so that col 1 = merid and  col 2 = zonal
              wind=reshape(wind,2,num_vertical_pts);
              wind=transpose(wind);
              %merid_all{i,j,k}=wind(:,1); %meridional component of wind in m/s
              %zonal_all{i,j,k}=wind(:,2); %zonal component of wind in m/s 
              wind_all{i,j,k}=wind;
              altitude_all{k}=alt;
              filename = sprintf('Lat=%.1f_Long=%.1f_Day=%.0f.txt',lat(i),long(j),day);
              dlmwrite(filename,wind_all{i,j,k},'\t');
           end
      end
  end
I'm looking for some advice on freeing up some memory to run the code shown above. At the moment, I am getting an "out of memory" error at the exact time that the first for loop (with int i) in the "get data" section ends. In other words, it writes (5/0.1)x(366)=18300 files before giving up. I want it to write 50 times more files than that!
Some background on the code – I'm calculating north-south and east-west winds for a range of altitudes (0-100 km with a resolution of 100 m) every day of a theoretical leap year for a range of different latitudes and longitudes (+/- 5 degrees centered on a particular observing station with a resolution of 0.1 degrees)
I'm just looking for any ideas that might make this work (other than manually changing i and running it for each latitude – i.e. 50 times!)
To run this code, you need the Aerospace Toolbox – it contains the hwm07 function, which is the function that calculates the wind. Its output is just 2 numbers – the north-south wind and the east-west wind.
Many thanks in advance, Sharon

Best Answer

I would recommend replacing the dlmwrite with your own code to fopen() the file, fprintf() the data, and fclose() it; that would protect against any memory leaks. It looks to me as if an appropriate format would be
fmt = [repmat('%g\t', 1, num_vertical_points-1), '%g\n');
fid = fopen(filename, 'w');
fprintf(fid, fmt, wind.');   %transpose is important here
fclose(fid);
I note, however, that you are using
wind_all{i,j,k}=wind;
which is accumulating all of the data in memory. Running out of memory is not surprising. If you do not need to store all of the information in memory, then do not do it. If you do need to store all of the information in memory, you should have a look at map reduce
Related Question