MATLAB: Data is not saving to the workspace

formattingif loopindexingMATLABwhile loop

I have a large text file composed of a single row of 52480000 numbers separated by semicolons. I'm attempting to organize the data into 51250 rows of 1024 numbers and then separate this into distinct blocks of 1025 x 1024. The numbers need to stay in the same order they were in in the original file (with every 1025th number being the start of a new row) I have tried using a while and if loop.

R = 51250;
C = 1024;
fid = fopen( 'TEST_A.asc');
k = 0;
while ~feof(fid)
z = textscan( fid, '%d', R*C, 'EndOfLine', ';');
if ~isempty(z{1})
k = k + 1;
s = fprintf( 'TEST_A.asc', ';');
dlmwrite( s, reshape( z{1}, 1025, []), ';')
end
end
fclose(fid);

This code does not create an initial cell of 52480000 numbers, which means that none of the subsequent data sets (s & z) are created in the workspace. The problem is that if I textscan the data into Matlab before formatting it, the file creates a memory error. Does anyone notice anything that I don't about this code or have any pointers?

Best Answer

Thank you for the file. What did I learn from the actual data file: that it is not "composed of a single row", but in fact there are 51200 rows in the file that I received.

Why is this important? Because computers are stupid, and they do exactly what they are told to do. Knowing how to read a file correctly requires knowing what format the file has. In this case it is also quite handy for us, because it is trivial to read and write lines without much processing.

The code below worked correctly for me, reading the 200 MB file, and creating 50 smaller files with the rows following the same order as the original file.

sbd = 'temp';
f2d = fopen(fullfile(sbd,'temp_01.asc'),'wt');
f1d = fopen(fullfile(sbd,'TEST_A.asc'),'rt');
k = 0;
while ~feof(f1d)
    str = fgetl(f1d);
    if sscanf(str,'%d')==1
        k = k+1;
        fclose(f2d);
        fnm = fullfile(sbd,sprintf('temp_%02d.asc',k));
        f2d = fopen(fnm,'wt');
    end
    fprintf(f2d,'%s\n',str);
end
fclose(f1d);
fclose(f2d);

Note that:

the size of the output matrices is 1024x1025 (because there are 1025 numbers per line). This is correct because the first number of each line is simply a line count (check the files and you will see).
the lines are exactly the same as the original file.
MATLAB hold one line at a time: the lines are simply read from the large file and written directly to a new file.
as a result: no matrix, no converting from string to numeric and back to string.
it is slow because the file is large... reading and writing 51200 lines of 1025 numbers each will take some time.

Related Solutions

MATLAB: Making movie from images

Ali - how do you know that your images are in the correct order? For example, what does

 Files = dir('*.jpg');

return? I'm going to guess that your file names have a number in them, but you may not be getting the order that you expect. For example, the following loop creates a 100 (empty) text files

 for k=1:100
      fid = fopen(sprintf('test%d.txt',k),'wt');
      fclose(fid);
 end

Now when I do

 dir('test*.txt')

I observe that

               test18.txt   test28.txt   test38.txt   test48.txt   test58.txt   test68.txt   test78.txt   test88.txt   test98.txt   
  test1.txt    test19.txt   test29.txt   test39.txt   test49.txt   test59.txt   test69.txt   test79.txt   test89.txt   test99.txt   
  test10.txt   test2.txt    test3.txt    test4.txt    test5.txt    test6.txt    test7.txt    test8.txt    test9.txt    
  test100.txt  test20.txt   test30.txt   test40.txt   test50.txt   test60.txt   test70.txt   test80.txt   test90.txt   
  test11.txt   test21.txt   test31.txt   test41.txt   test51.txt   test61.txt   test71.txt   test81.txt   test91.txt   
  test12.txt   test22.txt   test32.txt   test42.txt   test52.txt   test62.txt   test72.txt   test82.txt   test92.txt   
  test13.txt   test23.txt   test33.txt   test43.txt   test53.txt   test63.txt   test73.txt   test83.txt   test93.txt   
  test14.txt   test24.txt   test34.txt   test44.txt   test54.txt   test64.txt   test74.txt   test84.txt   test94.txt   
  test15.txt   test25.txt   test35.txt   test45.txt   test55.txt   test65.txt   test75.txt   test85.txt   test95.txt   
  test16.txt   test26.txt   test36.txt   test46.txt   test56.txt   test66.txt   test76.txt   test86.txt   test96.txt   
  test17.txt   test27.txt   test37.txt   test47.txt   test57.txt   test67.txt   test77.txt   test87.txt   test97.txt

Note the ordering of the files: 1,10,100,11-19,2,20-29,3,30-39,etc. with early images being inserted every ten images. If your files are named in a similar fashion then you will need to force the correct order or rename he files. If, in the code above, I had named the files as

 for k=1:100
      fid = fopen(sprintf('test%03d.txt',k),'wt');
      fclose(fid);
 end

where each integer is padded with up to three zeros, then I get the correct ordering of the files as

dir('test*.txt')
test001.txt  test012.txt  test023.txt  test034.txt  test045.txt  test056.txt  test067.txt  test078.txt  test089.txt  test100.txt  
test002.txt  test013.txt  test024.txt  test035.txt  test046.txt  test057.txt  test068.txt  test079.txt  test090.txt  
test003.txt  test014.txt  test025.txt  test036.txt  test047.txt  test058.txt  test069.txt  test080.txt  test091.txt  
test004.txt  test015.txt  test026.txt  test037.txt  test048.txt  test059.txt  test070.txt  test081.txt  test092.txt  
test005.txt  test016.txt  test027.txt  test038.txt  test049.txt  test060.txt  test071.txt  test082.txt  test093.txt  
test006.txt  test017.txt  test028.txt  test039.txt  test050.txt  test061.txt  test072.txt  test083.txt  test094.txt  
test007.txt  test018.txt  test029.txt  test040.txt  test051.txt  test062.txt  test073.txt  test084.txt  test095.txt  
test008.txt  test019.txt  test030.txt  test041.txt  test052.txt  test063.txt  test074.txt  test085.txt  test096.txt  
test009.txt  test020.txt  test031.txt  test042.txt  test053.txt  test064.txt  test075.txt  test086.txt  test097.txt  
test010.txt  test021.txt  test032.txt  test043.txt  test054.txt  test065.txt  test076.txt  test087.txt  test098.txt  
test011.txt  test022.txt  test033.txt  test044.txt  test055.txt  test066.txt  test077.txt  test088.txt  test099.txt

MATLAB: Creating multiple equally sized matrices from a single numerical cell

Firstly, the idea of generating lots of variables is popular with beginners, but really should be avoided:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

Also note that the MATLAB documentation is really good. It is readable, and has articles on lots of topics. Such as this one, which gives a good, robust method for reading a large file into MATLAB:

https://www.mathworks.com/help/matlab/import_export/import-large-text-files.html

The core idea of that code is to call textscan in a loop, use textscan's N option to specify how much data to read, and save the data into a cell array. The N option simply defines how many times the format is applied when reading the file.

You should be able to work it out from the examples in the documentation.

As an alternative you might like to read about Tall Arrays, which are a special kind of data type especially for working with very large data files that cannot be read into memory:

https://www.mathworks.com/help/matlab/tall-arrays.html

EDIT 2017-02-10: add code from comment:

%%Create Fake Datafile %%
% fid = fopen('temp2.txt','wt');
% for k = 1:50,
%     fprintf(fid,'%d;',randi([0,255],1,1025*1024));
% end
% fclose(fid);
%%Read DataFile %%
R = 1025;
C = 1024;
opt = {'EndOfLine',';', 'CollectOutput',true};
fid = fopen('temp2.txt','rt');
k = 0;
while ~feof(fid)
    Z = textscan(fid,'%d', R*C, opt{:});
    if ~isempty(Z{1})
        k = k+1;
        S = sprintf('temp2_%02d.txt',k);
        dlmwrite(S,reshape(Z{1},[],R).',';') % might need to translate
    end
end
fclose(fid);

Best Answer

Related Solutions

MATLAB: Making movie from images

MATLAB: Creating multiple equally sized matrices from a single numerical cell

Related Question