MATLAB: Pre-Allocate structure with String / Datetime fields slows code down considerably

Question

Hi,

I am trying to read through and sort two large .txt files, around 300 mb at the largest.

Originally, for each line of code I read, I would re create the matrix lile this:

strarray.full = [strarray.full ; new_info]
strarray.newdate = [strarray.newdate ; new_info ] 

This slowed down considerably once the files reached around 20 mb. I've seen that Pre Allocating matrices prevent MATLAB from having to re create the growing matrix each iteration. So now I have the following:

strarray.newdate =  NaT(2000000,1);
strarray.full = strings(2000000,1);

where I have a counting varaible ' j ' that counts each time something should be added into the matrix.

strarray.full(j,1) = new_info;
strarray.newdate(j,1) = new_info;

When I did this, the code slowed down considerably, both starting off slower and slowing down faster as time progressed. After running a profiler, it says that nearly all the time is spent putting the info into the pre-allocated matrix.

I've got permission to attach the file, but I cant attach the .txt files directly so I have to strip it down here.

.txt Format 1:

Datetime2 ~ *string* ~ *string* ~ *string*
	           *string*
Datetime2 ~ *string* ~ *string* ~ *string*
	           *string*
	           *string*
	           *string*
	           *string*
Datetime2 ~ *string* ~ *string* ~ *string*
	           *string*
	           *string*

.txt Format 2:

datetime1 ~ *string* ~~~ *string* ~~~ *string* ~*~
datetime1 ~ *string* ~~~ *string* ~~~ *string* ~*~
datetime1 ~ *string* ~~~ *string* ~~~ *string* ~*~

Thanks.

Answer 1

You are not using struct array. You are putting newdate (which is a datetime array) and full (which is a string array) into a struct strarray (see code difference below). In this case, I wonder if you just use newdate=NaT(2e6,1) and full=strings(2e6,1) directly would be faster. After all, combine these two big array into one struct won't help at all.

You can try struct array following the below pattern to see if it helps. I doubt it.

s1.newdate=NaT(20,1);
s1.newdate(1)
s1.newdate(20)
s2(20).newdate=NaT;
s2(1).newdate
s2(20).newdate