I'm working on a large data set that is is stored as a cell array X, where each cell contains an array of about 2MB, totaling 1GB. I'm running it in parallel on 50 cores that are each on separate nodes. I get the error "Attempt to serialize data which is too large" when I try to run the whole array . When I run a small array X I don't get the error. I check on the nodes and there's plenty of memory left, it seems the bottleneck is data transmission
I'm using the following code
Y1= cell(1,N1);Y2 = cell(1,N1);parfor i = 1:N1 XX = X{i}; Np = size(XX,1); Y1_temp = zeros(400, Np-1); Y2_temp = zeros(400, Np-1); for j= 1:(Np-1) Y1_temp(:,j) = someFunction; Y2_temp(:,j) = someOtherFunction; end Y1{i} = Y1_temp; Y2{i} = Y2_temp; end
I don't understand why this is, has anyone ever got this error? I couln't find anything online. I also tried writing it as an spmd block where I operate on the local worker data but I get the same error. Any help will be very appreciated.
Regards Germán
Best Answer