MATLAB: Avoid sending large array to all workers in parfor loop

MATLABmemoryParallel Computing Toolboxparfor

Hello —
I'm processing some fairly large point clouds which are stored as 8-column tables with billions of rows (stored as datastores/tall arrays, but that's probably not important here).
In my workflow I load up a reasonable chunk of the point cloud (~3.5 GB) into memory and then I ultimately generate an image based on the data. The image is 37×44 pixels which means that I'm indexing into this point cloud 37 times row-wise and 44 times column wise. This is a very parallelizable task and am running the outer loop with parfor. However, frequently, I'm erroring out because workers abort. The workers seem to abort when my memory hits my limit (32 GB).
I think my problem is obvious when you look at my code below but i'm not sure the best way to fix it. Note below where pcl_line is subset from pcl. I'm assuming that here pcl (which is my 3.5 GB variable) is still being sent to every worker which seems bad. How can I avoid this though? Is this a job for C = parallel.pool.constant(pcl)? Seems promising but my knowledge here is a bit shaky. If not, other thoughts? — Thanks much, Mike
%set up holders for image outputs
    tot_skew = NaN(n_num_px, e_num_px); %just two example output images of many
    tot_kurt = NaN(n_num_px, e_num_px);
     
    parfor ii = 1:length(n_chunk_bounds)-1
        
        %temporary vars by row
        temp_skew = NaN(1,e_num_px);
        temp_kurt = NaN(1,e_num_px);
           
        %slice pcl by line (1/37th size of pcl since there are 37 rows in output image
        sub_idx_n = find(pcl.n<n_chunk_bounds(ii) & pcl.n>=n_chunk_bounds(ii+1));
        pcl_line = pcl(sub_idx_n,:);  %<--guessing this is the problem since pcl is still inside the parfor loop?
	%run code for each output pixel for a given image line
        for j = 1:length(e_chunk_bounds)-1
            sub_idx = find(pcl_line.e>=e_chunk_bounds(j) & pcl_line.e<e_chunk_bounds(j+1));
           temp_skew(j) = skewness(pcl_line.h(sub_idx));  
           temp_kurt(j) = kurtosis(pcl_line.h(sub_idx));          
        end
        %final assignment
        tot_skew(ii,:) = temp_skew;
        tot_kurt(ii,:) = temp_kurt;
    end

Best Answer

[...] extract the chunks outside the parfor loop, into a cell array, and index the cell array inside the loop.
Related Question