MATLAB: TreeBagger using obscene amount of memory when run in parallel

nonparametric supervised learning random forestout of memorytreebagger

Hi,

Im experiencing issues when running TreeBagger on a cluster. I run this code on a large cluster with 64 processors and 128 GB of memory. However, when I try to use TreeBagger on my dataset (~200 MB in size) with 5000 trees, matlab errors out after a few hours with OUT of MEMORY issues.

Here are my steps:

1. send a batch job to the cluster via distributed computing toolbox and open a matlabpool with 32 workers.

2. options = statset('UseParallel', 'Always');

3. B= TreeBagger(ntrees, tsp, tsp_label, 'Fboot', fboot, 'Options', options); where ntrees = 5000 and fboot=0.5.

I dont understand why TreeBagger is using so much memory (>128GB). When I run this same job locally on my 16GB computer, the memory use does not exceed 16GB. Am I doing something improperly?

Thanks for your help!

Best Answer

Nicholas,

Each worker in the matlabpool is a separate matlab executable with its own working memory. In the case of TreeBagger, each worker has a separate copy of the TreeBagger data, which includes your full dataset, and eventually, all or most of the trees, plus any additional object contents. Thus, for TreeBagger, total memory consumption tends to increase quasi-linearly with the size of the matlabpool.

If you run in serial mode on your own computer, there is only one copy of this memory. (Though if you run in parallel on K cores locally, there will be K copies of the data.)

You might try to run with a smaller matlabpool if total memory consumption across the matlabpool is a limiting factor.

Best,

Steve

Related Solutions

MATLAB: What fraction of input data is used for out-of-bag observations when creating a TREEBAGGER object using Statistics Toolbox 7.1 (R2009a)

For every tree, the bagger randomly selects N*bagger.FBoot out of N observations with replacement (default) for training. Observations that were not selected for training are out-of-bag observations. If bagger.FBoot=1 (default), on an average roughly 2/3 of input data is selected for training for every tree and the remaining 1/3 is used as out-of-bag observations. This number can fluctuate from one tree to another, and out-of-bag observations for one tree are not identical to out-of-bag observations for another tree.

You can use the following code as an example to determine the fraction of out-of-bag observations per tree.

load imports-85;
Y = X(:,1);
X = X(:,2:end);
ntrees = 50;
for j = [0.5 0.8 1]
      b = TreeBagger(ntrees,X,Y,'oobvarimp','on','Fboot',j);
      [obs vars] = size(b.X);
      num_oob_per_tree = sum(sum(b.OOBIndices))/ntrees;
      fprintf(['\n\nFor ' num2str(ntrees) ' trees and FBoot = ' num2str(j) ':\n'])
      frac_oob_observations = num_oob_per_tree/obs
end

MATLAB: Is the function TreeBagger(NumTrees,Tbl,ResponseVarName) with NumTrees = 300 considered as random forest

TreeBagger grows a forest of trees, but that's a random forest not just with 300 of them.

Best Answer

Related Solutions

MATLAB: What fraction of input data is used for out-of-bag observations when creating a TREEBAGGER object using Statistics Toolbox 7.1 (R2009a)

MATLAB: Is the function TreeBagger​(NumTrees,​Tbl,Respon​seVarName) with NumTrees = 300 considered as random forest

Related Question

MATLAB: Is the function TreeBagger(NumTrees,Tbl,ResponseVarName) with NumTrees = 300 considered as random forest