MATLAB: Logging parellel pool worker diaries to single file

dataqueuediaryparellelworker

I'm converting a procedural script into a parallel script. The existing script dumps the command window output using Diary. So each job's output was nicely lumped together. Now though, as I have tasks running in parallel my log file is very disjointed. Not ideal.

Is there a way I can collect each worker's diary and send it to a dataQueue for storage in a single file? In pseudo code it would be:

parfor ...
  run task (including command window outputs)
  send Diary content back to client dataPool for writing to log file
end

Is there some method within my parfor loop of saying 'send(q, thisWorker.Diary)'?

Best Answer

Instead of parfor, you could use parfeval to run stuff on the workers. This will require a bit of restructuring - you'll need to divide up your work into decent-sized chunks (this is something that parfor takes care of for you automatically). Then, you can use the Diary property of each parallel.Future instance.

For example

% request execution on a worker  
f = parfeval(@() fprintf('worker diary text: %d\n', rand()), 0);
wait(f), f.Diary

Related Solutions

MATLAB: Loop through files of remote server

Perhaps a PARFOR loop on the cluster would be simplest. Something like this:

     function x = doStuff
     % list the files on the cluster
     d = dir( '/path/on/cluster/*.dat' );
     % loop over the files - this will spread the work
     % among the workers
     parfor ii = 1:numel( d )
       x(ii) = someFcn( d(ii).name );
     end

Then, you need to submit 'doStuff' as a matlabpool job to the cluster, something like this:

     job = createMatlabpoolJob( sched, 'MaximumNumberOfWorkers', 4 );
     createTask( job, @doStuff, 1, ... );

Or, in this situation, you can even use the BATCH command to submit the job

     job = batch( @doStuff, 1, {}, 'Matlabpool', 4 );

MATLAB: DataQueue and batch combined

It's not possible to use DataQueue with a batch job in this way - DataQueue can only be used to communicate between client and workers in a parallel pool. So, what you could do is this:

parpool(1); % Only 1 worker required
queue = parallel.pool.DataQueue;
queue.afterEach(@handle_message);
future = parfeval(@LoopFunction, 0, queue);

where you need to modify your LoopScript to be a function so that it can work with parfeval.

Best Answer

Related Solutions

MATLAB: Loop through files of remote server

MATLAB: DataQueue and batch combined

Related Question