MATLAB: How to store and retrieve multiple data sent from worker nodes to client node in MATLAB

data importparallel computingParallel Computing Toolbox

q = parallel.pool.DataQueue;
parfor i = 1:20  
    data = ['test - ', num2str(i)];
    send(q,data);
end

This is the code I used to send data from the worker nodes to the client node. But I don't know if it helps me store the data I send from the workers to the client. The Queue length happens to be 20 which is correct but if it contains the data, I don't know how to retrieve it. I tried using a PollableDataQueue and poll() but it retrieves only one value which I guess is the last value.

So my questions are :

Is the method I'm using to store data sent from the workers correct?
If yes, how do I retrieve the stored data?
If no, how do I store data sent from the workers in the client?
Is the method I'm using to send data from the workers to the client correct?

Best Answer

https://www.mathworks.com/help/distcomp/aftereach.html

Related Solutions

MATLAB: Parallel pool in batch processing

Until R2016b, all workers received a single thread. The relevant R2016b release note is:

"Multithreaded Workers: Use multiple computational threads on your MATLAB workers

MATLAB workers used to run in single-threaded mode. Now you can control the number of computational threads so that workers can run in multithreaded mode and use all the cores on your cluster. This enables you to increase the number of computational threads, NumThreads, on each worker, without increasing the number of workers, NumWorkers. If you have more cores available, increase NumThreads to take full advantage of the built-in parallelism provided by the multithreaded nature of many of the underlying MATLAB libraries. For more information, see Create and Modify Cluster Profiles."

The documentation for batch shows examples of creating a batch without opening a parpool, and an example of requesting a pool. No explicit "parpool" statement is used.

If you are running on a cluster, you might need to parcluster() and batch() against that cluster.

MATLAB: How to send data on the fly to a worker when using parfeval

I think it's possible to do this sort of thing by getting the workers to create a parallel.pool.PollableDataQueue, and getting the client to send messages to the worker. It's a bit awkward to set up, and in the example below, I'm going to use a parallel.pool.Constant to get the worker to hold onto the queue instance.

The general plan is this:

Get the worker to construct a pollable dataqueue
Send the dataqueue instance back to the client
Set the worker off into an loop waiting for messages from the client
Send messages from the client to get the worker to do stuff
When done, get the client to send a "poison pill" which gets the worker to return the result.

Note the code below only works for a pool of size 1. To run on a larger pool, you would need to use parfevalOnAll rather than parfeval, and you'd also need to manage the multiple queues from the workers.

% First, create a parallel pool if necessary
if isempty(gcp())
    parpool('local', 1);
end
% Get the worker to construct a data queue on which it can receive
% messages from the client
workerQueueConstant = parallel.pool.Constant(@parallel.pool.PollableDataQueue);
% Get the worker to send the queue object back to the client
workerQueueClient = fetchOutputs(parfeval(@(x) x.Value, 1, workerQueueConstant));
% Get the worker to start waiting for messages
future = parfeval(@doStuff, 1, workerQueueConstant);
% Send a few messages to the worker
for idx = 1:10
    send(workerQueueClient, idx);
end
% Send [] as a "poison pill" to the worker to get it to stop
send(workerQueueClient, []);
% Get the result
fetchOutputs(future)
% This function gets the worker to keep processing messages from the client
function out = doStuff(qConstant)
q = qConstant.Value;
out = 0;
while true
    % Wait for a message
    data = poll(q, Inf);
    if isempty(data)
        return
    else
        out = out + data;
    end
end
end

Best Answer

Related Solutions

MATLAB: Parallel pool in batch processing

MATLAB: How to send data on the fly to a worker when using parfeval

Related Question