MATLAB: Is there a built-in function to automatically transfer the output files created by a worker to the client system as part of a job using Parallel Computing Toolbox 4.1 (R2009a)

clientfileMATLAB Parallel ServerParallel Computing Toolboxsharedsystemtransferworker

I am using Parallel Computing Toolbox 4.1 (R2009a) to execute tasks on a cluster of workers. Each of the tasks generates MAT files and text files. However, these files are created locally on the workers. I would like to be able to automatically transfer these files back to the client system once the task has completed preferably as part of the FinishedFcn callback.

I would like to know if there is some functionality in Parallel Computing Toolbox to automatically transfer generated result files from the worker to a target directory on the client system. Ideally, this would behave just like job results.

Best Answer

The ability to transfer files automatically from the worker to the client system is not available in the Parallel Computing Toolbox.

To work around this issue, you can try the following:

1. Use the MOVE or COPY command as part of the FinishedFcn callback.

2. Create the files at a location which is accessible to both the workers and the client.

Related Solutions

MATLAB: Is it possible to use mixed client-cluster OS with PBSPro with Parallel Computing toolbox 3.3 (R2008a)

Due to difficulties in setting up uniform permissions for PBS Pro clusters involving Windows client machines and UNIX cluster machines, the Parallel Computing Toolbox does not support this configuration. In cases where PBS Pro is set up so that Windows machines can submit directly to a UNIX cluster, the attached files allow submission from the Windows client machines. In this case, it is necessary to have a shared filesystem between the Windows submission machines and the UNIX cluster machines. Also, the “getDebugLog” function of the scheduler object will have reduced functionality because there may be difficulties in delivering the output files from the cluster back to the client machine.

Here are the steps:

1. Find the $MATLABROOT directory where $MATLABROOT is the MATLAB root directory on your machine, as returned by typing

matlabroot

at the MATLAB Command Prompt.

2. Quit MATLAB.

3. Copy the matlabroot\toolbox\distcomp directory for backup and save it as matlabroot\toolbox\distcomp_nopatch where matlabroot has the value of the directory found in step 1.

4. Unzip the archive to extract the files to matlabroot\toolbox\distcomp directory. The following files should be overwritten by the archive:

@distcomp\@pbsscheduler\pSetClusterOsType.m

bin\util\pbs\execute.sh

bin\util\pbs\header.sh

bin\util\pbs\job-array-prologue.sh

bin\util\pbs\nonshared-copyin.sh

bin\util\pbs\nonshared-copyout.sh

bin\util\pbs\pbsParallelWrapper.sh

bin\util\pbs\rcp.sh

5. Restart MATLAB

6. After restarting MATLAB, issue the following command at the MATLAB Command Prompt to test the mixed cluster:

sched = findResource('scheduler','type','pbspro')   
DataLocation = struct('pc',  'XXX', ... 
           'unix','YYY); % where XXX is the Window path to the DataLocation directory for the client and YYY is the Unix or Linux path to the DataLocation directory for the cluster.
set(sched, 'DataLocation', DataLocation); 
set(sched, 'HasSharedFileSystem', true); 
set(sched, 'ClusterMatlabRoot', 'ML'); %where ML is MATLAB root for cluster
set(sched, 'ClusterOSType', 'unix'); 
job = createJob(sched); 
createTask(job, 'rand', 1, {5}) 
submit(job) 
waitForState(job); 
getAllOutputArguments(job)

MATLAB: How to determine the amount of available memory on a given worker on a cluster using Parallel Computing Toolbox 4.1 (R2009a)

The ability to obtain information about the available memory on each worker using the "Admin Center" utility or the "Configurations Manager" is not available in Parallel Computing Toolbox.

As a workaround, you can use the following code snippets to obtain this information. Use the workaround that best fits your need:

1. If you want information about the available statically i.e. before submitting a job or after a job has been run, you can create a job that queries for memory information and execute it on all workers that are part of the cluster. The code snippet below indicates how this can be done:

sched = findResource('scheduler', 'configuration', 'dinesh_jm');
myJob = createParallelJob(sched);
set(myJob, 'MinimumNumberOfWorkers', sched.ClusterSize);
set(myJob, 'FileDependencies', {'memoryCheck.m'});
myTask = createTask(myJob, @memoryCheck, 2, {});  
submit(myJob);
wait(myJob);
allTasks = get(myJob, 'tasks');
memoryInfo = get(allTasks, 'OutputArguments');
workerInfo = get(allTasks, 'Worker');

where the function "memoryCheck" is defined as below:

function [result1 result2] = memoryCheck()
if ispc
      [result1 result2] = memory;
else
      [result1 result2] = system('free');
end

2. To obtain memory information, as the tasks are being executed, include the call to the function "memoryCheck" in the function that is being executed as part of the task periodically. The results returned by the function can be written to a text file which can be inspected to obtain the memory usage trend.

Best Answer

Related Solutions

MATLAB: Is it possible to use mixed client-cluster OS with PBSPro with Parallel Computing toolbox 3.3 (R2008a)

MATLAB: How to determine the amount of available memory on a given worker on a cluster using Parallel Computing Toolbox 4.1 (R2009a)

Related Question