MATLAB: How to use a nonshared file system with the Distributed Computing Toolbox 3.1 (R2007a)

MATLAB Parallel Servernonnon-sharednonsharedParallel Computing Toolboxsharedun-sharedunshared

I have a non-shared filesystem which I would like to use with the Distributed Computing Toolbox.

Best Answer

The DCT interface to third party schedulers requires that both a MATLAB client and a MATLAB worker have direct access to a shared directory, known as the Data Location.

You may use your own desktop computer to submit jobs to a compute cluster, as well as to retrieve results. It is often the case that this desktop does not share a network drive with the nodes of the cluster.

Example scripts that work around this problem have been provided with Distributed Computing Toolbox 3.1 R2007a. They are present under the directory

 <matlabroot>\toolbox\distcomp\examples\integration\

where "<matlabroot>" is the directory returned by executing the following at the MATLAB prompt:

matlabroot

The solution implemented by the scripts is as follows. There are two Data Location directories: one on the client’s computer and one on the cluster. For example, the local Data Location might be C:\Temp\DCT, while the cluster Data Location might be /share/DCT.

The configuration for the cluster (stored in the file distcompUserConfig) will contain the hostname of the cluster login node, as well as the name of the cluster Data Location (/share/DCT).

When the MATLAB client creates jobs and tasks, they are stored in the local Data Location (C:\Temp\DCT). A call to submit a job leads to all job information, including the tasks, being copied over to the cluster DataLocation (/share/DCT), before the scheduler’s submit command is called. For example, qsub for PBS and bsub for LSF.

If the MATLAB client calls waitForState(job) or get(job, ‘state’), a timer is launched that checks the state of the job every 10 seconds. It does this by copying the job’s state file from the cluster to the local Data Location.

The scheduler dispatches jobs to the cluster. MATLAB workers start up, execute the tasks and save the results in the cluster Data Location (/share/DCT). They also switch the job state to ‘finished’.

Then the MATLAB client will copy the state file back to the client and recognize that the job has completed. It will then copy the output data to the client’s local Data Location (C:\Temp\DCT), where it becomes available for retrieval by the MATLAB client.

The examples that are shipped with DCT cover the following two scenarios.

1. UNIX client submitting to UNIX cluster

(scp is used to copy data back and forth from the cluster. ssh is used to issue commands on the cluster login node e.g.

ssh cluster01 “qsub jobscript”

2. Windows client submitting to UNIX cluster

Putty is used to copy data and issue commands, pscp for data transfer and plink for command execution.

Note: The solution can be adapted to cover the case of a Windows client submitting to Windows cluster.

Example Scripts:

The README files in each of the directories provide more detailed instructions.

1. PBS

 MATLAB\toolbox\distcomp\examples\integration\pbs\nonshared\unix
 MATLAB\toolbox\distcomp\examples\integration\pbs\nonshared\windows

2. LSF

 MATLAB\toolbox\distcomp\examples\integration\lsf\nonshared\unix
 MATLAB\toolbox\distcomp\examples\integration\lsf\nonshared\windows

3. Other schedulers

You may adapt the scripts provided for PBS and LSF

Changing the remote command/copy mechanism:

All remote data transfer and command execution is controlled by three files. You may edit these files to use other mechanisms like rsh, psexec, etc. The files are:

 copyDataToCluster.m
 copyDataFromCluster.m
 runCmdOnCluster.m

These files are also shipped with the example scripts, e.g.

MATLAB\toolbox\distcomp\examples\integration\pbs\nonshared\unix\copyDataToCluster.m

Related Solutions

MATLAB: Can a worker registered with the MathWorks Job Manager be accessed from another scheduler (such as PBS)

A worker is a MATLAB session executing on a machine (node), and is associated with a job manager or with the scheduler that created it. Multiple schedulers can execute tasks on the same machine. For each scheduler accessing the machine, a different MATLAB worker session is started.

Each worker is dedicated to its job manager or scheduler, even if the same machine is being used by different schedulers. An idle worker that is associated with a job manager remains available (idle) to that job manager when another scheduler starts a worker on the same machine.

Note that a worker started by a third-party scheduler runs for only as long as it takes it to evaluate its task, then it is stopped. A worker that is registered with a job manager remains running and available between tasks.

MATLAB: Is it possible to use mixed client-cluster OS with PBSPro with Parallel Computing toolbox 3.3 (R2008a)

Due to difficulties in setting up uniform permissions for PBS Pro clusters involving Windows client machines and UNIX cluster machines, the Parallel Computing Toolbox does not support this configuration. In cases where PBS Pro is set up so that Windows machines can submit directly to a UNIX cluster, the attached files allow submission from the Windows client machines. In this case, it is necessary to have a shared filesystem between the Windows submission machines and the UNIX cluster machines. Also, the “getDebugLog” function of the scheduler object will have reduced functionality because there may be difficulties in delivering the output files from the cluster back to the client machine.

Here are the steps:

1. Find the $MATLABROOT directory where $MATLABROOT is the MATLAB root directory on your machine, as returned by typing

matlabroot

at the MATLAB Command Prompt.

2. Quit MATLAB.

3. Copy the matlabroot\toolbox\distcomp directory for backup and save it as matlabroot\toolbox\distcomp_nopatch where matlabroot has the value of the directory found in step 1.

4. Unzip the archive to extract the files to matlabroot\toolbox\distcomp directory. The following files should be overwritten by the archive:

@distcomp\@pbsscheduler\pSetClusterOsType.m

bin\util\pbs\execute.sh

bin\util\pbs\header.sh

bin\util\pbs\job-array-prologue.sh

bin\util\pbs\nonshared-copyin.sh

bin\util\pbs\nonshared-copyout.sh

bin\util\pbs\pbsParallelWrapper.sh

bin\util\pbs\rcp.sh

5. Restart MATLAB

6. After restarting MATLAB, issue the following command at the MATLAB Command Prompt to test the mixed cluster:

sched = findResource('scheduler','type','pbspro')   
DataLocation = struct('pc',  'XXX', ... 
           'unix','YYY); % where XXX is the Window path to the DataLocation directory for the client and YYY is the Unix or Linux path to the DataLocation directory for the cluster.
set(sched, 'DataLocation', DataLocation); 
set(sched, 'HasSharedFileSystem', true); 
set(sched, 'ClusterMatlabRoot', 'ML'); %where ML is MATLAB root for cluster
set(sched, 'ClusterOSType', 'unix'); 
job = createJob(sched); 
createTask(job, 'rand', 1, {5}) 
submit(job) 
waitForState(job); 
getAllOutputArguments(job)

Best Answer

Related Solutions

MATLAB: Can a worker registered with the MathWorks Job Manager be accessed from another scheduler (such as PBS)

MATLAB: Is it possible to use mixed client-cluster OS with PBSPro with Parallel Computing toolbox 3.3 (R2008a)

Related Question