MATLAB: Is it possible to reuse MATLAB files passed to workers as ‘FileDependencies’ without resending them in the Distributed Computing Toolbox

editeditededitorMATLAB Parallel ServerParallel Computing Toolboxrefreshupdate

I submit several jobs which all use the same files. I would like my jobs to share the files which I submit using the 'FileDependencies' parameter without resending them.

Best Answer

The ability to share files passed to the workers using 'FileDependencies' is unavailable with Distributed Computing Toolbox.

To workaround this issue, use the 'PathDependencies' property of the job.

To get help on 'PathDependencies' execute the following command at the MATLAB prompt:

doc PathDependencies

Related Solutions

MATLAB: Is the ‘FileDependencies’ property unused even when specified in the Local Scheduler Configuration Properties, when I use the BATCH function in the Parallel Computing Toolbox 4.0 (R2008b)

The ability to use Configurations Properties without explicitly specifying the configuration name in the BATCH function is not available in Parallel Computing Toolbox 4.0 (R2008b).

To workaround this issue and to use the configuration's settings applied to the job properties, you must explicitly specify the configuration in the BATCH function.

job=batch('dc_test','matlabpool',3, 'configuration', 'local')

MATLAB: How to use FileDependencies to make a standalone executable program available to the workers running MATLAB Distributed Computing Engine 2.0 (R14SP3+)

This issue occurs because the temporary directory for FileDependencies is added to the MATLAB path only, and not to the system path. This is the expected behavior. If you are using Release R2006b or later, use the getFileDependencyDir function in your task function to locate the executable on the workers' path, for example:

%Find the current directory for FileDependencies. 
ddir = getFileDependencyDir; 
%Change to that directory to invoke an executable. 
cdir = cd(ddir); 
%Invoke the executable. 
[OK, output] = system('myexecutable'); 
%Change back to the original directory. 
cd(ddir);

The getFileDependencyDir function does not ship with versions of MATLAB Distributed Computing Engine prior to Release R2006b. If you are using an older version, download the attached P-file, getFileDependencyDir.p, and make it available to your workers via the "FileDependencies" property of the job:

set(job, 'FileDependencies', {'myFile.exe', 'getFileDependencyDir.p'})

In your task code, use getFileDependencyDir in the same way as shown above, i.e., have the workers navigate to the directory containing the executable to run it from there.

An alternative to using "FileDependencies" is to put the executable on a shared file system accessible to all workers and update the system paths of all workers to include the path to the executable.

Best Answer

Related Solutions

MATLAB: Is the ‘FileDependencies’ property unused even when specified in the Local Scheduler Configuration Properties, when I use the BATCH function in the Parallel Computing Toolbox 4.0 (R2008b)

MATLAB: How to use FileDependencies to make a standalone executable program available to the workers running MATLAB Distributed Computing Engine 2.0 (R14SP3+)

Related Question