MATLAB: Function code not updated when running “parfor” on a cluster

Question

I have a function file in a different path. When I run the following code:

 

parfor i=1:1,
    addpath('path-to-function');
    testFunction();
    rmpath('path-to-function');
 end

the function executes fine. However, if I make changes to testFunction.m, these are not seen by the workers until I call "rehash".

Answer 1

The reason for this is that the workers in your parallel pool are "headless" versions of MATLAB, and therefore behave, in many ways, similarly to how MATLAB would behave. When MATLAB uses a function, it sometimes caches that function and doesn't necessarily update this cached version immediately. You can force an update like that with the "rehash" command, which you have been using.

Specifically for the parallel pool, however, there is an alternative workflow that achieves this update. This workflow relies on attached files.

You can attach a file (or folder) to a parallel pool using the command "addAttachedFiles". Attaching files to a pool transfers a copy of the files to each worker.

If you have a pool already open, you can attach files by entering the commands:

 

pool = gcp; % Get current pool;
pool.addAttachedFiles('Path-to-file');

The documentation page for "addAttachedFiles" can be found in the following link:

https://www.mathworks.com/help/distcomp/addattachedfiles.html

If you want to create a pool with a number of files attached, you can do that with the command:

 

parpool(<poolname>, 'AttachedFiles', 'Path-to-file');

In both cases, you can replace the string 'Path-to-file' with a cell array including multiple files and even include folders: {'Path-to-one-file', 'Path-to-a-folder'}.

If you have attached files to the pool, it is not required to add their path to the MATLAB path before using them. Your code will therefore be:

 

parfor i=1:1,
    testFunction();
end

Finally, after making any updates to a file attached to the parallel pool, you can update the files using the commands:

 

pool = gcp;
pool.updateAttachedFiles;