MATLAB: Loop through files of remote server

dirdistributedfilesparallelremote

I'm using matlab to perform simulations on a cluster, where the jobs are submitted from a local machine through matlab to the cluster.
A sample submission script might look like this:
%%Get handle to the job scheduler
sched = findResource();
%%Create a job
job = createJob(sched, 'FileDependencies', {'Analysis.m'});
%%Create the tasks
filelist=dir('/dir1/dir2/')
for tidx = 1:length(filelist)
   tasks(tidx) = createTask(job,@Analysis, 1, {tidx}); 
end
%%Submit the job
submit(job)
I'm now trying to obtain and loop through some files on the cluster and run a script on the files, say, Analysis.m
How would I do this to get a file list on the cluster and not on the local machine from which the job-scheduling, and then pass each file to the Analysis.m one at a time?

Best Answer

Perhaps a PARFOR loop on the cluster would be simplest. Something like this:
     function x = doStuff
     % list the files on the cluster
     d = dir( '/path/on/cluster/*.dat' );
     % loop over the files - this will spread the work
     % among the workers
     parfor ii = 1:numel( d )
       x(ii) = someFcn( d(ii).name );
     end
Then, you need to submit 'doStuff' as a matlabpool job to the cluster, something like this:
     job = createMatlabpoolJob( sched, 'MaximumNumberOfWorkers', 4 );
     createTask( job, @doStuff, 1, ... );
Or, in this situation, you can even use the BATCH command to submit the job
     job = batch( @doStuff, 1, {}, 'Matlabpool', 4 );
Related Question