MATLAB: Cluster validation used to pass, now fails after upgrade to 2011b

MATLAB Parallel Serverparallel computingParallel Computing Toolboxvalidation

We have an SGE cluster on which we were running MATLAB. Worked fine till R2011a but fails after we have upgraded to R2011b. Validation just hangs and does not even time out from my Desktop (MAC OSX). Used to work before the upgrade. Cannot submit jobs from the desktop either.
No other change made on the cluster. (eg filesystem, paths, etc..)

Best Answer

Ok, Does the local validation pass? Run the validation from the command window so that it gives out more details:
setSchedulerMessageHandler(@disp) % additional debug information
jm =findResource('scheduler','configuration','NAME_OF_CONFIGURATION')
If the command window throws out a number of old jobs here, that means there was some issue when you moved up and it is trying to find metadata files of old jobs on the cluster and in your working directory and hence cannot complete the validation (this causes it to not time out either since it is still working). IF so do the following procedure
1. clear the local_scheduler _data (you can also rename the folder) /Users/USERNAME/.matlab/local_scheduler_data/R2011a
2. Empty all the metadata files and job directories in the DataLocation (on your desktop) - Parallel>Manage Configurations>Slect your Configuration and find the Data Location > Folder where Job directory is stored.
3. Remove the files that Matlab writes on the cluster for each job i.e Job#.lockstate, Job#.in.mat, Job#.out.mat, Job#.common.mat, Job#.jobout.mat, and Job#.state.mat
(In some cases these files can be found where you put the ParallelSubmitFcn, and SubmitFcn on the cluster)
Rerun Validation.
Related Question