MATLAB: How to reset a MATLAB Distributed Computing Engine worker session that appears to be hung

hanghungMATLAB Parallel ServermdceParallel Computing Toolboxworker

There are two possible phases in which an MDCE worker session can hang:

– When I start a worker with the startworker command, and the system does not return the prompt to me, the worker could be hung. (If this is the case, skip step 1 in the solution.)

– If a task (and therefore its job) appears to be stuck in the running state, or if a task times out, it could be because of a hung worker session.

Best Answer

The solutions to clearing a hung worker are presented here in the order of safest to most drastic. You should try them in the suggested order, testing after each step to see if the problem is cleared.

1. If a job is stuck in the running state because one of its tasks is stuck running on a hung worker, you can try destroying the job from the client MATLAB session by using the destroy function. Submit another job and see if the worker in question now properly evaluates its tasks.

2. Use the stopworker command on the worker node to end the worker session. Restart the worker session with the command startworker -clean.

3. Shut down all MDCE services on the worker node with the command mdce stop. Note that this will shut down all worker and job manager sessions on the node. Restart all sessions accordingly.

4. If MDCE stop does not return a prompt, then as a last resort you can delete the worker's checkpoint directories and reboot the node to restart its MDCE sessions.

Related Solutions

MATLAB: Is the communicating job stuck behind a concurrent job in queue in R2017b

The resolution is to avoid starting a parallel pool when submitting other jobs to the cluster.

The "parpool" object is generally used for interactive "parfor" or "smpd" operations. When the parallel pool is created with "parpool", it creates a concurrent job on the cluster that readies all of the workers to be available for when the user interactively calls "parfor" or "smpd". The concurrent job will continue running, causing other jobs to wait in queue until the parallel pool is shut down.

MATLAB: Do I see an increase in memory usage on the workers as I keep running jobs successively when using the Distributed Computing Toolbox

The DESTROY command in the Distributed Computing Toolbox removes the job object reference or task object reference from the local session, and removes the object from the job manager memory.

destroy(obj)

However, when a job is finished, it remains in the job manager even if you clear all the objects from the client session. The job manager keeps all the jobs it has executed until you restart the job manager in a clean state.

Additionally, tasks running on the workers may have allocated memory that is not released: global or persistent variables, locked functions, memory leaks or persistent data in MEX files, etc.

After destroying the job or task, you can restart a worker using the RestartWorker property. The memory is then returned to the worker.You can set RestartWorker to true (or logical 1) if you want the job to restart the MATLAB session on any workers before they evaluate their first task for that job.

For example, submitting the following job after your main jobs have finished will force all the workers to restart their MATLAB instances:

     function forceRestartAllWorkers(sched)
       %After this function completes, the MDCE will have 
       % restarted each worker.   The argument 'sched' should
       % be a scheduler object, that is, the result of
       % a findResource operation.
       memFlushJob = createParallelJob(sched, 'RestartWorker', true);
       t = createTask(memFlushJob, @() true, 0, {});
       submit(memFlushJob);
       waitForState(memFlushJob);
       destroy(memFlushJob);
end

More information about this can be found at the following URLs:

<http://www.mathworks.com/help/releases/R2010b/toolbox/distcomp/restartworker.html>

Best Answer

Related Solutions

MATLAB: Is the communicating job stuck behind a concurrent job in queue in R2017b

MATLAB: Do I see an increase in memory usage on the workers as I keep running jobs successively when using the Distributed Computing Toolbox

Related Question