MATLAB: Can a worker registered with the MathWorks Job Manager be accessed from another scheduler (such as PBS)

dctlabMATLAB Parallel ServermdceParallel Computing Toolboxschedulerworker

One of my cluster computers is running a MATLAB worker associated with a MathWorks job manager. On the same machine, I want to start workers by using another scheduler. I want to know whether or not the original worker will be accessible to the other scheduler.

Best Answer

A worker is a MATLAB session executing on a machine (node), and is associated with a job manager or with the scheduler that created it. Multiple schedulers can execute tasks on the same machine. For each scheduler accessing the machine, a different MATLAB worker session is started.

Each worker is dedicated to its job manager or scheduler, even if the same machine is being used by different schedulers. An idle worker that is associated with a job manager remains available (idle) to that job manager when another scheduler starts a worker on the same machine.

Note that a worker started by a third-party scheduler runs for only as long as it takes it to evaluate its task, then it is stopped. A worker that is registered with a job manager remains running and available between tasks.

Related Solutions

MATLAB: What is Job Manager Security and how to implement it in the MATLAB Parallel Server configuration

As of R2010b, Job Manager Security was introduced to add some extra security to jobs that a submitted using the MathWorks Job Manager. By utilizing this feature, you can restrict who has access to job data, what user account is used to execute tasks, and which users are allowed to submit jobs. For more information about Job Manager Security see the documentation here:

https://www.mathworks.com/help/matlab-parallel-server/set-matlab-job-scheduler-cluster-security.html

NOTE: The Job Manager Security Tools are only available if you are using the MathWorks Job Manager provided with MATLAB Parallel Server.

Security Levels

==========

By default, Job Manager Security is off. There are 4 levels that can be used:

0: (Off) Job Manager Security is off. Any user can access the jobmanager and data. Tasks will run as the user who started the mdce service

1: (User identified) Jobs are identified with a user. When the user accesses the job manager, a user name is required. This user name does not have to match any user on the system and no passwords are required. Any user can access any job, but a dialog box warns users who access other users' jobs.

2: (Password protected) Same as level 1, however, users can only access their own jobs and users must provide a password for jobs. When you start the jobmanager, you create an admin account for the jobmanager. This admin has full access to job data as well. Usernames and passwords do not need to match the system/network name.

3: (System/Network Authenticated) Similar to level 2, however, user accounts must match that of the system/network. Additionally, when tasks are run, these tasks run as the user who submits the job. Requires the use of secure communication using a shared secret key file (see below).

Security Options

==========

In addition to these levels there are some additional options you can set for Job Manager Security. These options as well as the security level are set the in mjs_def file located in $MATLAB/toolbox/distcomp/bin (where $MATLAB represents the folder where MATLAB is installed). The mjs_def has the following options for Job Manager Security:

set SECURITY_LEVEL=

Specifies the security level as above.

set USE_SECURE_COMMUNICATION=

Specifies whether the shared key is used for security between the jobmanager and workers. If level 3 security is set, this value must be true.

set SHARED_SECRET_FILE=

Path to the shared secret file. If you set USE_SECURE_COMMUNICATION to true, you must point to the secret file.

set ALLOW_CLIENT_PASSWORD_CACHE=

If true, the jobmanager remembers user passwords for future sessions.

set ALLOWED_USERS=

Names the users that are allowed to submit jobs when using level 1 or higher. To allow all users, set the value to ALL.

set WORKER_DOMAIN=

When using security level 3 on a Windows system, this specifies the required domain name that the workers will run as.

Creating a Shared Secret File

==========

When using the secure communication method, you need to use a shared key file. To generate this key file you can use the following commands:

$MATLAB/toolbox/distcomp/bin/createSharedSecret (UNIX)

$MATLAB\toolbox\distcomp\bin\createSharedSecret.bat (Windows)

Where $MATLAB is the path to the MATLAB installation. Once complete, you can either place the file on a network share and point each node to it in the mjs_def file, or copy the file locally. If SHARED_SECRET_FILE= is unset, the shared secret is expected in %CHECKPOINTBASE%\security\secret. (See the mjs_def file to find where CHECKPOINTBASE is located).

Implementing Job Manager Security

==========

Job Manager Security is setup when you start the Job Manager and workers. If you already have a Job Manager setup and want to change the security, you must first stop that Job Manager and its associated workers and restart the MATLAB Parallel Server service (run "mdce stop" and then run "mdce start" or restart the service from the Windows Services applet). Once complete, you can use the Admin Center or command line to start the Job Manager and workers. If prompted for a job manager administrator account, this account will serve to be the main account for starting and stopping workers in addition to accessing all job data.

NOTE: Starting in R2019a the following name changes occurred: * MATLAB Distributed Computing Server was renamed to MATLAB Parallel Server * mdce_def was renamed to mjs_def * mdce binary was renamed to mjs

MATLAB: How many MATLAB Distributed Computing Engine worker sessions should I run on the cluster

You can run as many MDCE worker sessions on a machine as you want to, but the question is whether you gain any enhancement in performance by running more workers. This issue depends on many factors, such as

- How fast is the processor speed and how much memory does each machine have?

- Are the tasks computationally intensive, or do they involve a lot of idle time?

- Is there a lot of data transferred across the network for each task?

- How does the execution time of the tasks compare to the time to transfer data?

Without this specific information, you cannot derive a clear formula about the number of worker sessions to run on your machines. But a general guideline for forming a basis for comparison is to run one MDCE worker session per processor or core. So on single-processor/single-core machines, start with one worker session; on dual-processor/dual-core machines, run two workers; etc.

Starting with MATLAB 7.4 (R2007a), it is possible to configure MATLAB to use multiple threads to perform computations. In MATLAB Distributed Computing Engine, this behavior is disabled. As a result, you should still use the general rule of one worker per processor/core.

How many schedulers you should run and how many workers should be registered per scheduler depends on how many workers you need to perform the jobs of your application. You need to register at least the minimum number of workers needed for your job with a single job manager or other third-party scheduler. Given that, you can configure your workers and schedulers in whatever ratio works best for your cluster. Generally, you minimize the number of idle workers by minimizing the number of schedulers. If all your jobs are submitted to the same scheduler, that scheduler can make best use of all available workers. It does not matter how many client sessions connect to a job manager; it makes no difference to the job manager if it receives many jobs from each of a few client sessions, or a few jobs from each of many client sessions. The same applies to schedulers other than the job manager.

Best Answer

Related Solutions

MATLAB: What is Job Manager Security and how to implement it in the MATLAB Parallel Server configuration

MATLAB: How many MATLAB Distributed Computing Engine worker sessions should I run on the cluster

Related Question