MATLAB: How to configure MATLAB Parallel Server using the MATLAB Job Scheduler to work within a firewall

MATLAB Parallel Server

I am installing MATLAB Parallel Server and Parallel Computing Toolbox. I plan on using the MATLAB Job Scheduler.

My network has a firewall, and I would like to know which ports need to be opened.

Best Answer

All ports listed here should be opened to inbound traffic from any machine unless otherwise stated. Outbound communication for cluster happens through ephemeral ports.

======================================================================================

Cluster Machine Ports

The BASEPORT is set to 27350 by default in the mjs_def file found in: MATLABROOT/toolbox/parallel/bin/

NOTE: In R2019a and later, mdce_def was renamed to mjs_def.

These formulas should be calculated per machine and should reflect the number of each type of service running on that machine rather than across the entire cluster.

nJ is the number of job managers running on the machine.
nW is the number of workers running on the machine.

R2010b - R2014b:

BASEPORT              to        BASEPORT+5+nJ+3*nW
BASEPORT+1000         to        BASEPORT+1000+2*nW   (only to other cluster machines)

R2015a - R2016b:

BASEPORT              to        BASEPORT+5+nJ+3*nW
BASEPORT+1000         to        BASEPORT+1000+2*nW   (only to other cluster machines)

Two additional required port ranges were introduced in these versions. The default value of these new port ranges are shown in brackets below:

JOBMANAGER_PEERSESSION_MIN_PORT  to   JOBMANAGER_PEERSESSION_MAX_PORT (14351 to 14383)
WORKER_PARALLELPOOL_MIN_PORT     to   WORKER_PARALLELPOOL_MAX_PORT    (14384 to 14448)

To customize the jobmanager_peersession or worker_parallelpool ranges please follow the instructions in:

<https://www.mathworks.com/matlabcentral/answers/375539>

R2017a+:

Port ranges in this version were simplified.

BASEPORT              to        BASEPORT+6+nJ+4*nW
BASEPORT+1000         to        BASEPORT+1000+2*nW   (only to other cluster machines)

======================================================================================

Client Machine Ports

These port ranges can be controlled using the function pctconfig in MATLAB on the client machine.

R2006b - R2009a:

27370     to    27371

R2009b - R2014b:

27370     to    27470

R2015a - R2016b:

No inbound port range needed.
Outbound ports 14351 through 14351 + nJ

R2017a+:

No inbound port range needed.
Outbound ports 14351 through 14351 + nJ

======================================================================================

Admin Center

Port 7 on the machine using Admin Center needs to be open for all connectivity tests to pass without warning. This port is *not *required for cluster setup or general cluster operation.

======================================================================================

Online Licensing

Port 443 for https needs to be open for outbound communication from cluster machines if the cluster is set to use Online Licensing.

---------------------------------------------------------------------------

Additional note/update regarding R2019a:

mjs_def.bat is located in $MATLAB/R20XXx/toolbox/distcomp/bin for R2019a and earlier releases.
In R2019a and later, mdce_def was renamed to mjs_def.

Related Solutions

MATLAB: How to configure MATLAB Parallel Server using a 3rd party scheduler to work within a firewall

If you plan on using the MATLAB Job Scheduler as your scheduler please see this information instead:

https://www.mathworks.com/matlabcentral/answers/94254

======================================================================================

Cluster Ports

For communicating jobs and parallel pools the SMPD process opens a listening port in the range: 20000 to 30000

The exact port is based on the job ID. This port is only used for communication between workers. Access to this port is not required from outside of the cluster.

======================================================================================

Client Ports

Ports only need to be opened for interactive parallel pool jobs. By default, the port range needed is: 27370 to 27470

This port range can be controlled using the function pctconfig in MATLAB on the client machine.

The client machine will be expecting inbound communications on this port range from the cluster workers. The cluster workers will use ephemeral ports to make their outbound connection.

If the MATLAB client machine does not have the daemons needed for the particular cluster submission on the same machine then submission to the cluster will take place via ssh using port 22. This will apply for generic integration script clusters using remote or non-shared script templates.

======================================================================================

FlexNet Licensing Ports

The Network License Manager will need the port 27,000 and 1 additional port to be open for inbound connections on the machine it is running on. The MATLAB workers will need to be able to connect to these listening ports to successfully request licenses. These ports can be configured please see this information: https://www.mathworks.com/matlabcentral/answers/96756.

======================================================================================

MathWorks Hosted Licensing

Port 443 for https needs to be open on for outbound communication from cluster machines if the cluster is set to use Online Licensing.

NOTE: Starting in R2019a the following name changes occurred:

MATLAB Distributed Computing Server was renamed to MATLAB Parallel Server
mdce_def was renamed to mjs_def
mdce binary was renamed to mjs

MATLAB: Are the custom port ranges for an MATLAB Parallel Server cluster using the MATLAB Job Scheduler not being used (R2015a to R2016b)

This is an issue where the variables set within the mjs_def file are not used to initialize the port ranges used by MATLAB Job Scheduler and the workers. These port ranges only exist in MATLAB Parallel Server versions R2015a to R2016b.

Please see the following external bug report for this issue:

https://www.mathworks.com/support/bugreports/1546740

To work around this issue please use the following flags for the terminal interface method of starting an MATLAB Job Scheduler cluster. These flags need to be added to the call for ./mjs with their desired values.

The appropriate flag for each variable:

JOBMANAGER_PEERSESSION_MIN_PORT =  -jobmanagerpeersessionminport
JOBMANAGER_PEERSESSION_MAX_PORT =  -jobmanagerpeersessionmaxport
WORKER_PARALLELPOOL_MIN_PORT    =  -workerparallelpoolminport
WORKER_PARALLELPOOL_MAX_PORT    =  -workerparallelpoolmaxport

For example, to start MJS with a custom port range of 14384-14848 for starting parallel pools on workers, type the following command into a command prompt (where <MATLABROOT> is the root of your MATLAB Parallel Server installation):

cd <MATLABROOT>/toolbox/parallel/bin 
mdce start -workerparallalpoolminport 14384 -workerparallelpoolmaxport 14848

These flags only need to be included in the start up of the mjs service on each node of the cluster. After this point you are able to return to using admin center to start the job manager and workers.

NOTE: Starting in R2019a the following name changes occurred: * MATLAB Distributed Computing Server was renamed to MATLAB Parallel Server * mdce_def was renamed to mjs_def * mdce binary was renamed to mjs

Best Answer

Related Solutions

MATLAB: How to configure MATLAB Parallel Server using a 3rd party scheduler to work within a firewall

MATLAB: Are the custom port ranges for an MATLAB Parallel Server cluster using the MATLAB Job Scheduler not being used (R2015a to R2016b)

Related Question