MATLAB: Matlab Parallel Server worker to worker communication port

MATLABMATLAB Parallel Server

Hi,
I have been setting up Matlab Parallel Server on GKE. I have opened ports according to this post: https://www.mathworks.com/matlabcentral/answers/94254-how-do-i-configure-matlab-parallel-server-using-the-matlab-job-scheduler-to-work-within-a-firewall.
However, when I tried to run validation test from my client node with 2 or more worker, I got error from worker node saying: 
Error using parallel.internal.getJavaFutureResult (line 33)
MatlabPoolPeerInstance{fLabIndex=1, fNumberOfLabs=4, 
fUuid=277db6ba-1f54-4ca9-8045-eb58a0f82d08} was unable 
to connect to [an ip].bc.googleusercontent.com/[my-worker-ip]:35721 most probably because 
[an ip].bc.googleusercontent.com/[my-worker-ip]:35721 refused the connection: Connection refused
I am confused and don't know how to resolve this issue. I opened all the ports required:
BASEPORT              to        BASEPORT+6+nJ+4*nW
BASEPORT+1000         to        BASEPORT+1000+2*nW   (only to other cluster machines)
But seems like it's trying to reach some random port > 30000. I believe my head-worker, client-worker, client-head communication are good, but worker-worker communication is blocked. Is there any specific port for communication from worker to worker? It seems like very random, and I get different ports from the error message all the time.
Thanks in advance!
Ziwei

Best Answer

I've found a solution towards this with the help of Matlab Support team. Thank you!
In my set up, I need to expose head and worker externally so that my client UI, outside of the VPC could connect with them. However, Matlab Parallel Server has an assumption that communication between machines within the cluster is not restricted, which implicates all ports need to be open for woker-worker communication.
The solution is to add 
export MDCE_OVERRIDE_INTERNAL_HOSTNAME="myWorkerHost"
in the mjs_def file, with the hostname in that file to be external IP. In this case, all worker-worker communication is no longer through external Load Balancer service, but locally. (In my set up, all worker workloads are within the same GKE cluster so they could talk internally)
Related Question