MATLAB: How to set LD_PRELOAD for a Hadoop cluster

glibchadoopmapreduceMATLAB Compilerrhelshimspark

I am testing two scripts, one that uses Spark and one that uses MapReduce, on a Hadoop cluster running Red Hat Enterprise Linux 7.6. I am receiving the following error:

Error: failed /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/amd64/server/libjvm.so, ...
because /apps/matlab/R2020b/sys/os/glnxa64/libstdc++.so.6: undefined symbol: __cxa_thread_atexit_impl

This seems to be because the nodes have GLIBC 2.17 installed and this symbol starts being defined in GLIBC 2.18. I understand that there is a shim library for GLIBC 2.17 to work with MATLAB R2020b, but how can I force the cluster to find and load it?

Best Answer

We can preload the library in the MATLAB scripts through cluster properties without modifying the system environment variables. The two scripts require slightly different code, because they are built on different technologies. In both of these, $MATLAB_ROOT needs to be replaced with the full path to the MATLAB installation on the worker.

For "mapreduce", modify your code as below:

cluster = parallel.cluster.Hadoop(..)
...
cluster.HadoopProperties('mapred.child.env') = 'LD_PRELOAD=$MATLAB_ROOT/bin/glnxa64/glibc-2.17_shim.so'
...
mapreducer(cluster)

For the Spark-based script, modify the code as below:

cluster = parallel.cluster.Hadoop(..)
...
cluster.SparkProperties('spark.executorEnv.LD_PRELOAD') = '$MATLAB_ROOT/bin/glnxa64/glibc-2.17_shim.so'
...
mapreducer(cluster)

Related Solutions

MATLAB: Matlab and Hadoop integration

Without your whole code (apart1.m), it would be difficult to investigate why the error occurs, but as far as I guess, it might be missing to set mapreducer to use parallel.cluster.Hadoop.

As this document explains, at least the three lines are needed to run MapReduce on Hadoop.

% Specify HADOOP_HOME in MATLAB
setenv('HADOOP_HOME', '/path/to/hadoop/install') 
% This will run mapreduce on Hadoop
cluster = parallel.cluster.Hadoop;
% If you want to change properties of parallel.cluster.Hadoop,
% please see https://www.mathworks.com/help/distcomp/parallel.cluster.hadoop_function.html#bujzb46-2
% for example, if the installation path of MATLAB Distributed Computing Server on Hadoop clusters 
% is different from that of MATLAB Desktop on Hadoop node, you need to change ClusterMatlabRoot property.
% cluster.ClusterMatlabRoot = '/path/to/MDCS/install';
mr = mapreducer(cluster);

After this mapreducer setting, mapreduce will be run on Hadoop.

MATLAB: How to solve” The user home directory ‘/homes/’ for the cluster either did not exist or was not writable by the HADOOP user.”

The '/homes/' folder is related to the Hadoop property 'mapreduce.admin.user.home.dir'. This is used by Hadoop to set the HOME environment variable for worker processes.

The HOME environment variable is used by MATLAB for things such as preferences and caches. This is why as part of setting up Hadoop alongside MATLAB Distributed Computing Server, you need to ensure this property is set to a local folder that is writable by the cluster, for example "/home/hduser".

All required configuration steps are listed at:

http://www.mathworks.com/help/mdce/configure-a-hadoop-cluster.html

I hope this helps,

Rick

Best Answer

Related Solutions

MATLAB: Matlab and Hadoop integration

MATLAB: How to solve” The user home directory ‘/homes/’ for the cluster either did not exist or was not writable by the HADOOP user.”

Related Question