MATLAB: Do I get an error when trying to deploy applications to Cloudera Spark using the MATLAB API for Spark

clouderagentraversableonceMATLAB Compilerscalaspark

I am following trying the example described on this documentation page:

https://www.mathworks.com/help/compiler/spark/example-on-deploying-applications-to-cloudera-spark-using-the-matlab-api-for-spark.html

This workflow is further explained in this MATLAB answers post:

https://www.mathworks.com/matlabcentral/answers/305729-how-do-i-create-a-deployed-matlab-applications-to-run-against-cloudera-spark

When I try executing the jobs with "yarn-client", I am seeing many warning and error messages in the yarn log. Specifically, the yarn-log contains the following message:

java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class

I am currently using:

– Cloudera CDH 5.14.1

– Spark 2.2.0 v Cloudera2-1

– Scala version 2.11.8 (Java HotSpot™ 64-Bit Server VM, Java 1.8.0_161)

The exact commands that I used were the following:

1) For compilation:

>> mcc -C -W 'Spark:flightsByCarrierDemoApp' flightsByCarrierDemo.m

2) For execution:

>>  ./run_flightsByCarrierDemoApp.sh /pkgs/cdh/matlab/runtime/v92 yarn-client hdfs://nameservice1/user/ed003898/airlinesmall.csv

Best Answer

First, make sure that you set the property ''spark.executorEnv.MCR_CACHE_ROOT'' to something like '/tmp/matlabapp'.

Next, when using Spark version 2, the compilation command should be:

>> mcc -C -W 'Spark:flightsByCarrierDemoApp,2' flightsByCarrierDemo.m

Also, it is recommend that the code explicitly sets the SparkVersion, as shown below

>> conf = SparkConf( ...
     'AppName','flightsByCarrierDemo', ...
     'Master',master, ...
     'SparkProperties',sparkProperties, ...
     'SparkVersion', '2' ...
     );

Related Solutions

MATLAB: How to create a deployed MATLAB® applications to run against Cloudera Spark™

In R2016b, MATLAB Compiler supports running MATLAB applications as standalone executables against a Spark enabled cluster. The ability to deploy MATLAB applications against a Cloudera Spark distribution requires an alternate workflow that is undocumented in the release documentation.

To deploy MATLAB applications against a Cloudera distribution of Spark requires a new wrapper type that can be generated using the mcc command. Using this new wrapper type generates a jar file as well as a shell script which calls spark_submit. The spark-submit script in Spark’s bin directory is used to launch applications on a cluster. It supports both yarn-client mode and yarn-cluster mode.

MATLAB applications that use tall arrays or the MATLAB API for Spark can be deployed using this workflow.

Example 1: Deploy Tall Arrays to a Cloudera Spark Enabled Hadoop Cluster

This example shows you how to deploy a MATLAB application that uses tall arrays to a Cloudera Spark enabled Hadoop cluster. The application meanArrivalDemo.m computes the mean arrival delay from airline data. The inputs to the application are:

master—URL to the Spark cluster.

inputFile—the file containing the input data.

outputFile—the file containing the results of the computation.

Prerequisites:

Install the MATLAB Runtime in the default location on the desktop. This example uses as the default location for the MATLAB Runtime.
Install the MATLAB Runtime on every worker node.
Copy the airlinesmall.csv from folder of your MATLAB install area into Hadoop Distributed File System (HDFS™) folder /datasets/airlinemod.

If you don't have the MATLAB Runtime, you can download it from the website at: https://www.mathworks.com/products/compiler/matlab-runtime.html.

Procedure:

1. At the MATLAB command prompt, use the mcc command to generate a jar file and shell script for the MATLAB application meanArrivalDemo.m

>> mcc -vCW 'Spark:meanArrivalDemoApp' meanArrivalDemo.m

or, if using Spark version 2:

>> mcc -vCW 'Spark:meanArrivalDemoApp, 2' meanArrivalDemo.m

This creates a jar file named meanArrivalDempApp.jar and a shell script named run_meanArrivalDemoApp.sh.

Note: In order to use the shell script, you need the environment variables HADOOP_PREIX, HADOOP_CONF_DIR and SPARK_HOME to be set up correctly.

The general syntax to execute the shell script is:

./run_meanArrivalDemoApp.sh <runtime install root> [Spark arguments] [Application arguments]

a. yarn-client mode

Run the following command from a Linux terminal:

$ ./run_meanArrivalDemoApp.sh \
   /usr/local/MATLAB/MATLAB_Runtime/v91 \
    yarn-client \
    hdfs://hadoop01glnxa64:54310/datasets/airlinemod/airlinesmall.csv \
    hdfs://hadoop01glnxa64:54310/user/someuser/meanArrivalResult

To examine the result, enter the following from the MATLAB command prompt:

>> ds = … datastore('hdfs://hadoop01glnxa64:54310/user/someuser/meanArrivalResult/*');
>> readall(ds)

b. yarn-cluster mode

Run the following command from a Linux terminal:

$ ./run_meanArrivalDemoApp.sh \
   /usr/local/MATLAB/MATLAB_Runtime/v91 \
   --deploy-mode cluster --master yarn yarn-cluster \
   hdfs://hadoop01glnxa64:54310/datasets/airlinemod/airlinesmall.csv \ 
   hdfs://hadoop01glnxa64:54310/user/someuser/meanArrivalResult

In yarn-cluster mode, since the driver is running on some worker node in the cluster, any standard output from the MATLAB function will not be displayed on your desktop. In addition, files can end up being saved anywhere. In order to prevent such behavior, this example uses the write function to explicitly save the results to a particular location in HDFS.

Example 2: Deploy Applications Using the MATLAB API for Spark

This example shows you how to deploy a MATLAB application developed using the MATLAB API for Spark against a Cloudera Spark enabled Hadoop cluster. The application flightsByCarrierDemo.m computes the number of airline carrier types from airline data. The inputs to the application are:

master—URL to the Spark cluster.

inputFile—the file containing the input data.

Prerequisites:

Install the MATLAB Runtime in the default location on the desktop. This example uses as the default location for the MATLAB Runtime.
Install the MATLAB Runtime on every worker node.
Copy the airlinesmall.csv from folder of your MATLAB install area into Hadoop Distributed File System (HDFS™) folder /datasets/airlinemod.

If you don't have the MATLAB Runtime, you can download it from the website at: http://www.mathworks.com/products/compiler/mcr.

Procedure:

1. At the MATLAB command prompt, use the mcc command to generate a jar file and shell script for the MATLAB application flightsByCarrierDemo.m

>> mcc -C -W 'Spark:flightsByCarrierDemoApp' flightsByCarrierDemo.m

This creates a jar file named flightsByCarrierDemoApp.jar and a shell script named run_flightsByCarrierDemoApp.sh.

2. You can execute the shell script in yarn-client mode or yarn-cluster mode. In yarn-client mode, the driver runs on the desktop. In yarn-cluster mode, the driver runs in the Application Master process in the cluster. The results of the computation in both cases are saved to a text file on HDFS by calling the saveAsTextFile method on the RDD.

a. yarn-client mode

Run the following command from a Linux terminal:

$ ./run_flightsByCarrierDemoApp.sh \ 
   /usr/local/MATLAB/MATLAB_Runtime/v91 \
   yarn-client \
   hdfs://hadoop01glnxa64:54310/datasets/airlinemod/airlinesmall.csv

To examine the results, enter the following from a Linux terminal.

   $ hadoop fs -cat flightsByCarrierResults/*

b. yarn-cluster mode

Run the following command from a Linux terminal:

$ ./run_flightsByCarrierDemoApp.sh \
   /usr/local/MATLAB/MATLAB_Runtime/v91 \
   --deploy-mode cluster --master yarn yarn-cluster \            
   hdfs://hadoop01glnxa64:54310/datasets/airlinemod/airlinesmall.csv

MATLAB: How to print text to the MATLAB command window from a FORTRAN MEX file

Use mexPrintf to print to the MATLAB command window when using MEX files. Two examples are shown below.

Example 1: Print "API says hello!" to the MATLAB command window

#include "fintrf.h" 
      subroutine mexFunction(nlhs, plhs, nrhs, prhs) 
C     Specify alert to be printed 
      character (len=*), PARAMETER :: txt = "API says hello!"
C     Call the API     
      call mexPrintf(txt)
      end

Example 2: Print the number of input arguments to the MATLAB command window:

#include "fintrf.h" 
      subroutine mexFunction(nlhs, plhs, nrhs, prhs) 
        character (len=1) :: c
        character (len=50) :: txt
C     Specify alert to be printed
      write(txt,'(A)') 'Number of inputs (mexPrintf): ' 
C     Convert nrhs to char
      write(c,'(i1)') nrhs
C     Call the API
      call mexPrintf(txt)
      call mexPrintf(c)
      end

There are many FORTRAN MEX function source code examples available at:

https://www.mathworks.com/help/matlab/matlab_external/table-of-mex-file-source-code-files.html

Look in the table for examples with the .F file extension.

Best Answer

Related Solutions

MATLAB: How to create a deployed MATLAB® applications to run against Cloudera Spark™

MATLAB: How to print text to the MATLAB command window from a FORTRAN MEX file

Related Question