MATLAB: How to switch between Hadoop and MATLAB Job Scheduler

MATLAB Parallel Server

How do I switch between Hadoop and MATLAB Job Scheduler?

Best Answer

Currently, the only way to switch between Hadoop and any other scheduler is to reconfigure Hadoop every time you want to switch to it. To simplify the task, you can write a script and run this script every time you want to use Hadoop. Follow instructions here to setup Hadoop:

http://www.mathworks.com/help/mdce/configure-a-hadoop-cluster.html

When setting it up you can save these commands into a .m file, which you can any time within MATLAB.

NOTE: Starting in R2019a the following name changes occurred:
MATLAB Distributed Computing Server was renamed to MATLAB Parallel Server
mdce_def was renamed to mjs_def
mdce binary was renamed to mjs

Related Solutions

MATLAB: Can a worker registered with the MathWorks Job Manager be accessed from another scheduler (such as PBS)

A worker is a MATLAB session executing on a machine (node), and is associated with a job manager or with the scheduler that created it. Multiple schedulers can execute tasks on the same machine. For each scheduler accessing the machine, a different MATLAB worker session is started.

Each worker is dedicated to its job manager or scheduler, even if the same machine is being used by different schedulers. An idle worker that is associated with a job manager remains available (idle) to that job manager when another scheduler starts a worker on the same machine.

Note that a worker started by a third-party scheduler runs for only as long as it takes it to evaluate its task, then it is stopped. A worker that is registered with a job manager remains running and available between tasks.

MATLAB: Matlab and Hadoop integration

Without your whole code (apart1.m), it would be difficult to investigate why the error occurs, but as far as I guess, it might be missing to set mapreducer to use parallel.cluster.Hadoop.

As this document explains, at least the three lines are needed to run MapReduce on Hadoop.

% Specify HADOOP_HOME in MATLAB
setenv('HADOOP_HOME', '/path/to/hadoop/install') 
% This will run mapreduce on Hadoop
cluster = parallel.cluster.Hadoop;
% If you want to change properties of parallel.cluster.Hadoop,
% please see https://www.mathworks.com/help/distcomp/parallel.cluster.hadoop_function.html#bujzb46-2
% for example, if the installation path of MATLAB Distributed Computing Server on Hadoop clusters 
% is different from that of MATLAB Desktop on Hadoop node, you need to change ClusterMatlabRoot property.
% cluster.ClusterMatlabRoot = '/path/to/MDCS/install';
mr = mapreducer(cluster);

After this mapreducer setting, mapreduce will be run on Hadoop.

Best Answer

Related Solutions

MATLAB: Can a worker registered with the MathWorks Job Manager be accessed from another scheduler (such as PBS)

MATLAB: Matlab and Hadoop integration

Related Question