MATLAB: What tool boxes do I need to integrate with Hadoop.

hadoopMATLABMATLAB Compilermatlab distributed computing serverMATLAB Parallel ServerParallel Computing Toolbox

Hi, I am currently looking into integrating Matlab with a Hadoop Cluster. I have looked all over the website but it isn't clear which tool boxes are actually necessary to do this, I know that Matlab Compiler, Parallel Computing Tool Box, and the Matlab Distributed Computing Server(MDCS), are related, but I have found the website very unclear, and if all, none, or some of these are actually necessary. Thanks

Best Answer

Hi Adam,

To integrate MATLAB with a cluster (whether a Hadoop cluster or some other generic cluster), you need MATLAB Distributed Computing Server (MDCS).

Then to send mapreduce jobs to that Hadoop cluster from MATLAB, you'll need at minimum Parallel Computing Toolbox.

Matlab Compiler is only required if you wish to package MapReduce based algorithms for deploying to production Hadoop systems.

Required:

MATLAB, MDCS, Parallel Computing Toolbox

Optional:

Matlab Compiler

Related Solutions

MATLAB: How to solve ‘ All the input files must be Sequence files. Invalid file: ” ‘

This error message occurred because MATLAB could not find the output files generated by the Hadoop Job. For now, this error message should be treated as equivalent to MATLAB erroring that it could not find the output of the Hadoop Job. To resolve this error, make sure the output folder is a location that can be accessed by both your local machine and the Hadoop cluster.

I see that outputFolder is in "/home/rjy/logs/hadooplog". For MATLAB, this points to the home folder on your machine and is likely not to be accessible by the Hadoop cluster. As an alternative, could you try:

outputFolder = 'hdfs://ubuntu:8020/home/rjy/out';

This location is guaranteed to be accesible by both your local machine and the Hadoop cluster.

MATLAB: Matlab and Hadoop integration

Without your whole code (apart1.m), it would be difficult to investigate why the error occurs, but as far as I guess, it might be missing to set mapreducer to use parallel.cluster.Hadoop.

As this document explains, at least the three lines are needed to run MapReduce on Hadoop.

% Specify HADOOP_HOME in MATLAB
setenv('HADOOP_HOME', '/path/to/hadoop/install') 
% This will run mapreduce on Hadoop
cluster = parallel.cluster.Hadoop;
% If you want to change properties of parallel.cluster.Hadoop,
% please see https://www.mathworks.com/help/distcomp/parallel.cluster.hadoop_function.html#bujzb46-2
% for example, if the installation path of MATLAB Distributed Computing Server on Hadoop clusters 
% is different from that of MATLAB Desktop on Hadoop node, you need to change ClusterMatlabRoot property.
% cluster.ClusterMatlabRoot = '/path/to/MDCS/install';
mr = mapreducer(cluster);

After this mapreducer setting, mapreduce will be run on Hadoop.

Best Answer

Related Solutions

MATLAB: How to solve ‘ All the input files must be Sequence files. Invalid file: ” ‘

MATLAB: Matlab and Hadoop integration

Related Question