MATLAB: How to csvwrite an array in HDFS

csvwritehadoophdfsMATLABspark

Hi,

I have Matlab connected to a Hadoop cluster. I want to save an array as a csv file to HDFS. The only commend I found was:

location = 'hdfs://master/user/savecsvfile';
write(location, Array1 );

Then I check the HDFS respiratory to find a file "part-1-snapshot.seq" and the contents are different from what Array1 is.

Thanks,

Best Answer

write command is able to write only tall array as .seq file. So, if you want to save csv files to HDFS, you should first save csv files on locla then upload to HDFS using Hadoop commands. From MATLAB,

csvwrite('hoge.csv', Array1);
!hadoop fs -copyFromLocal ./hoge.csv /user/savecsvfile

Related Solutions

MATLAB: How to solve ‘ All the input files must be Sequence files. Invalid file: ” ‘

This error message occurred because MATLAB could not find the output files generated by the Hadoop Job. For now, this error message should be treated as equivalent to MATLAB erroring that it could not find the output of the Hadoop Job. To resolve this error, make sure the output folder is a location that can be accessed by both your local machine and the Hadoop cluster.

I see that outputFolder is in "/home/rjy/logs/hadooplog". For MATLAB, this points to the home folder on your machine and is likely not to be accessible by the Hadoop cluster. As an alternative, could you try:

outputFolder = 'hdfs://ubuntu:8020/home/rjy/out';

This location is guaranteed to be accesible by both your local machine and the Hadoop cluster.

MATLAB: Building Mex file with Fortan

Your main problem is that you need to create the output array PLHS(1) before you access it. Also your NLHS and NRHS types are wrong. Try this:

#include <fintrf.h>
C
C Gateway routine
C
    SUBROUTINE MEXFUNCTION(NLHS, PLHS, NRHS, PRHS)
    implicit none
C mexFunction arguments:
    MWPOINTER PLHS(*), PRHS(*)
    INTEGER NLHS, NRHS
    mwPointer, external :: mxGetPr, mxGetPi
    mwPointer, external :: mxCreateDoubleMatrix, mxCreateNumericArray
    mwSize, external :: mxGetM, mxGetN
    MWPOINTER YPP 
    mwSize M, N
    integer*4 ComplexFlag
    REAL *8 RYPP(1)
    RYPP(1) = 10
    M = 1
    N = 1
    ComplexFlag = 0
    PLHS(1) = mxCreateDoubleMatrix( M, N, ComplexFlag )
    YPP = MXGETPR(PLHS(1))
    CALL MXCOPYREAL8TOPTR(RYPP, YPP, M*N)
    RETURN
    END

Best Answer

Related Solutions

MATLAB: How to solve ‘ All the input files must be Sequence files. Invalid file: ” ‘

MATLAB: Building Mex file with Fortan

Related Question