MATLAB: “Error during serialization” with API for Java and large matrix

javaMATLABmatlabexceptionexception

Hi,
I'm using the MATLAB API for Java to factor large matrices. Within my Java code I start the MatlabEngine:
MatlabEngine eng = MatlabEngine.startMatlab();
Then I call the svd function and pass it a large 2D array of double values as input:
// 'a' is a large double[][]
Object[] results = eng.feval(3, "svd", a);
double[][] S = (double[][])results[1];
double[][] V = (double[][])results[2];
This works very nicely for sizes of the array 'a' up to around 23000×23000. However for larger matrices, the Matlab call throws an exception:
com.mathworks.engine.MatlabExecutionException: svd
	at com.mathworks.engine.FutureResult.get(FutureResult.java:64)
	at com.mathworks.engine.MatlabEngine.feval(MatlabEngine.java:464)
	at sandbox.TestSvdEvd.main(TestSvdEvd.java:184)
Caused by: com.mathworks.mvm.exec.MvmException: Error during serialization
	... 3 more
com.mathworks.mvm.exec.MvmException: Error during serialization
Matlab has no problem processing the same matrix in a standalone Matlab process. So I seem to be hitting some memory limit with the MATLAB API for Java, however there's plenty of RAM available to the JVM.
So I guess my question(s) is: am I doing something wrong with the Java API? Is there some way I can increase the memory available to the Matlab engine to avoid this exception?

Best Answer

After some reverse-engineering of the MatlabEngine I figured out the problem - it was caused by the MatlabEngine flattening my array and serializing it into a 1D array of shorts for transfer to the Matlab process. For a large input array this can cause the serialized array to be longer than the array size limit imposed by Java (around 2147483645 elements, or Integer.MAX_VALUE - 2). Arrays containing more than ~536870906 doubles or ~1073741809 floats will overflow that. There's no simple fix for this but I can work around it by transferring my data in chunks. In this particular application, where I'm handling large 2D arrays, this can be done by transferring one row at a time between Matlab and Java. Here's some Java code that does this:
// 'eng' is the MatlabEngine
// 'a' is a large nxn array of doubles, with n > 23170, which we want to factorise
// Simply doing:
//     Object[] results = eng.feval(3, "svd", a);
// fails with an 'com.mathworks.mvm.exec.MvmException: Error during serialization'
// Transfer one row at a time to the Matlab workspace
for(int row = 0; row < n; row++) {
    eng.putVariable("row_"+row, a[row]);
}
				
// Compile the individual rows into one matrix in the Matlab workspace,
// cleaning up as we go to preserve memory
StringBuilder build_a = new StringBuilder();
build_a.append("a = [ row_0 ");
StringBuilder clear_rows = new StringBuilder();
clear_rows.append("clear row_0 ");
for(int row = 1; row < n; row++) {
    build_a.append("; row_"+row);
    clear_rows.append(" row_"+row);
}
build_a.append(" ];");
clear_rows.append(";");
eng.eval(build_a.toString());
eng.eval(clear_rows.toString());
				
// Compute the matrix factorisation
eng.eval("[U, S, V] = svd(a);");
eng.eval("clear a;");
				
// Retrieve the matrix V one row at a time
double[][] V = new double[n][];
for(int row = 0; row < n; row++) {
    // row+1 due to Matlab 1-based indexing
    eng.eval("row = V(" + (row+1) + ",:);");
    V[row] = (double[])eng.getVariable("row");
    eng.eval("clear row;");
}
eng.eval("clear V;");
				
// Retrieve the matrix S one row at a time
double[][] S = new double[n][];
for(int row = 0; row < n; row++) {
    // row+1 due to Matlab 1-based indexing
    eng.eval("row = S(" + (row+1) + ",:);");   
    S[row] = (double[])eng.getVariable("row");
    eng.eval("clear row;");
}
eng.eval("clear S;");
Related Question