I am running MATLAB R2019a locally on Windows 10. I am using MapReduce on a "large" dataset (~10GB, where my system RAM is 16 GB). I am constrained to running the software locally (i.e. do not have the option to use Hadoop).
The map stage runs to 100%, after which the Reduce stage jumps from 0 to 100%, with the following error:
Error using mapreduce (line 124)
The input files must be MAT-files or they must be Sequence files that are outputs of mapreduce.
Invalid file: 'D:\wb\matlab\data_anal\results_1_06-Jul-2020_00-16-10_333.mat'
Error in tx_rx (line 5)
out_ds = mapreduce(start_ds, @wb_detector, @wb_reducer);
I also get a warning that I cannot trace:
Warning: Variable 'Value' was not saved. For variables larger than 2GB use MAT-file version 7.3 or later.
As I have not created a variable with this name.
The call to mapreduce is preceeded by:
start_ds = datastore("start_wb.csv");start_ds.SelectedVariableNames ={'Time', 'V_V_RX_'};out_ds = mapreduce(start_ds, @wb_detector, @wb_reducer);I noticed that the two files generated by the mapreduce function contain identical contents, which are the same as the data I see when I run preview(start_ds), i.e. the first 8 entries of each column.
I moved my files to D:/ drive, trying to see if aspects of my C:/ drive were causing the error (low drive space, running the code from a OneDrive foldler) but it did not resolve the issue. I have also tried reducing start_ds.ReadSize from 20000 to 1000 (thinking this may resolve the warning and thus the subsequent error), but this has not resolved the issue either.
Any tips would be greatly appreciated.
Best Answer