MATLAB: Importing large .raw file

data importimportMATLABmatrix

I am attempting to import a large (>6 GB) .raw file containing time series data from neurons. The goal of my script is to store all the voltage/time traces from each electrode from each of the 96 wells of my culture plate into a matrix that I can work with later. I have worked out all of the bugs in the code and now have gotten it to the point where it is just taking an eggregious amount of time to finish loading the data. I'm wondering if it is possible to speed this up any how. Thanks!
%% Script to take raw voltage data and convert into spike data for a rasterplot
%RawV=AxisFile(‘Filename.raw’).RawVoltageData.LoadData('A1','13',[0 300]); % Loads first 5 minutes of raw voltage data from "Filename.raw", well A1, electrode 13 and stores it in the variable "RawV"
% The above command returns a 4-D cell array in which all cells are empty
% except for RawV{1,1,1,3] where {WR,WC,EC,ER}
%[t,v]=RawV{1,1,1,3}.GetTimeVoltageVector; % Retrieves voltage (v) and time (t) vectors from the above raw data set
%plot(t,v) % Plots voltage against time to visualize work so far
%%
Rows = {'A','B','C','D','E','F','G','H'}; % Array of rows
Cols = [1:1:12]; % Vector of 12 columns by 1
elecC = [1:3]; % electrode column 1 to 3 by 1
elecR = [1:3]; % electrode row 1 to 3 by 1
cntr = 1;
for y = Cols % Iterate over each element in Cols
    for x = Rows % Iterate over each element in Rows
        for z=elecC % Iterate over each element in elecC
            for q=elecR % Iterate over each element in elecR
                RC_str = cell2mat([x,num2str(y)]); % Combines elements from row and column variables to identify the well e.g. 'A1' from 'A' and 1 and stores as a non-string variable
                eRC_str = [num2str(z),num2str(q)]; % Same thing for electrode row and colume but store as a string
                if strcmp(eRC_str,'32')
                    continue
                end
                RawV=AxisFile('Maestro_(000).raw').RawVoltageData.LoadData(RC_str,eRC_str,[0 300]); % Function loads data from file 'Maestro_(000).raw': well RC_str, electrode eRC_str
                RowVal = find(strcmp(Rows,x)); % Converts 'A' to 1, 'B' to 2 and etc.
                [t,v]=RawV{RowVal,y,z,q}.GetTimeVoltageVector; % Function to extract voltage vs time data from RawV: RowVal, y, z, q specify well and electrode
                Vdata(:,cntr) = v;
                cntr = cntr + 1;
            end
        end
    end
end

Best Answer

You should avoid creating the reading object each time, and you should avoid re-doing work. And you should pre-allocate
I suspect there are more efficiencies, but it would be necessary to study the AxisFile class more carefully.
I am not clear as to why Column 3, Electrode Row 2 is being skipped? That would reduce from 12*3*3 = 96 to 93 columns, and makes it more difficult to keep track of which column holds which data.
%% Script to take raw voltage data and convert into spike data for a rasterplot
%RawV=AxisFile(‘Filename.raw’).RawVoltageData.LoadData('A1','13',[0 300]); % Loads first 5 minutes of raw voltage data from "Filename.raw", well A1, electrode 13 and stores it in the variable "RawV"
% The above command returns a 4-D cell array in which all cells are empty
% except for RawV{1,1,1,3] where {WR,WC,EC,ER}
%[t,v]=RawV{1,1,1,3}.GetTimeVoltageVector; % Retrieves voltage (v) and time (t) vectors from the above raw data set
%plot(t,v) % Plots voltage against time to visualize work so far
%%
Rows = {'A','B','C','D','E','F','G','H'}; % Array of rows
Cols = [1:1:12]; % Vector of 12 columns by 1
elecC = [1:3]; % electrode column 1 to 3 by 1
elecR = [1:3]; % electrode row 1 to 3 by 1
cntr = 1;
RawVoltageData = AxisFile('Maestro_(000).raw').RawVoltageData;
for y = Cols % Iterate over each element in Cols
    ystr = sprintf('%d', y);
    for RowVal = Rows % Iterate over each element in Rows
        x = Rows{RowVal};
        RC_str = [x, ystr]; % Combines elements from row and column variables to identify the well e.g. 'A1' from 'A' and 1 and stores as a non-string variable
        for z=elecC % Iterate over each element in elecC
            zstr = sprintf('%d', z);
            for q=elecR % Iterate over each element in elecR
                qstr = sprintf('%d', q);
                eRC_str = [zstr, qstr]; % Same thing for electrode row and colume but store as a string
                if strcmp(eRC_str,'32')
                    continue
                end
                RawV = RawVoltageData.LoadData(RC_str,eRC_str,[0 300]); % Function loads data from file 'Maestro_(000).raw': well RC_str, electrode eRC_str
                [t,v] = RawV{RowVal,y,z,q}.GetTimeVoltageVector; % Function to extract voltage vs time data from RawV: RowVal, y, z, q specify well and electrode
                if cntr == 1
                    Vdata = zeros(length(v),96);  %pre-allocate!
                end
                Vdata(:,cntr) = v;
                cntr = cntr + 1;
            end
        end
    end
end
Vdata = Vdata(:,1:cntr - 1);  %we allocated for maximum size, but some items were skipped so trim down
Related Question