MATLAB: How to read just a part of a binary file with a predefined end position or a predefined amount of Bytes

binary filesfreadMATLAB

Hi. I have searched a lot to find the answer, but was not successful.

I want to get data records ({'uint16' 'uint16' 'uint16' 'uint8' 'uint8'} = 8 Bytes) out of a binary file.

The files have millions of records with 1 min time steps and a given start date.

Up to now, I was able to define the start position by skipping the wanted time duration (1 record of 8 Bytes = 1 min) with fseek.

My problem is, that I can not find a solution how to define the end position or the amount of records for fread.

One solution would be to use a Loop in which the record length is added to fseek each run and the rest of the file is skipped after every record. But this is grossly inefficient and likely would need even more time than reading the whole file and picking the wanted part out of the resulting matrix, I guess.

I hope you understand what I want to ask…

I need something like fread(fileID,start_position,end_position or number of records).

Thanks in advance.

Best Answer

I'm not entirely sure I completely understand, maybe that's what you want:

recordstart = ??? %some integer value. Index of first desired record
numrecords = ???  %how many records to get
filepath = ???    %path of the file
recordtypes = {'uint16', 'uint16', 'uint16', 'uint8', 'uint8'};
recordsizes = [2, 2, 2, 1, 1];  %size of each type in bytes. Must match recordtypes
fid = fopen(filepath, 'r')
fseek(fid, (recorstart - 1) * sum(recordsizes), 'bof');  
data = fread(fid, [sum(recordsizes), numrecords], '*uint8');  %read numrecords as uint8
data = mat2cell(data, recordsizes, numrecords);
data = cellfun(@(bytes, data) typecast(bytes(:), data), data', recordtypes, 'UniformOutput', false);

Related Solutions

MATLAB: Reading sequence of binary packets consisting of multiple datatypes from binary file

Use fread for binary data. Following your description, probably:

fid = fopen(filepath, 'r', 'l', 'ISO-8859-1');  %l for Little-endian which I assume is the LE in the description


%reading one record:
signal.name = fread(fid, [1 18], '*char');  %read 18 characters
signal.value = fread(fid, 1, 'int16');  %read 1 signed 16 bit integer. stored as double
signal.date = fread(fid, 1, 'int32');
signal.milliseconds = fread(fid, 1, 'int16');
signal.status = fread(fid, 1, '*uint8');  %read as unsigned 8 bit, keep as uint8
signal.type = fread(fid, 1, 'uint8');

For reading multiple records you can wrap the above in a loop. It won't be very fast though. Another option is to use the skip argument of fread to all the names in one go, rewind the file, read all the values, rewind the file, etc..:

fid = fopen(filepath, 'r', 'l', 'ISO-8859-1');  %l for Little-endian which I assume is the LE in the description
%reading all records:
signal.names = fread(fid, [18 Inf], '*char', 28)'; %skip 28 bytes between each name
fseek(fid, 18, 'bof');  %rewind back to the first value (18 bytes after the start)
signal.values = fread(fid, [1 Inf], 'int16', 28)';
fseek(fid, 20, 'bof');  %rewind back to the first date (20 bytes after the start)
signal.date = fread(fid, [1 Inf], 'int32', 28);
%etc.

Possibly, the fastest option may be to read the whole file in one go and perform the conversion afterward:

recordfields = {'name', 'value', 'date', 'milliseconds', 'status', 'type'};
recordtypes = {'char', 'int16', 'int32', 'int16', 'uint8', 'uint8'};
recordsizes = {18, 2, 4, 2, 1, 1};; %size of each type in bytes
fid = fopen(filepath, 'r', 'l', 'ISO-8859-1');  %l for Little-endian which I assume is the LE in the description
data = fread(fid, [sum(recordsizes), Inf], '*uint8')';  %read the whole lot as uint8, stored as uint8. transpose so that rows are records
data = mat2cell(data, size(data, 1), recordsizes); %split columns into each field
data = cellfun(@(col, type) typecast(col, type), data, recordtypes, 'UniformOutput', false);
record = cell2struct(data, recordfields, 2);

MATLAB: How to loop fopen/fread/fseek to analyze large binary files in shorter segments

file=('C:\Users\Admin\Documents\MATLAB\EOD examples\Examples 16ch\copy.mat');
fid=fopen(file,'r');
n_ch=16;                  % number channels (exclusive of timestamp)
Fs = 20000;               % sampling frequency
index = 0;
while ~feof(fid)
  index=index+1;          % minute number
  currData=fread(fid,[n_ch+1,Fs*60],'double').';  % one minute of data in 2D array
  % NB: must orient [time+N Channels X TimeSteps] and then transpose to
  % order data properly by column-major order internally.  
  disp(sprintf('Processing minute: %d',index))
  % process one minute worth of data here
  ...
end
fid=fclose(fid);

The key is to just read 20k*60 by 1+16 chnls each pass through the loop and tell fread to return in that orientation. Time vector will be currData(:,1), the channels currData(:,2:end)

Might look at memmapfile object as a higher-level abstraction if intending to read other than sequentially.

ADDENDUM/ERRATUM

Corrected initial above [sizeA} argument order to properly reflect data are in channel-order by time step in the file so must read the partial file indicating the nCH+1 dimension first for the desired number of time steps, then transpose to have as column-wise in internal data storage order.

If were reading the whole file, Matlab would do that automagically but a portion of the file must be treated as a separate grouping of the time plus data channels.

Best Answer

Related Solutions

MATLAB: Reading sequence of binary packets consisting of multiple datatypes from binary file

MATLAB: How to loop fopen/fread/fseek to analyze large binary files in shorter segments

Related Question