MATLAB: Import textfile with both numbers and characters into array while skipping part of the file’s content

fscanftext file

Dear all,

I want to import the following textfile (ongoing) as an array:

The result should be an array including all values except the content between 5002 and 6002 without including both numbers.

Result:

280

1465

1471

[…]

Thanks, Sarah

Best Answer

With the above analysis of the file structure and ignoring the remainder of the file as there are also no delimiters other than blanks to count to get a number of fields,

fid=fopen('expl_file.txt','r');   % open file
s=string(split(fgetl(fid)));      % read the first record & convert to words
fid=fclose(fid);                  % done with the file
i1=find(s=="5002");               % find the offending 5002
i2=find(s=="6002");               % and its partner...
s(i1(1):i2(2))=[];                % eliminate them and everything in between
v=cellfun(@str2double,s);         % convert remainder to numeric

So, what did we get?

>> whos v
  Name           Size             Bytes  Class     Attributes
  v         119996x1             959968  double              
>> v(1:20)
ans =
           1
           2
          40
         280
        1242
        1236
        1231
        1226
        1221
        1215
        1210
        1205
        1199
        1194
        1189
        1184
        1178
        1173
        1168
        1163
>>

I would imagine if one knew the definitions of the remainder of the encodings one could parse the file to actually compute the number of elements in that first record...but that wasn't revealed so have to just parse what one finds.

BTW, the data in the attached file are not those of the original image--I double-checked and the above data are those in the file at the equivalent locations.

ADDENDUM One guesses this is pretty brittle as there may well be no guarantee that either the magic numbers of 5002 and 6002 will be the pertinent values now and forever nor that there couldn't be such a value in the actual data stream itself.

We don't have enough info to know for sure what would be "the better way"

Related Solutions

MATLAB: Modify then write data in the given format

The documentation of textscan doesn't cover fixed-width very well. However, textscan has "undocumented"/hidden capabilities.

Approach

Read the first three columns to one string, since they shall only be copied to the output file.
Read the following six columns to a double array.
Add 1 to the prescribed elements of the array
Use the same format string to write the data (don't forget new-line)

Run example code (I use R2013b)

    fixed_width_format(13)

where

    function    fixed_width_format( N )
        fid = fopen( 'fixed_width_format.txt' );
        format_spec = '%20s%8.3f%8.3f%8.3f%8.4f%8.4f%8.4f';
        cac = textscan( fid, format_spec, N     ...
                    ,   'Whitespace'    , ''    ...
                    ,   'Delimiter'     , ''    ...
                    ,   'CollectOutput' , true  );
        fclose( fid );
        RowHead = cac{1};
        Data    = cac{2};
        % add 1 to the numbers in the columns 4 and 5 and rows 3 to 8. 
        Data( 3:8, 4:5 ) = Data( 3:8, 4:5 ) + 1;
        fid = fopen( 'fixed_width_format_out.txt', 'w' );
        for rr = 1 : N
           fprintf( fid, [format_spec,'\n'], RowHead{rr}, Data(rr,:) );
        end
        fclose( fid );   
    end

and where fixed_width_format.txt contains

        1SOL     OW    1   4.309   5.254   4.135 -0.2790  0.3440  0.2064
        1SOL    HW1    2   4.314   5.169   4.082 -1.5406  0.3918 -0.0293
        1SOL    HW2    3   4.388   5.312   4.114 -1.3375  0.9272 -2.6151
        2SOL     OW    4   1.743   1.687   2.366  0.2136  0.2777  0.3181
        2SOL    HW1    5   1.818   1.750   2.387  0.3115  0.1542  0.3431
     4502OCTA     H13545   2.108   5.326   1.045 -1.2169  0.4890 -2.6144
     4502OCTA     H13546   2.068   5.492   1.036  0.7609  0.6650  0.8612
     4502OCTA     H13547   2.285   5.388   1.207  3.0144  2.5562  1.0920
     4502OCTA     H13548   2.121   5.425   1.265 -1.2460 -1.3635  1.4829
     4502OCTA    Oc13549   2.131   5.677   1.238 -0.0183 -0.0221 -1.0402
     4502OCTA    Oh13550   2.353   5.635   1.208 -0.6036  0.2241 -0.8140
     4502OCTA     H13551   2.383   5.198   0.399  0.4893  0.7154 -0.9915
     4502OCTA    Ho13552   2.413   5.565   1.189 -0.4685 -0.0421 -2.1107
    ---  0---|--- 10---|--- 20---|--- 30---|--- 40---|--- 50---|--- 60---
    123456789|123456789|123456789|123456789|123456789|123456789|123456789
    '%20s%8.3f%8.3f%8.3f%8.4f%8.4f%8.4f%8.4f'

and where fixed_width_format_out.txt contains

        1SOL     OW    1   4.309   5.254   4.135 -0.2790  0.3440  0.2064
        1SOL    HW1    2   4.314   5.169   4.082 -1.5406  0.3918 -0.0293
        1SOL    HW2    3   4.388   5.312   4.114 -0.3375  1.9272 -2.6151
        2SOL     OW    4   1.743   1.687   2.366  1.2136  1.2777  0.3181
        2SOL    HW1    5   1.818   1.750   2.387  1.3115  1.1542  0.3431
     4502OCTA     H13545   2.108   5.326   1.045 -0.2169  1.4890 -2.6144
     4502OCTA     H13546   2.068   5.492   1.036  1.7609  1.6650  0.8612
     4502OCTA     H13547   2.285   5.388   1.207  4.0144  3.5562  1.0920
     4502OCTA     H13548   2.121   5.425   1.265 -1.2460 -1.3635  1.4829
     4502OCTA    Oc13549   2.131   5.677   1.238 -0.0183 -0.0221 -1.0402
     4502OCTA    Oh13550   2.353   5.635   1.208 -0.6036  0.2241 -0.8140
     4502OCTA     H13551   2.383   5.198   0.399  0.4893  0.7154 -0.9915
     4502OCTA    Ho13552   2.413   5.565   1.189 -0.4685 -0.0421 -2.1107

&nbsp

Finally, does this example rely on undocumented features of textscan?

&nbsp

Addendum triggered by comment

    >> data = fixed_width_format(7);
    >> data(1:5,:)
    ans =
        4.3090    5.2540    4.1350   -0.2790    0.3440    0.2064
        4.3140    5.1690    4.0820   -1.5406    0.3918   -0.0293
        4.3880    5.3120    4.1140   -0.3375    1.9272   -2.6151
        1.7430    1.6870    2.3660    1.2136    1.2777    0.3181
        1.8180    1.7500    2.3870    1.3115    1.1542    0.3431
    >> data(6:7,:)
    ans =
               0         111      111111     1222222     2222333     3333333
          123456      789012      345678     9012345     6789012     3456789
    >>

where

    function    data = fixed_width_format(N)
        fid = fopen( 'fixed_width_format_dpb.txt' );
        format_spec = '%6f%6f%6f%7f%7f%7f';
        cac = textscan( fid, format_spec, N     ...
                    ,   'Whitespace'    , ''    ...
                    ,   'Delimiter'     , ''    ...
                    ,   'CollectOutput' , true  );
        fclose( fid );
        data = cac{1};
    end

and where fixed_width_format_dpb.txt contains

     4.309 5.254 4.135-0.2790 0.3440 0.2064
     4.314 5.169 4.082-1.5406 0.3918-0.0293
     4.388 5.312 4.114-0.3375 1.9272-2.6151
     1.743 1.687 2.366 1.2136 1.2777 0.3181
     1.818 1.750 2.387 1.3115 1.1542 0.3431
    000000000111111111122222222223333333333
    123456789012345678901234567890123456789

MATLAB: Time converted to a str

Again, pasting a (partial even) row of text with the actual field of interest would relieve confusion regarding the form, but presuming the 'sss' is actually microseconds (since the .FFF includes down to the msec resolution) and is in the file, then I'd do something like

fmt=[repmat('%*s',1,'G'-'A') '%3f %2f:%2f:%f %*[^\n]'];
d=cell2mat(textscan(fid,fmt,'delimiter',','headerlines',1,'collectoutput',1));

Now you'll have have an array Nx4 that are the [day,hr,min,sec.fff] values and you can pass those directly to datevec with the leading columns yr,mo as zero or some arbitrary start date (it'll all get subtracted out in the end so is immaterial as to absolute value and the time fields are all stored in the fractional portion of the datenum value. Keeping the day will eliminate issues regarding clock rollover if any data were to have been collected over the bewitching midnight hour.

Then, simply looking for

diff(datenum([YR0 1 d])>Setpoint)

will give you the desired answer. The above should be pretty quick.

Again, all above is air-code w/o an example subset of data to 'spearmint with..."salt to suit" :)

ADDENDUM

OK, with the sample data file to check on formatting, the rest is pretty much as outlined. You don't have any calendar data other than DOY and Matlab datenum needs a reference year so just pick something arbitrary as you're only looking for differences so it all gets subtracted out again, anyway.

dn=datenum(2000,0,dat(:,1),dat(:,2),dat(:,3),dat(:,4)+dat(:,5)/1E6);
dt=diff(dn)*86400*1000;

Above takes the date information you have and creates a datenum sequence. As noted before, this will have the benefit of handling clock rollover, etc., automagically. Then the "delta-t" dt is the sample time differential in msec which you can test. For a sample subset of the file supplied, this looks like--

>> dt(10:40)
ans =
  0.0905
  0.4526
  0.7343
  0.6940
  0.6840
  0.6940
  0.6940
  0.6940
 10.2494
  0.3722
  0.3722
  0.6940
  0.5532
  0.6840
  0.1308
  0.3017
  0.6940
  0.5834
  0.2112
  0.1308
  0.4124
 14.8058
  0.3722
  0.3722
  0.6940
  0.5532
  0.6840
  0.1408
  0.2917
  0.1509
  0.3923
>>

Seems like the acquisition must be on a time-sharing system where there's a periodic latency, maybe???

Anyway, your end object is now simply to find the locations that don't satisfy your criterion. There are several choices for how to report this; you can use a logical vector of locations that satisfy, find locations that don't, report if any don't or all do, etc., etc., etc., ...

>> isOK=all(dt<90)
isOK =
   1
>> sum(dt>10)
ans =
   4
>> find(dt>90)
ans =
 Empty matrix: 0-by-1
>> find(dt>10)
ans =
  18
  31
  53
  61
>>

Above uses a couple different fixed setpoints; you'll make that a variable, of course. I used an example of 10 just to illustrate the case of a few failing. The logical-addressing logic vector solution returns a full-length logical array of T/F, of course. Which turns out easiest to use is all dependent upon end objectives.

Best Answer

Related Solutions

MATLAB: Modify then write data in the given format

MATLAB: Time converted to a str

Related Question