MATLAB: Loading a VERY inconsistent data

Question

Hi,

I have a ridiciously inconsistant data to load into Matlab and work with. The data has an inconsistent munber of columns, empty spaces and an inconsistent delimiter to cap it off.

I attach the data in a sample file and provide a screen (The blue arrows indicate a new row and are not a part of the file)

I have tried using textscan, unfortunatelly, without sucess to get a consistant result.

The most sucess I've had was using the following code

opts = detectImportOptions(filename);
opts = setvartype(opts,{'Var8','Var14','Var15'},'char');
M = readtable(filename,opts);

Unfortunatelly, columns still do not load consistantly.

I will be thankful for any help.

Answer 1

Supplying the FileType name-value pair as "fixedwidth" to detectImportOptions yields good results for your data.

Tweaking some of the VariableTypes helps, too.

filename = "vypis_pdtp3_200828_022446_example.txt";
opts = detectImportOptions(filename, "FileType", "fixedwidth");
opts.VariableTypes{2} = 'categorical';
opts.VariableTypes{3} = 'string';
opts.VariableTypes{8} = 'string';
opts.VariableTypes{12} = 'string';
opts.VariableTypes{15} = 'string';
opts.VariableTypes{16} = 'string';
opts.VariableTypes{17} = 'string';
opts.VariableTypes{18} = 'string';
t = readtable(filename, opts)

The code above yields the following MATLAB table:

t =
  75x18 table
       Var1       Var2       Var3          Var4          Var5          Var6          Var7         Var8       Var9    Var10    Var11      Var12      Var13    Var14      Var15        Var16                         Var17                                           Var18                    
    __________    ____    __________    __________    __________    __________    __________    _________    ____    _____    _____    _________    _____    _____    _________    _________    ____________________________________________    ____________________________________________
    6.3188e+05     s      "1dbb0ccc"    1.5986e+09    1.5613e+09    3.0249e+06    3.0249e+06    "3C56F4"     NaN      12        17     "168218"        0        0     "834D5C"     "FD8C72"     "8D_3C_56_F4_99_09_EB_94_98_08_2A_16_82_18_"    "00_00_00_01_00_00_00_00_00_00_01_00_00_00_"
    6.3188e+05     s      "1dbb0cf0"    1.5986e+09    1.5613e+09    3.2634e+06    3.2634e+06    <missing>    NaN      14         0     "90EF42"        0        0     "49C0F2"     "49C0F2"     "00_00_14_20_90_EF_42_"                         "00_00_00_00_00_00_00_"                     
    6.3188e+05     s      "1dbb0d08"    1.5986e+09    1.5613e+09    3.6168e+06    3.6168e+06    "440096"     NaN       8        11     "DE8C46"        0        0     "00002C"     "000037"     "5D_44_00_96_DE_8C_46_"                         "00_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb3814"    1.5986e+09    1.5613e+09    1.5529e+07    1.5529e+07    <missing>    NaN       2         0     "BD102F"        0        0     "4841AA"     "4841AA"     "02_E1_9A_1F_BD_10_2F_"                         "00_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb382c"    1.5986e+09    1.5613e+09    1.5533e+07    1.5533e+07    <missing>    NaN       8         0     "BD102E"        0        0     "4841AB"     "4841AB"     "02_E1_9A_1F_BD_10_2E_"                         "00_00_00_00_00_00_01_"                     
     6.319e+05     s      "1dbb3ea8"    1.5986e+09    1.5613e+09    2.4288e+06    2.4288e+06    <missing>    NaN       8         1     "141458"        0        1     "C466C9"     "C466C9"     "08_00_08_00_14_14_58_"                         "10_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb3ec0"    1.5986e+09    1.5613e+09    6.3836e+06    6.3836e+06    "440096"     NaN       8        17     "7FC4A7"        0        0     "000000"     "000000"     "8D_44_00_96_99_09_4B_AB_18_04_2C_7F_C4_A7_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb3ee4"    1.5986e+09    1.5613e+09    6.5866e+06    6.5866e+06    "49F01F"     NaN      14        18     "F781AB"        0        0     "000000"     "000000"     "90_49_F0_1F_33_09_D7_63_2F_B9_D8_F7_81_AB_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb3ee8"    1.5986e+09    1.5613e+09    6.5866e+06    6.5866e+06    "49F01F"     NaN      12        18     "F781AB"        0        0     "000000"     "000000"     "90_49_F0_1F_33_09_D7_63_2F_B9_D8_F7_81_AB_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb411c"    1.5986e+09    1.5613e+09     1.498e+07     1.498e+07    "4841AA"     NaN      15        11     "F493DD"        0        0     "000000"     "000000"     "5D_48_41_AA_F4_93_DD_"                         "00_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb4128"    1.5986e+09    1.5613e+09    1.4982e+07    1.4982e+07    "4841AA"     NaN      13        11     "F493DD"        0        0     "000000"     "000000"     "5D_48_41_AA_F4_93_DD_"                         "00_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb4134"    1.5986e+09    1.5613e+09    1.4983e+07    1.4983e+07    "4841AA"     NaN       3        11     "F493DD"        0        0     "000000"     "000000"     "5D_48_41_AA_F4_93_DD_"                         "00_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb4138"    1.5986e+09    1.5613e+09    1.4984e+07    1.4984e+07    <missing>    NaN      10         3     "F493DD"        0        1     "FF38B7"     "FF38B7"     "1D_48_41_AA_F4_93_DD_"                         "10_00_00_00_01_00_00_"                     
     6.319e+05     s      "1dbb410c"    1.5986e+09    1.5613e+09    1.4968e+07    1.4968e+07    <missing>    NaN       8        20     "CA0252"        0        0     "B0A2FF"     "B0A2FF"     "A0_00_17_B1_B2_29_E1_2A_C5_95_39_CA_02_52_"    "00_00_00_00_00_00_00_00_11_10_11_11_11_01_"
     6.319e+05     s      "1dbb414c"    1.5986e+09    1.5613e+09    1.4986e+07    1.4986e+07    "4841AA"     NaN       2        11     "F493DD"        0        0     "000000"     "000000"     "5D_48_41_AA_F4_93_DD_"                         "00_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb41c0"    1.5986e+09    1.5613e+09    1.5501e+07    1.5501e+07    <missing>    NaN      14         4     "2D51F8"        0        0     "4BB863"     "4BB863"     "20_00_10_B8_2D_51_F8_"                         "00_00_00_00_00_00_00_"                     
     6.319e+05     s      "1dbb41d8"    1.5986e+09    1.5613e+09    1.5666e+07    1.5666e+07    <missing>    NaN      12        20     "A68371"        0        0     "440130"     "440130"     "A0_00_13_38_20_05_50_72_31_38_20_A6_83_71_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     f      "1dbb42d0"    1.5986e+09    1.5613e+09         48335         48338    "1402"         1       7       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb42d8"    1.5986e+09    1.5613e+09         50509         50509    "1402"         1       5       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb42e0"    1.5986e+09    1.5613e+09    7.7729e+05     7.773e+05    "7324"         1      12       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb42e8"    1.5986e+09    1.5613e+09    7.7824e+05    7.7825e+05    "7324"         1      15       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb42f0"    1.5986e+09    1.5613e+09    7.8036e+05    7.8037e+05    "7324"         1      13       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb42f8"    1.5986e+09    1.5613e+09    7.8142e+05    7.8143e+05    "7324"         1       3       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb4300"    1.5986e+09    1.5613e+09    7.8212e+05    7.8212e+05    "7324"         1      10       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb4770"    1.5986e+09    1.5613e+09    1.5108e+07    1.5108e+07    "6640"         1       8       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb4778"    1.5986e+09    1.5613e+09     1.511e+07     1.511e+07    "7665"         1       6       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb4780"    1.5986e+09    1.5613e+09    1.5114e+07    1.5114e+07    "6640"         1       7       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     s      "1dbb485c"    1.5986e+09    1.5613e+09    1.9059e+06    1.9059e+06    <missing>    NaN       8        21     "F33E4F"        0        0     "440096"     "440096"     "A8_00_08_00_C8_48_00_30_A4_00_00_F3_3E_4F_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4880"    1.5986e+09    1.5613e+09    2.3469e+06    2.3469e+06    "4841AA"     NaN      12        17     "F5596C"        0        0     "000000"     "000000"     "8D_48_41_AA_68_D1_F4_B2_51_07_42_F5_59_6C_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4884"    1.5986e+09    1.5613e+09    2.3478e+06    2.3478e+06    "4841AA"     NaN      15        17     "F5596C"        0        0     "000000"     "000000"     "8D_48_41_AA_68_D1_F4_B2_51_07_42_F5_59_6C_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb488c"    1.5986e+09    1.5613e+09      2.35e+06      2.35e+06    "4841AA"     NaN      13        17     "F5596C"        0        0     "000000"     "000000"     "8D_48_41_AA_68_D1_F4_B2_51_07_42_F5_59_6C_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f04"    1.5986e+09    1.5613e+09    1.4717e+07    1.4717e+07    "49C264"     NaN       4        17     "1E4E20"        0        0     "3EB884"     "2B2EA5"     "8C_49_C2_64_39_9F_81_93_F2_10_08_1E_4E_20_"    "00_00_10_10_00_10_10_01_00_00_01_10_10_01_"
     6.319e+05     s      "1dbb4f0c"    1.5986e+09    1.5613e+09    1.4718e+07    1.4718e+07    "49D2E4"     NaN      19        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f14"    1.5986e+09    1.5613e+09    1.4719e+07    1.4719e+07    "49D2E4"     NaN      13        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f2c"    1.5986e+09    1.5613e+09    1.4721e+07    1.4721e+07    "49D2E4"     NaN      10        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_11_01_01_00_00_10_01_01_01_01_00_00_11_"
     6.319e+05     s      "1dbb4f48"    1.5986e+09    1.5613e+09    1.4723e+07    1.4723e+07    "49D2E4"     NaN       0        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f54"    1.5986e+09    1.5613e+09    1.4724e+07    1.4724e+07    "49D2E4"     NaN       2        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f58"    1.5986e+09    1.5613e+09    1.4724e+07    1.4724e+07    "49D2E4"     NaN       1        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f64"    1.5986e+09    1.5613e+09    1.4725e+07    1.4725e+07    "49C264"     NaN       9        17     "1E4E20"        0        0     "3EB884"     "2B2EA5"     "8C_49_C2_64_39_9F_81_93_F2_10_08_1E_4E_20_"    "00_00_10_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f74"    1.5986e+09    1.5613e+09    1.4727e+07    1.4727e+07    "49D2E4"     NaN       8        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4f80"    1.5986e+09    1.5613e+09    1.4728e+07    1.4728e+07    "49D2E4"     NaN       6        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb4fdc"    1.5986e+09    1.5613e+09    1.4734e+07    1.4734e+07    "49D2E4"     NaN       5        17     "BECE28"        0        0     "000000"     "000000"     "8C_49_D2_E4_39_BF_A1_97_F2_10_0A_BE_CE_28_"    "00_00_00_00_00_00_00_00_00_00_00_00_00_00_"
     6.319e+05     s      "1dbb50fc"    1.5986e+09    1.5613e+09    1.5305e+07    1.5305e+07    "4BB863"     NaN      14        17     "A1170B"        0        0     "000000"     "000000"     "8D_4B_B8_63_60_87_85_3E_70_C2_78_A1_17_0B_"    "00_00_00_00_00_00_00_00_00_01_01_01_00_00_"
     6.319e+05     f      "1dbb51f4"    1.5986e+09    1.5613e+09    1.5047e+06    1.5047e+06    "1000"         1      12       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb51fc"    1.5986e+09    1.5613e+09    1.5055e+06    1.5055e+06    "1000"         1      16       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5204"    1.5986e+09    1.5613e+09    1.5063e+06    1.5063e+06    "1000"         1      12       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb520c"    1.5986e+09    1.5613e+09    1.5071e+06    1.5071e+06    "1000"         1      15       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5214"    1.5986e+09    1.5613e+09     1.508e+06     1.508e+06    "1000"         1      13       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5304"    1.5986e+09    1.5613e+09    3.6582e+06    3.6582e+06    "6373"         1       2       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb530c"    1.5986e+09    1.5613e+09    3.6584e+06    3.6584e+06    "330"          1       1       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5314"    1.5986e+09    1.5613e+09    3.6586e+06    3.6586e+06    "6640"         1       9       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb531c"    1.5986e+09    1.5613e+09    3.6656e+06    3.6656e+06    "6640"         1       7       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5324"    1.5986e+09    1.5613e+09    5.7941e+06     5.794e+06    "1000"         1      12       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb532c"    1.5986e+09    1.5613e+09    5.7948e+06    5.7948e+06    "1000"         1      16       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5334"    1.5986e+09    1.5613e+09    5.7949e+06    5.7949e+06    "1000"         1      15       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb533c"    1.5986e+09    1.5613e+09     5.798e+06     5.798e+06    "1000"         1      13       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5344"    1.5986e+09    1.5613e+09    5.7994e+06    5.7994e+06    "1000"         1      10       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb534c"    1.5986e+09    1.5613e+09    5.7996e+06    5.7996e+06    "1000"         1       0       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5354"    1.5986e+09    1.5613e+09    5.8008e+06    5.8008e+06    "1000"         1       9       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb535c"    1.5986e+09    1.5613e+09    5.8017e+06    5.8017e+06    "1000"         1       8       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5364"    1.5986e+09    1.5613e+09    5.8027e+06    5.8027e+06    "1000"         1       2       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb536c"    1.5986e+09    1.5613e+09    5.8039e+06    5.8039e+06    "1000"         1       6       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5374"    1.5986e+09    1.5613e+09    5.8078e+06    5.8078e+06    "1000"         1       7       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb537c"    1.5986e+09    1.5613e+09    7.9629e+06    7.9629e+06    "6640"         1      12       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5484"    1.5986e+09    1.5613e+09    1.0134e+07    1.0134e+07    "1000"         1       2       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb548c"    1.5986e+09    1.5613e+09    1.0135e+07    1.0135e+07    "1000"         1       6       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb5494"    1.5986e+09    1.5613e+09    1.0136e+07    1.0136e+07    "1000"         1      11       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb549c"    1.5986e+09    1.5613e+09    1.0139e+07    1.0139e+07    "1000"         1       7       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb54a4"    1.5986e+09    1.5613e+09    1.0141e+07    1.0141e+07    "1000"         1       5       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb54ac"    1.5986e+09    1.5613e+09    1.2264e+07    1.2264e+07    "6640"         1      12       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb54b4"    1.5986e+09    1.5613e+09    1.2264e+07    1.2264e+07    "6640"         1      16       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb54bc"    1.5986e+09    1.5613e+09    1.2264e+07    1.2264e+07    "6640"         1      15       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb54c4"    1.5986e+09    1.5613e+09    1.2269e+07    1.2269e+07    "6640"         1      10       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb54cc"    1.5986e+09    1.5613e+09    1.2269e+07    1.2269e+07    "6640"         1       0       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>                                   
     6.319e+05     f      "1dbb54d4"    1.5986e+09    1.5613e+09     1.227e+07     1.227e+07    "6640"         1       9       NaN     <missing>     NaN      NaN     <missing>    <missing>    <missing>                                       <missing>