MATLAB: Fprintf is printing strange characters instead of numbers

fprintfMATLAB

I want to print a vector of unsigned integers to a text file, with a space between each number. But the file I get is just weird symbols. I must be doing some trivial mistake, it's not the first time it happens, I can't remember what could be the fix. It's happening on R2018b (but I remember it happening on older versions as well). Here's sample code below:

clear
data = uint32(zeros(1, 1615));
data(1:2:50) = 1;
output = fopen('output.txt', 'wt');
fprintf(output, '%d ', data);
fclose(output);

Output I get is: ‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‱‰‰‰‰‰‰‰‰‰‰ …

Output I want is: 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 …

Best Answer

The problem is due to Notepad broken encoding detection algorithm. For some reason it assumes that the file is encoded in UTF16, where the byte sequence [49 32] (characters '1' and ' ' in ansi and UTF8) indeed represents the character '‱'. Note that simply adding a space, ' ', before your sequence of numbers completely change the behaviour of Notepad's detection algorithm. With a space at the start, it correctly interprets the file as ANSI or UTF8.

As Stephen said don't use notepad. It's broken (and very limited in functionality). A good alternative is the open source Notepad++. You can also simply use matlab's editor.

edit: another option would be to save your output.txt as UTF16, which would then be read correctly by notepad, particularly, if you insert a BOM at the beginning. However, while matlab can read/write UTF16, it's not documented so there may be some edge cases where it doesn't work appropriately. Additionally, notepad is probably the only software that tends to assume UTF16, everything else tends to assume UTF8 by default, so the display will look odd in almost every other text editor (matlab's included).

Related Solutions

MATLAB: Do I get “Out of memory.” when reading only 16 chars

"However what I could think of is that Matlab tries to guess the encoding"

I've had discussions with Mathworks support about this. The whole process is not properly documented unfortunately, which I told them can be a problem.

Indeed, if you open a file without specifying a character encoding, matlab will try to guess the file encoding the first time you either:

use any character reading function such as fgetl, fgets, fscanf, etc.
use fread with a 'char' or '*char' precision
ask for the encoding with the multi-output version of fopen.

I haven't been given the full process of character set detection, but it does read the whole file which indeed can be an issue for large files. If any byte sequence in the file is not a valid UTF8 code point, then the algorithm uses some heuristics to see if it's a CJK encoding and if it still doesn't match, it assumes the local encoding.

To prevent this autodetection to take place, you have to specify an encoding when you fopen the file. If you don't know what the encoding is for your binary file, I'd suggest using 'US-ASCII'. As we mentioned in the comment, it's unlikely that a binary file uses UTF8 unless it prefixes the text by a length.

Unfortunately, it's not easy to go back to pre-2020a behaviour of automatically using the native encoding whatever it is, as R2020a has lost the ability of easily getting the local encoding. On the other hand, relying on native encoding when reading a binary file is asking for trouble.

MATLAB: How to extract sections of data from a csv file

d=[];  % initialize an array for the data
fmt=repmat('%f',1,14);  % format string to match file
fid=fopen('filename');
for i=1:10
  d=[d;cell2mat(textscan(fid,fmt,12,'headerlines',2,  ...
                                    'collectoutput',1 ...
                                    'delimiter',','))]; % read each section;concatenate
  fgetl(fid)  % had to do this to get synchronized again...find it often
end
fid=fclose(fid);

The above returns the data in one array; if you instead want each simulation separately, instead of the concatenation above use a cell array to store each read section--

ERRATUM

fmt=[repmat('%f',1,14) '\n'];  % format string to match line of file data
fid=fopen('filename');
for i=1:10        % repeat for all sections in file
  d(i)=textscan(fid,fmt,12,'headerlines',2,  ...
                           'collectoutput',1 ...
                           'delimiter',',')]; % read each section into cell array
end
fid=fclose(fid);

NB: The \n (newline) in the format string solves the position in the file it seems. I guess it's one of those cases where it's probably technically wrong without it but the scanning routines skip over it transparently in the vectorized portion but not when the scan is started over. Again, "why" is indecipherable as far as I can tell, it "just is".

Best Answer

Related Solutions

MATLAB: Do I get “Out of memory.” when reading only 16 chars

MATLAB: How to extract sections of data from a csv file

Related Question