MATLAB: What’s going on with the precision calculation

round precision

Last week I posted a question on how to round in Matlab to match a simple calculator but now I think Matlab is getting the calculation all wrong. For example, when calculating:

p15 = 0.8683*1.15;
round(p15,5) = 0.99854

I believe the answer should be 0.99855 and here is why. If I take 10% of 0.8683 (.08683) and 5% of 0.8683 (or 1/2 of 0.08683) = 0.043415 and then sum p15 + 10% + 5% (0.8683 +.08683 + .043415), then the answer is 0.998545.

0.998545 rounds to 0.99855 at 5 decimal places.

Why is Matlab calculating something different than 0.998545???

Best Answer

No. MATLAB is not getting it all wrong. The problem lies in your understanding of floating point arithmetic, and how numbers are stored.

p15 = 0.8683*1.15;
format long g
p15
p15 =
                0.998545

So, it LOOKS like the result is what you would expect. Is it? No. The issue is that floating point numbers are stored NOT in decimal form, but in binary. So, for example, 0.1 is NOT exactly representable in binary in a finite number of binary bits, just like you cannot exactly represent the fraction ⅓ in a finite number of decimal digits. As well, 0.998545 is also not exactly representable in binary.

So, the decimal representation of the number that is stored in binary for p15 is actually:

0.99854499999999990489385481851059012115001678466796875

This is the number as it is actually stored. Well, actually, it is stored in binary. But when you convert it into decimal form, you get that long mess of a decimal number.

So, how is the number 0.998545 stored as a sequence of binary bits? First, here is the repeating bit string that would represent 0.998545:

0.1111111110100000101001010010011010010101100101011111111011011010011001100001001010000011100100000100...

However, as stored in MATLAB, the binary bit used would look more like this:

0.1111111110100000101001010010011010010101100101100

As you can see, the two numbers become subtly different when you go past 50 binary bits or so.

Now, when you convert to decimal and round that result to 5 DECIMAL digits, you get 0.99854. The round goes down, because that is in fact how you should round the complete number shown above.

Welcome to the world of floating point arithmetic. As it turns out, no matter what base you store your numbers in, there will be problems like this that arise. (I believe that some calculators use a decimal form to store numbers, but there too you can contrive simple enough problems that may appear paradoxical if you do not understand how those numbers are stored and used.)

Related Solutions

MATLAB: When with rounding and precision

round(p15 * 1e6)/1e6

By the way, you cannot see the full precision on MS Windows by using sprintf(), but you can on OS-X

0.99854499999999990489385481851059012115001678466796875

On MS Windows you should see http://www.mathworks.com/matlabcentral/fileexchange/22239-num2strexact--exact-version-of-num2str-

MATLAB: 0.35 divide 0.001 return double, and 0.34 divide 0.001 return int.

MATLAB does not represent numbers with fractions in decimal (not even in the Symbolic toolbox, but you have to push hard to prove that for the Symbolic toolbox).

MATLAB itself represents most numbers in IEEE 754 Double Precision, which is a representation that uses one sign bit, 11 bits of binary exponent, and 52 bits of binary fraction. For technical reasons, this gives 53 bits of precision: numbers are effectively represented as a 53 bit integer times a power of 2 (that might be negative). The only fractions that can be exactly represented are those involving powers of 2. It is like having an unreduced fraction in which the denominator is always 2^53. Other denominators such as 10 are only possible if they are powers of 2: for example the system can exactly represent 16ths, but never 3rds or 10ths, not exactly.

This system has exactly the same variety of limitations that sticking strictly to a fixed number of decimal places would have. For example if you pick any fixed number of decimal places, you can never exactly represent 1/3 or 1/7, and instead those end up in decimal being infinite repeating decimals, just like how 1/3 in decimal starts 0.333333333333333 but continues on infinitely. Now in decimal, take that number and multiply it by 3 again, and what you get is 0.999999999999999 rather than exactly 1.

Just so, in binary, 1/10 is an infinite repeating number, and if you truncate it to any finite number of decimal places, and multiply by 10 you do not get exactly 1 back.

Now, with some multiples of 1/10 in binary, when you multiply by 10, the result rounds to the representation of an integer, but that is not the case for all multiples of 1/10 in finite binary: for some of them, after multiplication you get a number that is 1 bit different than an exact integer, just like the 0.99999999999999 is 1 trailing decimal place different from an exact decimal integer.

So 0.34/0.001 with both numbers expressed in finite binary approximation ends up rounding to an exact integer but 0.35/0.001 with both numbers expressed in finite binary approximation ends up rounding to one bit different from an exact integer.

The moral of the story is: Don't Do That!! Avoid using fractions to calculate integer indices, because fractions are only approximations.

Example: instead of

for f=0.01:0.01:1
  G(100*f)=f.^2;
end

You can use

for fi = 1:100
  f=fi/100;
  G(fi) = f.^2;
 end

Best Answer

Related Solutions

MATLAB: When with rounding and precision

MATLAB: 0.35 divide 0.001 return double, and 0.34 divide 0.001 return int.

Related Question