My scale is set to 2. Yet arc (using 10.2) is showing (in my select by attribute – get unique values) my number of 26.67 as 26.670000000000002. My value of 25.25 is shown correctly as 25.25 in the same field.
Again my value of 32.60 is shown as 32.6000000000001.
I have tried replacing the values with the field calculator, but it goes back to these long decimal numbers all on its own.
Why is it doing this and how do I fix it?
These values are not more than 2 decimals, showing more is wrong. but it has to be a double because there are some negatives ..
Best Answer
This is more an overall question of how computers deal with and represent numbers and is an important question to grasp for computer programming/GIS in general. Considering the vast amount of papers on this topic I think it is best to give you a few links and they will lead into more research if you are interested on the topic.
One of the classics on the topic is 'What Every Computer Scientist Should Know About Floating-Point Arithmetic' or just go to any number of questions on stack overflow that deal with this topic (look on the right side for links to other similar questions on the topic). One more interesting link that documents how often this question is asked and provides many helpful links (at the bottom of the page) for better understanding.