I was studying Numerical Analysis by K. Mukherjee; there he discussed Loss of Significant Figures by Subtraction, as followed:
In the subtraction of two approximate numbers, a serious type of error may be present when the numbers are nearly equal. …
He showed how the number of significant figures got decreased after subtraction.
In order to dig more, I googled and came across the term catastrophic cancellation.
I did try to read the wiki article but it was full of terms especially floating point arithmetic which I am really not acquainted it; I did try to read the latter's wiki article but couldn't comprehend the definition:
The term floating point refers to the fact that a number's radix point (decimal point, or, more commonly in computers, binary point) can "float"; that is, it can be placed anywhere relative to the significant digits of the number.
Radix point can float? Unfortunately, I failed to visualise that.
However, my main question, even if there is an error due to the loss of the significant figures, why is this error "catastrophic"?
I found in a decade old page that
The relative error here is infinite.
But how could the loss of significant figures led to an infinite magnitude of relative error?
Could anyone please explain the reason behind the term "catastrophic" keeping in mind that I'm not acquainted with floating point arithmetic?
Best Answer
The root of the evil is that calculators work with a finite number of digits (well, we can't afford machines with infinite resources yet). When you want to represent real-life numbers, some can be huge, some can be tiny, and this raises an issue.
Floating-point:
If the decimal point is fixed, you can' represent them all, there can be so-called overflow or underflow; and for intermediate orders of magnitude, you lose significant digits.
For example, using the $5.5$ fixed-point representation, $127500.$ and $0.0000096695$ cannot be represented, and $\pi=3.14159$ just uses six significant digits, four positions are wasted.
The floating-point representation solves these problems by specifying the position of the decimal point, as in the scientific notation:
$127500.=1.275000000\cdot10^5, 0.0000096695=9.669500000\cdot10^{-6}, \pi=3.141592654\cdot10^0$.
Catastrophic cancellation:
(This concept is actually unrelated to floating-point.)
Assume you want to play with the approximation $\pi\approx355/113=3.14159292\cdots$.
If you want to take the sum with $\pi$, all goes well.
$$\frac{3.14159265+3.14159292}2=3.14159278.$$
But now if you want to take the difference,
$$\frac{3.14159265-3.14159292}2=-0.000000135,$$
only three significant decimals remain. This is a catastrophy, because you started with nine figures and end-up with just three, and this is irreversible ! (Floating-point will not come to the rescue, $-0.000000135=-1.35\cdot10^{-7}$ still has three significant digits.)
This phenomenon makes some problems very difficult to solve numerically.
In some cases, catastrophic cancellation can be avoided. For example, when solving the quadratic equation
$$x^2-1000.001x+1=0,$$
the classical formula says
$$x=\frac{1000.001\pm\sqrt{1000.001^2-4}}2=\frac{100.001\pm999.999}2.$$
Assuming you can only compute with five significant digits, using floating-point, you obtain
$$x=\frac{1000.00\pm999.99}2=999.95\text{ and }0.005,$$ where the second root is very very inaccurate.
You can get a much better estimate by using the fact that the product of the roots is $1$, and the second root evaluates to
$$\frac1{999.95}=0.0010005$$