I learned that condition number of an operator/function is a measure of how sensitive the output of the function is with respect to a small change in the input argument.
But then, what is the significance of condition number in the context of numerical methods/analysis? I am not able to see why people emphasize condition number particularly in numerical analysis.
Isn't condition number a characteristic of a given problem regardless of what numerical methods are used to solve the problem?
Best Answer
Condition numbers are relevant because they establish clear boundaries on the accuracy that can be achieved on a given computer architecture.
For the sake of simplicity, let us consider the problem of computing a real function $f : \mathbb{R} \rightarrow \mathbb{R}$. Assuming that $f$ is differentiable, and $x \not = 0$ is such that $f(x) \not = 0$, then the condition number of $f$ is given by $$ \kappa_f(x) = \left|\frac{x f'(x)}{f(x)}\right|.$$ As you said, the condition number measures the sensitivity of the output to small changes in the input. Specifically, if $\bar{x} \approx x$, then $$ \left |\frac{f(x) - f(\bar{x})}{f(x)} \right| \approx \kappa_f(x) \left| \frac{x-\bar{x}}{x}\right| \tag{1}.$$ In popular terms, the condition number magnifies the relative error. If the condition number is large, then we can not afford to be sloppy when computing an approximation $\bar{x}$ of $x$ or $f(\bar{x})$ will be useless as an approximation of $f(x)$.
Now consider an implementation which realizes the function $f$. We seek to compute $y = f(x)$. It is exceedingly unlikely, that our code will return the correct value $y$, rather we will get a good approximation $\hat{y} \approx y$. If our code is backward stable, then $$\hat{y} = f(\hat{x})$$ for a value $\hat{x}$ which is exceedingly close to $x$, ideally, $$ \left| \frac{x-\hat{x}}{x} \right| \leq C u, \tag{2}$$ where $C > 0$ is a modest constant independent of $x$ and $u$ is the unit round off error. In this case, we have $$ \left |\frac{f(x) - f(\hat{x})}{f(x)} \right| \lesssim C \kappa_f(x) u.$$