The classical Brouwer fixed point theorem is stated as follows:
Every continuous map $h: D^2 \to D^2$ has a fixed point, that is, a point $x \in D^2$ with $h(x) = x$. (Here, $D^2$ denotes the closed unit disk in $\Bbb{R}^2$.)
The proof Hatcher (and many others) gives in his Algebraic Topology is as follows:
Like many other authors, Hatcher does not give a formal explanation for the continuity of $r$. Specifically, I'm not convinced by the argument that a small perturbation in $x$ yields a small perturbation in the ray between $h(x)$ and $x$. It seems that if $x$ and $h(x)$ are close, it is seemingly possible that a small perturbation in $x$ may cause the ray to "jump" discontinuously, and point in the other direction.
I am seeking a formal explanation for this argument.
Best Answer
"If $x$ and $h(x)$ are close, why wouldn't a small perturbation in $x$ cause the ray to point in the other direction?"
No, even if $x$ and $h(x)$ are close (whatever the precise meaning is), all sufficiently small perturbations of $x$ (which cause small perturbations of $h(x)$) produce only small changes of the direction of the ray.
Anyway, you are right, one must give a formal proof.
By assumption $h(x) \ne x$ for all $x \in D^2$. Hence the points $h(x)$ and $x$ lie on a unique line $L(x) \subset \mathbb R^2$ which is parameterized by $$l(t) = h(x) + t(x - h(x)) = tx + (1-t)h(x), t \in \mathbb R .$$ This parameterization gives $L(x)$ a direction which we may understand as the difference vector $d(x) = x - h(x) \ne 0$. The ray from $h(x)$ through $x$ is parameterized by $$l(t) = h(x) + td(x), t \ge 0 .$$ It starts at $h(x) \in D^2$ for $t=0$ and reaches $x \in D^2$ for $t=1$. Moreover we have $\lVert l(t) \lVert < 1$ for $0 < t < 1$. If $\lVert x \lVert < 1$ or $\lVert h(x) \lVert < 1$, this follows easily from $$\lVert l(t) \lVert \le t\lVert x \lVert + (1-t)\lVert h(x)\lVert .$$ If $\lVert x \lVert = \lVert h(x) \lVert = 1$, we need the Cauchy-Schwartz-inequality $\lvert a \cdot b \rvert \le \lVert a \rVert \cdot \lVert b \rVert$ in which equality holds iff $a, b$ are linearly dependent. Note that here it is essential that we work with the Euclidean norm. Let us take $a = tx, b = (1-t)h(x)$. Since $\lVert a \rVert + \lVert b \rVert = t\lVert x \rVert + (1-t)\lVert h(x) \rVert = t + 1-t =1$, we get $$\lVert l(t) \rVert^2 = (a + b) \cdot (a +b) = \lVert a \rVert^2 + 2 a \cdot b + \lVert b \rVert^2 \le \lVert a \rVert^2 + 2 \lVert a \rVert \cdot \lVert b \rVert + \lVert b \rVert^2 =(\lVert a \rVert + \lVert b \rVert)^2 = 1$$ where equality is possible only when $a,b$ are linearly dependent. By definition of $a,b$ this means $h(x) = \lambda x$ for some $\lambda$ since $t, 1-t \ne 0$. Clearly $\lvert \lambda \rvert = 1$, thus we must have $h(x) = -x$. But then $l(t) = tx -(1-t)x = (2t-1)x$. Since $-1 < 2t-1 < 1$, we get $\lVert l(t) \rVert = \lvert 2t-1 \rvert \cdot \lVert x \rVert = \lvert 2t-1 \rvert < 1$, a contradiction. Thus equality does not occur.
Geometrically it is clear that $L(x)$ intersects $S^1$ in exactly two points, one of them having the form $l(\tau)$ with $\tau \ge 1$. Formally we must find $t$ such that $l(t) \in S^1$, i.e. $\lVert h(x) + td(x) \rVert = 1$. This yields the equation $$\lVert h(x) \rVert^2 + 2th(x)\cdot d(x) + t^2\lVert d(x) \rVert^2 = 1 . \tag{1}$$ Its solutions are $$t_{1^/2}(x) = \frac{-2h(x)\cdot d(x) \pm \sqrt{4(h(x)\cdot d(x))^2 - 4 \lVert d(x) \rVert^2 (\lVert h(x) \rVert^2-1)}}{2\lVert d(x) \rVert^2} \\= \frac{-h(x)\cdot d(x) \pm \sqrt{(h(x)\cdot d(x))^2 + \lVert d(x) \rVert^2 (1 - \lVert h(x) \rVert^2)}}{\lVert d(x) \rVert^2} \tag{2}$$ Since $\lVert h(x) \rVert \le 1$, we have $1 -\lVert h(x) \rVert^2 \ge 0$ which shows that both $t_{1/2}(x)$ are real. Moreover $\sqrt{(h(x)\cdot d(x))^2 + \lVert d(x) \rVert^2 (1 - \lVert h(x) \rVert^2)} \ge \lvert h(x)\cdot d(x) \rvert$ which shows that $$t_1(x) \ge 0 \ge t_2(x) .\tag{3}$$.
Clearly $$r(x) = l(t_1(x))$$ gives a continuous $r : D^2 \to S^1$. For $x \in S^1$ we have $l(1) = x \in S^1$, i.e. $1$ is a positve solution of $(1)$. By $(3)$ we must have $t_1(x) =1$. Therefore $$r(x) = l(1) = x$$ for all $x \in S^1$.
Although is irrelevant for the definition of $r$, we would geometrically expect that $r(x)$ does not lie before $x$ on the ray from $h(x)$ to $x$, i.e. that $t_1(x) \ge 1$. This can indeed be shown.
Since $l(t) \notin S^1$ for $0 < t < 1$, we see that either $t_1(x) = 0$ or $t_1(x) \ge 1$. Assume that $t_1(x) = 0$. This implies $h(x) = l(0) = r(x) \in S^1$, i.e. $\lVert h(x) \rVert = 1$. From $(2)$ we then get $0 = t_1(x) = \frac{-h(x)\cdot d(x) + \lvert h(x)\cdot d(x)\rvert}{\lVert d(x) \rVert^2}$ which shows that we must have $h(x)\cdot d(x) = h(x)\cdot x - h(x)\cdot h(x) = h(x)\cdot x - 1 \ge 0$, i.e. $$h(x)\cdot x \ge 1 . \tag{4}$$ Hence $$1 \le h(x)\cdot x = \lvert h(x)\cdot x \rvert \le \rVert h(x) \rVert \cdot \rVert x \rVert \le \rVert x \rVert .$$ Thus $\rVert x \rVert \ge 1$ which implies $\lVert x \rVert = 1$ and therefore $\lvert h(x)\cdot x \rvert = \rVert h(x) \rVert \cdot \rVert x \rVert$. This can only happen if $h(x),x$ are linearly dependent, i.e. $x = \lambda h(x)$ with $\lvert \lambda \rvert = 1$. Since $x \ne h(x)$, we see that $\lambda = -1$. But then $h(x)\cdot x = - h(x)\cdot h(x) = - \rVert h(x) \rVert^2 = -1$ which contradicts $(4)$.