A definition of the connected sum of two $n$-manifolds $M$ and $M'$ begins by considering two $n$-balls $B$ in $M$, $B'$ in $M'$, and glueing the varieties $M\setminus \mathring B$ and $M'\setminus \mathring B'$ along their boundary (an $(n-1)$-sphere) by an orientation-reversing homeomorphism. The construction depends a priori on these various choices, but it is asserted at many places of the litterature (Lee's book on topological manifolds for example, as well as Wikipedia)
that the result does not depend on these choices.
In the differentiable case, a reference is given to a theorem of Palais (Natural operations on differential forms, Thm. 5.5) which asserts — roughly — that two embedding of $n$-balls differ by a global diffeomorphism which is isotopic to identity.
Are the details of this independence written somewhere in the litterature,
both in the continuous and the smooth case?
Best Answer
In the topological category the proof that connected sum is well-defined depends on the Annulus Theorem, first proved by Kirby; the necessity of the Annulus Theorem is seen from Bruno Martelli's answer. So you are not likely to find a proof before Kirby's paper. Perhaps someone jotted a proof down, maybe someone who thought about the Annulus Theorem when it was still a conjecture, and realized that well-definedness of connected sum was a good application. But, I do not know.
Anyway, the proof is straightforward once you have the Annulus Theorem. Here's a sketch.
There's a couple of missing hypotheses. One must assume $M,M'$ are connected. One must also assume $M,M'$ are oriented. And one must assume the balls $B,B'$ are "nicely embedded"; at the minimum, assume that the boundary spheres $S,S'$ are locally bicollared, which implies globally bicollared by Brown's theorem. This rules out nastiness like an Alexander horned ball.
Now one shows that the connected sum is independent of the choice of gluing map $S \to S'$ (assumed to reverse orientation). This follows from the fact that any two homeomorphisms $S \to S'$ which agree on orientations are isotopic: once that is known, one absorbs the isotopy into the collar neighborhoods. Proving this fact may already require the Annulus Theorem.
For the rest, it suffices to prove that for any two nicely embedded balls $B_1,B_2 \subset M$ there exists an orientation preserving homemeorphism of $M$ taking $B_1$ to $B_2$, in fact an ambient isotopy. Using the boundary bicollaring, we may assume $B_1,B_2$ are contained respectively in open balls $U_1,U_2$, which are centered on points $p_1,p_2$ in some coordinate chart. We can also assume that $p_1=p_2$, because there is an ambient isotopy of $M$ taking $p_1$ to $p_2$: connect $p_1$ to $p_2$ by a path, cover the path by finitely many charts, and concatenate a sequence of ambient isotopies supported in these finitely many charts, moving $p_1$ along the path step by step to $p_2$. We can also replace $B_1$ by an arbitrarily small subball in $U_1$ centered at $p_1$, and similarly for $B_2$; this is straightforward to check using an ambient isotopy supported in the coordinate charts for $U_1$ and $U_2$. In particular, we can assume $B_1$ is contained in the interior of $B_2$.
Now apply the annulus theorem: the difference $B_2 \setminus B_1$ is homeomorphic to a sphere crossed with an interval. Using this, one can then ambiently isotope $B_2$ to $B_1$.