The operation of addition is a binary operation: it is an operation defined on pairs of real (or complex) numbers. When we write something like $a+b+c$, apparently adding three numbers, we’re really doing repeated addition of two numbers, either $(a+b)+c$ or $a+(b+c)$ (assuming that we don’t change the order of the terms); one of the basic properties of this operation is that it doesn’t actually matter in which order we do these repeated binary additions, because they all yield the same result.
It’s easy enough to understand what it means to do two successive additions to get $a+b+c$, or $200$ to get $a_0+a_1+\ldots+a_{200}$; it’s not so clear what it means to do infinitely many of them to get $\sum_{k\ge 0}a_k$. The best way that’s been found to give this concept meaning is to define this sum to be the limit of the finite partial sums:
$$\sum_{k\ge 0}a_k=\lim_{n\to\infty}\sum_{k=0}^na_k\tag{1}$$
provided that the limit exists. For each $n$ the summation inside the limit on the righthand side of $(1)$ is an ordinary finite sum, the result of performing $n$ ordinary binary additions. This is always a meaningful object. The limit may or may not exist; when it does, it’s a meaningful object, too, but it’s the outcome of a new kind of operation. It is not the result of an infinite string of binary additions; we don’t even try to define such a thing directly. Rather, we look at finite sums, which we can define directly from the ordinary binary operation of addition, and then take their limit. In doing this we combine an algebraic notion, addition, with an analytic notion, that of taking a limit.
Finite sums like $a_0+\ldots+a_{200}$ all behave the same way: they always exist, and we can shuffle the terms as we like without changing the sum. Infinite series do not behave the same way: $\sum_{n\ge 0}a_n$ does not always exist, and shuffling the order of the terms can in some cases change the value. This really is a new operation, with different properties.
The logic is as follows:
- As yet, there is no meaning to the sequence of symbols $\sum_{n=1}^\infty a_n$, given the values of $a_n$, if the sum does not converge in the classical sense.
- We want to give this expression some meaning, perhaps because computing the answer to some problem in some naïve way gives such a divergent sum as an answer, which is obviously nonsensical, but yet which is connected to a useful answer in a way we want to determine. (Frequently the case in physics.)
- The only way we could ever hope to gain anything useful from our definition is if
- it agrees with the convergent case where both are defined; and
- it shares key properties with the convergent case.
- Let's try to choose some potential properties first. Write $S_n[a_n] = \sum_{n=1}^\infty a_n$ Here are three ideas:
- $S_n[a_n] = a_1 + S_n[a_{n+1}]$ (stability), i.e. $$a_1 + a_2 + \cdots = a_1 + (a_2 + \cdots)$$
- $S_n[a_n + b_n] = S_n[a_n] + S_n[b_n]$ (linearity part 1)
- $S_n[k a_n] = k S_n[a_n]$ (linearity part 2)
- These are compatible with the normal rules, and therefore we can also require that:
- for convergent series, $S_n$ is indeed the usual summation (regularity).
- They are also consistent, so we shouldn't be introducing any contradictions using this definition.
- Would these rules suffice to compute actual results for divergent sums? Yes! For example, stability and linearity mean that $1 + 2 + 4 + \cdots$ can be computed by observing $S_n[a_n] = 1 + 2 S_n[a_n]$ and so $S_n[a_n] = -1$.
So now one question we can answer is why do loads of different summation techniques give you this same answer? If the technique is constructed to obey the above rules too! If these rules are sufficient to compute the sum, then regardless of what else you specify about $S_n$, you will have to get the above answer - provided it is defined. (On the other hand, Cesàro summation does not give a value to the above sum, for instance.)
However, whilst Nørlund means (including Cesàro sums) and Abelian means are regular, linear and stable, some other techniques are not. (Note that these are not equivalent: this means some sums cannot be computed using just regularity, linearity and stability even though these schemes do define them whilst obeying these properties.) Notably, stability is quite often dropped or weakened.
An example? Consider the series $S=1+1+1+\cdots$. By zeta function regularization, this has an answer of $-\frac 1 2$. However, if we ascribed it a value in a stable scheme, we would have $S = 1+S$, which is a contradiction because $1 \neq 0$. Fine, this just means that no stable scheme will define this quantity. Therefore zeta function regularization is not stable.
Weird, huh? Regularity and linearity can also be dropped, but then we are getting beyond the realm of most physics applications, I think.
Apparently (Wikipedia says) Borel summation is not stable, yet it still often gives the same answers as other summation methods. I'm not quite sure why, but I suspect Borel summation will turn out to be stable 'in certain situations'.
The analogy with the Gamma function is good. If one chooses some integral to define $\Gamma(z)$ which only works for some (let's say open) domain $z\in D$, then one has the problem of trying to decide what $\Gamma$ should be outside $D$. But noticing that within $D$ the $\Gamma$ function has the property that it is analytic (meromorphic perhaps) one can just impose two requirements on $\Gamma(z)$:
- it agrees with the previous definition on $D$; and
- it is meromorphic on $\mathbb C$.
These are exactly analogous to the two requirements made of $S_n$ above - and because they once more turn out to uniquely specify $\Gamma$ everywhere, they will agree with any other method of expressing $\Gamma$, so long as one keeps the meromorphic property.
As far as Cesàro summation and Fourier series go, this is slightly different. Here, one takes a Fourier series which does convergence pointwise. The basic point is that Fourier series can oscillate rapidly causing uniform convergence issues (there is always a big error somewhere, but not in the same place as earlier), but if one smooths out this oscillation one can make the convergence uniform.
This is not a very mysterious application of Cesàro summation - it just decreases the effect of adding a term which will eventually be cancelled by later terms.
Edit: In answer to your concerns that this is all "nonsense" - your problem is that you are thinking about summation techniques as being used for improving numerical convergence, whereas in general there are lots more properties of summations that you are interested in. Here is one.
Suppose you have a physical system which is described by some property $x$, which might be a coupling constant for instance. Now suppose that you are interested in some property $F(x)$ which is a measurable output from the system. Further, suppose you reckon that $F(x)$ is an analytic function of $x$.
In fact, if you assume $x$ is really small, you are very happy to find $F(x) = b_0 + b_1 x + b_2 x^2 + \cdots$, where perhaps $b_n = (-2)^n$. For nice small numbers, you can work out the answer by taking a few terms.
Now imagine you're interested in measuring at $x=37.4$. Suppose further that you didn't really understand about series only working when they're small, so you just plugged in $x$.
You get $S = \sum (37.4)^n (-2)^n = \sum (-74.8)^n = 1 - 74.8 + 74.8^2 - \cdots$ and think "~%*£%!". But now you laugh cheekily, and apply the rules from above. You find $S = 1 - 74.8 S$ and so $S = 1/75.8 = 0.0132\ldots$.
That's rubbish, right?
Wrong. Notice that for small $x$, $$F(x) = 1 - 2x + 4x^2 - \cdots = \frac1{1+2x}$$and the unique analytic continuation of this results in $$F(37.4) = \frac{1}{1+2\times 37.4} = \frac 1 {75.8} = 0.0132\ldots$$
Magic? Nope. Just the fact that our definition plays nice with analyticity. Whenever you sum up analytic functions outside their radius of convergence, you should get the right answer.
Now the real places where these tricks are used (and not always really understood) in physics are things like the critical dimension of a string, or in the quick computation of the magnitude of the Casimir effect. The deal is similar to the above, except that one doesn't have an $x$ so one doesn't work out the dependence on a parameter; one simply computes a single quantity in an expansion, then hopes and prays that the physics in the background is really nice (like the analyticity above) so that it plays ball with your summation - and then you can use one of these summation tricks and it should give you the right answer, even if the series you wrote down diverged!
Having thought about the pictorial aide, I think all Carl Bender is trying to say is that - working on the Riemann sphere to make things nice and clear, where once again complex infinity is a unique point at the northern hemisphere - there is a definite sense in which lines can go to infinity and come back from the 'other side'. 
It's not clear to me that this is anything more than a vague hint though, most notably because if one plots the partial sums on the sphere (and treat it as a normal sphere) it is clear that they accumulate to the point of complex infinity, and can never reach it or pass it. I'm curious about this though, and whether it can be shown to have any rigorous value.
Perhaps a better way to look at this is to say that if one does the sum by a technique equivalent to analytic continuation, then one can obtain any value for the function. If one calculated $1+2+4+\cdots$ by writing $1+2x+4x^2+\cdots = 1/(1-2x)$ then analytically continued in the complex plane from near 0 to 1 to avoid the singularity at $\frac 1 2$, one can get negative results because one is wandering around the complex plane; specifically if one makes a really tight circle around $\frac 1 2$ then the value for the sum goes round a 'small' circle near complex infinity. Drawing this on the sphere links up with the intuition above.
Best Answer
The official meaning of the assertion that $$1+r+r^2+r^3+\cdots=S$$ is that $$\lim_{n \to \infty}(1+r+r^2+\cdots +r^n) =S$$.
By the usual formula for the sum of a finite geometric series, $$1+r+r^2+\cdots +r^n=\frac{1-r^{n+1}}{1-r}$$ (unless, of course, $r=1$). It is reasonably clear that if $|r|<1$, $$\lim_{n\to\infty}\frac{1-r^{n+1}}{1-r}=\frac{1}{1-r}$$ and that if $|r|\ge 1$, then $$\lim_{n \to \infty}(1+r+r^2+\cdots +r^n)$$ does not exist.
But that is not really your question. What I think you are saying is roughly this: let $$S=1+r+r^2+r^3+\cdots$$ Then at the level of formal "algebraic" manipulation, we have (??) $$S-1=rS$$ and "therefore" (??) $S=1/(1-r)$ (at least if $r \ne 1$).
But do "formal" manipulations always yield correct results? Not necessarily. For example, you have probably already seen plausible-looking algebraic manipulations that appear to yield the absurd "result" that $0=1$ (usually here the flaw is a carefully hidden division by $0$.)
The formal manipulation yields, if we take $r=1/2$, the perfectly correct result that $$1+\frac{1}{2}+\frac{1}{2^2}+\frac{1}{2^3}+\cdots =2$$ This makes physical sense. If you look at the real number line, and put a dot at $1$, then at $1+1/2$, then at $1+1/2+1/4$, and so on, your dots are clearly approaching $2$, and are after a while indistinguishably close to $2$.
Now look at the formal sum $$S=1+2+2^2+2^3+\cdots$$ Purely formal manipulation then yields that $S=-1$. Does this make any kind of physical sense? Certainly if you add more and more terms of the above series, you are not getting in any sense close to $-1$. And at the crude informal level, it is clear that the sum is "infinite," whatever that may mean. Formal manipulations on non-existent objects can yield absurd results.
So in this case formal manipulation has produced a result that makes no physical sense. There are a number of instances where blind manipulation yields wrong results, so early in university, students are trained to apply algebraic "rules" only in situations where these rules are valid.
However, the situation is more complicated than that! Once upon a time complex numbers, when they came up in a formal calculation, were dismissed as "absurd." Now they are part of the essential toolkit of the electrical engineer! There are similar phenomena with series.
In combinatorics, for example, formal power series are successfully used, at the purely manipulational level, even when the series technically do not converge. And in various areas of mathematics, divergent series of a suitable type can be very useful, even in computations!
But at this stage, you should be disciplined, and apply formal manipulations only in situations where you know, or have been assured that, they are "safe." Later, perhaps, you can explore situations where venturing beyond safe confines yields interesting and useful results.