In mathematics we introduce many different kinds of notation, and sometimes even a single object or construction can be represented by many different notations. To take two very different examples, the derivative of a function $y = f(x)$ can be written $f'(x)$, $D_x f$, or $\frac{dy}{dx}$; while composition of morphisms in a monoidal category can be represented in traditional linear style, linearly but in diagrammatic order, using pasting diagrams, using string diagrams, or using linear logic / type theory. Each notation has advantages and disadvantages, including clarity, conciseness, ease of use for calculation, and so on; but even more basic than these, a notation ought to be correct, in that every valid instance of it actually denotes something, and that the syntactic manipulations permitted on the notation similarly correspond to equalities or operations on the objects denoted.
Mathematicians who introduce and use a notation do not usually study the notation formally or prove that it is correct. But although this task is trivial to the point of vacuity for simple notations, for more complicated notations it becomes a substantial undertaking, and in many cases has never actually been completed. For instance, in Joyal-Street The geometry of tensor calculus it took some substantial work to prove the correctness of string diagrams for monoidal categories, while the analogous string diagrams used for many other variants of monoidal categories have, in many cases, never been proven correct in the same way. Similarly, the correctness of the "Calculus of Constructions" dependent type theory as a notation for a kind of "contextual category" took a lot of work for Streicher to prove in his book Semantics of type theory, and most other dependent type theories have not been analogously shown to be correct as notations for category theory.
My question is, among all these notations which have never been formally proven correct, has any of them actually turned out to be wrong and led to mathematical mistakes?
This may be an ambiguous question, so let me try to clarify a bit what I'm looking for and what I'm not looking for (and of course I reserve the right to clarify further in response to comments).
Firstly, I'm only interested in cases where the underlying mathematics was precisely defined and correct, from a modern perspective, with the mistake only lying in an incorrect notation or an incorrect use of that notation. So, for instance, mistakes made by early pioneers in calculus due to an imprecise notion of "infinitesimal" obeying (what we would now regard as) ill-defined rules don't count; there the issue was with the mathematics, not (just) the notation.
Secondly, I'm only interested in cases where the mistake was made and at least temporarily believed publically by professional (or serious amateur) mathematician(s). Blog posts and arxiv preprints count, but not private conversations on a blackboard, and not mistakes made by students.
An example of the sort of thing I'm looking for, but which (probably) doesn't satisfy this last criterion, is the following derivation of an incorrect "chain rule for the second derivative" using differentials. First here is a correct derivation of the correct chain rule for the first derivative, based on the derivative notation $\frac{dy}{dx} = f'(x)$:
$$\begin{align}
z &= g(y)\\
y &= f(x)\\
dy &= f'(x) dx\\
dz &= g'(y) dy\\
&= g'(f(x)) f'(x) dx
\end{align}$$
And here is the incorrect one, based on the second derivative notation $\frac{d^2y}{dx^2} = f''(x)$:
$$\begin{align}
d^2y &= f''(x) dx^2\\
dy^2 &= (f'(x) dx)^2 = (f'(x))^2 dx^2\\
d^2z &= g''(y) dy^2\\
&= g''(f(x)) (f'(x))^2 dx^2
\end{align}$$
(The correct second derivative of $g\circ f$ is $g''(f(x)) (f'(x))^2 + g'(f(x)) f''(x)$.) The problem is that the second derivative notation $\frac{d^2y}{dx^2}$ cannot be taken seriously as a "fraction" in the same way that $\frac{dy}{dx}$ can, so the manipulations that it justifies are incorrect. However, I'm not aware of this mistake ever being made and believed in public by a serious mathematician who understood the precise meaning of derivatives, in a modern sense, but was only led astray by the notation.
Edit 10 Aug 2018: This question has attracted some interesting answers, but none of them is quite what I'm looking for (though Joel's comes the closest), so let me clarify further. By "a notation" I mean a systematic collection of valid syntax and rules for manipulating that syntax. It doesn't have to be completely formalized, but it should apply to many different examples in the same way, and be understood by multiple mathematicians — e.g. one person writing $e$ to mean two different numbers in the same paper doesn't count. String diagrams and categorical type theory are the real sort of examples I have in mind; my non-example of differentials is borderline, but could in theory be elaborated into a system of syntaxes for "differential objects" that can be quotiented, differentiated, multiplied, etc. And by saying that a notation is incorrect, I mean that the "understood" way to interpret the syntax as mathematical objects is not actually well-defined in general, or that the rules for manipulating the syntax don't correspond to the way those objects actually behave. For instance, if it turned out that string diagrams for some kind of monoidal category were not actually invariant under deformations, that would be an example of an incorrect notation.
It might help if I explain a bit more about why I'm asking. I'm looking for arguments for or against the claim that it's important to formalize notations like this and prove that they are correct. If notations sometimes turn out to be wrong, then that's a good argument that we should make sure they're right! But oppositely, if in practice mathematicians have good enough intuitions when choosing notations that they never turn out to be wrong, then that's some kind of argument that it's not as important to formalize them.
Best Answer
Here is an example from set theory.
Set theorists commonly study not only the theory $\newcommand\ZFC{\text{ZFC}}\ZFC$ and its models, but also various fragments of this theory, such as the theory often denoted $\ZFC-{\rm P}$ or simply $\ZFC^-$, which does not include the power set axiom. One can find numerous instances in the literature where authors simply define $\ZFC-{\rm P}$ or $\ZFC^-$ as "$\ZFC$ without the power set axiom."
The notation itself suggests the idea that one is subtracting the axiom from the theory, and for this reason, I find it to be instance of incorrect notation, in the sense of the question. The problem, you see, is that the process of removing axioms from a theory is not well defined, since different axiomizations of the same theory may no longer be equivalent when one drops a common axiom.
And indeed, that is exactly the situation with $\ZFC^-$, which was eventually realized. Namely, the theory $\ZFC$ can be equivalently axiomatized using either the replacement axiom or the collection axiom plus separation, and these different approaches to the axiomatization are quite commonly found in practice. But Zarach proved that without the power set axiom, replacement and collection are no longer equivalent.
He also proved that various equivalent formulations of the axiom of choice are no longer equivalent without the power set axiom. For example, the well-order principle is strictly stronger than the choice set principle over $\text{ZF}^-$.
My co-authors and I discuss this at length and extend the analysis further in:
We found particular instances in the previous literature where researchers, including some prominent researchers (and also some of our own prior published work), described their theory in a way that leads actually to the wrong version of the theory. (Nevertheless, all these instances were easily fixable, simply by defining the theory correctly, or by verifying collection rather than merely replacement; so in this sense, it was ultimately no cause for worry.)