The first of these is easy:
How are tensors from QM and tensors from linear algebra widely used in geometry related?
They're the same thing ─ though sometimes QM will choose to look only at some specific subset of tensors, e.g. sets which carry group-theoretic representations of the rotation group.
The apparent conflict arises because there's a wide spectrum of ways to talk about tensors, and you're pulling examples from two complete extremes of that spectrum.
Let's bridge the gap by taking the universal-property tensor product, as in the first understanding,
- Theorem: Given two vector spaces $U$ and $V$ over $F$, there exists a vector space $U\otimes V$ and a bilinear product $\otimes:U\times V\to U\otimes V$ such that for every bilinear $f:U\times V\to F$ there exists a $g:U\otimes V$ such that $f(u,v) = g(u\otimes v)$. Moreover, this vector space and mapping are unique up to a canonical isomorphism.
and provide an instantiation of that abstract tensor-product state:
- Claim. Let $U,V$ be vectors spaces over $F$, and choose coordinate representations $i_U:U\to F^n$ and $i_V:V\to F^m$ for them. Then $F^{n\times m}$ and $\otimes: U\times V\to F^{n\times m}$, defined via
$$\otimes:(u,v)\mapsto \left(i_U(u)_ji_V(v)_k\right)_{j,k=1}^{n,m},$$
are an instantiation of the abstract tensor-product as defined above.
Put another way, this just says that you can do tensor products in a coordinate-wise way, in a manner that is reasonably straight-forward to work out.
Quantum mechanics uses tensors in the second sense, in that a $\boldsymbol U\bf\otimes \boldsymbol V$-tensor-valued operator is defined as an $(n\times m)$-tuple of operators $\hat w_{jk}:\mathcal H \to\mathcal H$, with the understanding that if we have $U$-vector and $V$-vector operator tuples $\hat u_j$ and $\hat v_k$ we can form their tensor product (where order now matters) as $\hat u_j \hat v_k$.
Of course, now that we've done this, we need to walk back some distance, since we're just trying to talk about $U$-vector and $U\otimes V$-tensor operators, and those spaces don't come equipped with canonical coordinate maps $i_U$ and so on. Thus, we also need to demand that if we change our choices of coordinate maps, then the operator tuples will change to the same linear combinations that they would if they were plain coordinates instead of operators. This is what the usual requirement that 'tensor operators need to transform as tensors' means.
(I should also mention that the traditional treatment is a bit more obtuse than it strictly needs to be. This great answer shows that you can define $U$-vector operators to be simply linear operators $\hat u:\mathcal H \to \mathcal H \otimes U$, and it is easy to extend that formalism to $U\otimes V$-tensor operators defined as linear operators $\hat w:\mathcal H \to \mathcal H \otimes U\otimes V$.)
As you can see, then, quantum mechanics is perfectly happy to talk about tensor operators occupying any abstract tensor product you wish to pull from classical mechanics. However, when one is actually out and about doing quantum mechanics, one usually doesn't care about arbitrary tensor products - we specifically care about tensor products of $\mathbb R^3$ with itself, and we care about how those tensor products interact with the additional structure carried by our vector spaces, including in particular its inner-product structure and with it the symmetry group of that structure, the rotation group.
This is where the representation theory comes in, and it does so in the most natural way. Let me phrase this in a general setting:
- Let $U$ and $V$ be vector spaces over $F$, $G$ be a group, and $R:G\to \mathrm{GL}(U)$ and $S:G\to \mathrm{GL}(V)$ be representations of $G$. There is then a natural representation $T:G\to \mathrm{GL}(U\otimes V)$ in the tensor product space, which is uniquely specified by its action on tensor-product vectors,
$$T(g)(u\otimes v) = R(g)(u)\otimes S(g)(v).$$
Normally, of course, we have $U=V=\mathbb R^3$ and $G=\mathrm{SO}(3)$. Typically, even if the single factor representations $R$ and $S$ are irreducible, the tensor-product representation will not be irreducible. When we speak of tensors being reducible or irreducible, we're talking about the word in the representation-theoretic way: a reducible tensor lives in a tensor-product space that carries a reducible representation of $\mathrm{SO}(3)$, while an irreducible tensor lives in a restricted subspace such which carries an irreducible representation.
This is also where spherical tensors come in: they are simply a convenient basis for the restricted subspaces that carry irreducible representations. When they are notated as $T_q^{(k)}$, it normally means that you have a tensor-valued operator (i.e. living in some bigger tensor product space, whose size and number of factors is not that relevant) that's been restricted to a subspace $\mathrm{span} \mathopen{}\left(\{T_{-k}^{(k)}, T_{-k+1}^{(k)},\ldots, T_{k-1}^{(k)}, T_{k}^{(k)}\}\right)\mathclose{}$ that carries the $k$ representation of $\mathrm{SO}(3)$.
In this sense, your final question (why are all tensors in QM spherical tensors?) can be rephrased as follows: why are all tensors in QM separated into irreducible representations of the rotation group? The answer, of course, is that they aren't, and there's nothing intrinsic about QM that requires tensors to live in irreducible representations ─ it's just that they're more useful so we use them more often.
Best Answer
The result you described says that the projection of antisymmetric rank 2 $SO(4)$ tensors onto self-dual and anti-self-dual subspaces commutes with the action of $SO(4)$. This just implies that the space of antisymmetric rank 2 tensors of $SO(4)$ is reducible.
To show that the self-dual and anti-self-dual subspaces are themselves irreducible, think of these rank 2 tensors as separate Lie algebras, and look for isomorphisms with $\mathfrak{su}(2)$ (recall that $\wedge^2V $ can be identified with $\mathfrak{so}(4)$, which decomposes into a direct sum).