The first of these is easy:
How are tensors from QM and tensors from linear algebra widely used in geometry related?
They're the same thing ─ though sometimes QM will choose to look only at some specific subset of tensors, e.g. sets which carry group-theoretic representations of the rotation group.
The apparent conflict arises because there's a wide spectrum of ways to talk about tensors, and you're pulling examples from two complete extremes of that spectrum.
Let's bridge the gap by taking the universal-property tensor product, as in the first understanding,
- Theorem: Given two vector spaces $U$ and $V$ over $F$, there exists a vector space $U\otimes V$ and a bilinear product $\otimes:U\times V\to U\otimes V$ such that for every bilinear $f:U\times V\to F$ there exists a $g:U\otimes V$ such that $f(u,v) = g(u\otimes v)$. Moreover, this vector space and mapping are unique up to a canonical isomorphism.
and provide an instantiation of that abstract tensor-product state:
- Claim. Let $U,V$ be vectors spaces over $F$, and choose coordinate representations $i_U:U\to F^n$ and $i_V:V\to F^m$ for them. Then $F^{n\times m}$ and $\otimes: U\times V\to F^{n\times m}$, defined via
$$\otimes:(u,v)\mapsto \left(i_U(u)_ji_V(v)_k\right)_{j,k=1}^{n,m},$$
are an instantiation of the abstract tensor-product as defined above.
Put another way, this just says that you can do tensor products in a coordinate-wise way, in a manner that is reasonably straight-forward to work out.
Quantum mechanics uses tensors in the second sense, in that a $\boldsymbol U\bf\otimes \boldsymbol V$-tensor-valued operator is defined as an $(n\times m)$-tuple of operators $\hat w_{jk}:\mathcal H \to\mathcal H$, with the understanding that if we have $U$-vector and $V$-vector operator tuples $\hat u_j$ and $\hat v_k$ we can form their tensor product (where order now matters) as $\hat u_j \hat v_k$.
Of course, now that we've done this, we need to walk back some distance, since we're just trying to talk about $U$-vector and $U\otimes V$-tensor operators, and those spaces don't come equipped with canonical coordinate maps $i_U$ and so on. Thus, we also need to demand that if we change our choices of coordinate maps, then the operator tuples will change to the same linear combinations that they would if they were plain coordinates instead of operators. This is what the usual requirement that 'tensor operators need to transform as tensors' means.
(I should also mention that the traditional treatment is a bit more obtuse than it strictly needs to be. This great answer shows that you can define $U$-vector operators to be simply linear operators $\hat u:\mathcal H \to \mathcal H \otimes U$, and it is easy to extend that formalism to $U\otimes V$-tensor operators defined as linear operators $\hat w:\mathcal H \to \mathcal H \otimes U\otimes V$.)
As you can see, then, quantum mechanics is perfectly happy to talk about tensor operators occupying any abstract tensor product you wish to pull from classical mechanics. However, when one is actually out and about doing quantum mechanics, one usually doesn't care about arbitrary tensor products - we specifically care about tensor products of $\mathbb R^3$ with itself, and we care about how those tensor products interact with the additional structure carried by our vector spaces, including in particular its inner-product structure and with it the symmetry group of that structure, the rotation group.
This is where the representation theory comes in, and it does so in the most natural way. Let me phrase this in a general setting:
- Let $U$ and $V$ be vector spaces over $F$, $G$ be a group, and $R:G\to \mathrm{GL}(U)$ and $S:G\to \mathrm{GL}(V)$ be representations of $G$. There is then a natural representation $T:G\to \mathrm{GL}(U\otimes V)$ in the tensor product space, which is uniquely specified by its action on tensor-product vectors,
$$T(g)(u\otimes v) = R(g)(u)\otimes S(g)(v).$$
Normally, of course, we have $U=V=\mathbb R^3$ and $G=\mathrm{SO}(3)$. Typically, even if the single factor representations $R$ and $S$ are irreducible, the tensor-product representation will not be irreducible. When we speak of tensors being reducible or irreducible, we're talking about the word in the representation-theoretic way: a reducible tensor lives in a tensor-product space that carries a reducible representation of $\mathrm{SO}(3)$, while an irreducible tensor lives in a restricted subspace such which carries an irreducible representation.
This is also where spherical tensors come in: they are simply a convenient basis for the restricted subspaces that carry irreducible representations. When they are notated as $T_q^{(k)}$, it normally means that you have a tensor-valued operator (i.e. living in some bigger tensor product space, whose size and number of factors is not that relevant) that's been restricted to a subspace $\mathrm{span} \mathopen{}\left(\{T_{-k}^{(k)}, T_{-k+1}^{(k)},\ldots, T_{k-1}^{(k)}, T_{k}^{(k)}\}\right)\mathclose{}$ that carries the $k$ representation of $\mathrm{SO}(3)$.
In this sense, your final question (why are all tensors in QM spherical tensors?) can be rephrased as follows: why are all tensors in QM separated into irreducible representations of the rotation group? The answer, of course, is that they aren't, and there's nothing intrinsic about QM that requires tensors to live in irreducible representations ─ it's just that they're more useful so we use them more often.
Best Answer
The proper analogous formalization of spinors is not to view them as some sort of different functions from tensors on the same underlying vector space $V$, but instead to expand our idea of the underlying geometry: Where tensors are multilinear functions on vector spaces, tensors with "spinor" and "vector" parts are multilinear functions on super vector spaces $V = V_0\oplus V_1$ where the odd part $V_1$ is a spinorial representation of $\mathrm{Spin}(V_0)$. (nlab calls these spaces super-Minkowski spacetimes).
Via the dual representation, the linear functions on $V_1$ inherit a representation of the spin group. The (multi)linear functions also inherit the super-grading (a linear function that is zero on the odd part is even, and a linear function that is zero on the even part is odd), and purely even such functions are just ordinary tensors, and purely odd functiona are pure spinors.
Note that we still put in the spin representation $V_1$ by hand - the choice is not determined by the base space $V_0$. This is, in some way, not surprising - a notion of "spin" and spinor is genuinely more than just having a vector space: All (pseudo-Riemannian) manifolds (modeled on the vector spaces $\mathbb{R}^n$) have a notion of tensors built on tensor products of the (co)tangent spaces, but not all manifolds have spinors, i.e. the possibility to associate consistently a spinorial representation to every point of the manifold. For simple vector spaces the choice of a notion of spin is not obstructed, but it is still a choice.
That the supergeometric approach is nevertheless the "correct" (or at least a useful) one is seen when we turn to field theory, where one must represent fermionic/spinorial degrees of freedom by anti-commuting variables, and the $\mathbb{Z}/2$-grading of the underlying vector space then allows us to do this simply by declaring that the odd components anti-commute.