I am confusing myself about where physical quantities become mathematical objects. Where does one end and the other begin?
E.g. displacement $\vec{s}$. A displacement is a physical quantity that can be measured. In contrast to a mathematical vector which is an element of a vector space, which is a set V over a field F (with accompanying properties(+, $\cdot$ ) and axioms). In this case, are the elements of the set V the physical displacements? In that case, the (chosen) basis vectors $\vec{b}_i$ would also displacements and a general displacement can be written as a linear combination of basis vectors as
$$
\vec{s} = s_i\vec{b_i}
$$
The scalars are then elements of the field F which must be the set of real numbers. If the basis vectors are orthonormal (cartesian) then the norm/length of the vector is
$$
|\vec{s}|^2 = s_i s_i
$$
but this is just a number with no units. So the units are hidden in the basis vector $\vec{b_i}$. So maybe we should write this as
$$\vec{b_i} = (b \cdot [1\text{m}])\hat{e_i}$$.
Here $[1\text{m}]$ represents our unit of measure. But now $\hat{e_i}$ is a vector in a different vector space (the set of orientations, i.e. not a displacement). And $(b \cdot [1\text{m}])$ can't be a scalar of the vector space of displacements because we already said that was the set of real numbers So the actual "measurement" $(b \cdot [1\text{m}])$ should maybe be thought of as a map between two vector spaces (orientation and displacement)?
And you can't bake in the units in the field F either because if you scale a vector with a scalar twice (which should be allowed in a vector space) you get the unit squared. For the same reason you can't bake it into the scalar field used for the vector space of orientations.
So when we write
$$
\vec{s} = (0.1 m)\hat{x} + (0.2 m) \hat{y}
$$
maybe we should think about the thing in the parentheses as a map between two vector spaces (displacement and orientation) and the number there is a product of the scalar field F (no units) and the length of the basis vector (with units). (This makes the covariance of the units and the contravariance of the components somewhat intuitive also….maybe).
In summary: confused!
In summary #2: Maybe this boils down to: If you want to think about physical quantities in a the language of mathematical vector spaces, where should you build in the units?
Best Answer
There is an excellent blog post by Terry Tao on the mathematical formulation of dimensionsal analysis.
The easiest answer to your particular question is probably to take his second approach: A dimensionful quantity $Q$ is modeled by an associated 1-dimensional real vector space $V_Q$. A choice of isomorphism of vector spaces $u_Q : \mathbb{R}\to V_Q$ is a choice of units for our quantity, with $u_Q(1)\in V_Q$ being "one" of the chosen unit. Multiplication of two quantities $Q_1,Q_2$ corresponds to the universal map $V_{Q_1}\times V_{Q_2} \to V_{Q_1Q_2} = V_{Q_1}\otimes V_{Q_2}$ into the tensor product.
Now, an $n$-dimensional vector that represents a vector-like quantity with dimension $Q$ is just an element of $V_Q^n$, and the choice of unit $u_Q$ induces a choice of basis in $V_Q^n$ (namely $(u_Q(1),0,\dots,0)$, $(0,u_Q(1),\dots,0)$, $\dots$, $(0,\dots,0,u_Q(1))$). If we call these induced basis vectors $u_i$, then we can expand every $v\in V_Q^n$ as $$ v = \sum_i v_i u_i$$ and the $v_i\in\mathbb{R}$ are the numerical values in the chosen unit for the components of the vector.