Simply reverse the order in which you multiply your matrices:
$$R_{41}=R(\psi)R(\theta)R(\varphi)$$
To understand the why: $R(\varphi)$ has columns which contain the basis of the fouth system, expressed in coordinaes of the third. So when you multiply a coordinate vector in the fourth system from the right, you obtain the representation in the third. By a similar consideration, $R(\theta)$ will turn that representation into that for the second and eventually via $R(\psi)$ into the original coordinate system.
As to your various formulas about transposition: the general rules are
\begin{align*}\left(A^T\right)^T &= A & (AB)^T &= B^TA^T\end{align*}
So as you can see, if you have the transpose of a product, and change that into a product of transposed factors, then you have to reverse the order. This is the reason why the formula you gave for $R_{41}$ is essentially just a reversal of factors. It also verifies your operations on row vectors:
$$\vec w_2 = \vec v_2^T = \left(R_{41}\vec v_1\right)^T = \vec v_1^TR_{41}^T = \vec w_1R_{41}^T $$
So within $S^3(\sqrt{2})$, the singular matrices form the torus parametrized by
$$ \begin{cases} x=\cos\theta \\ w=\sin\theta \\ y=\cos\phi \\ z=\sin\phi \end{cases} $$
Based on your picture, let $\theta$ be toroidal coordinates, and $\phi$ poloidal coordinates. (Supposedly that's opposite of the usual convention but I'm already invested in it this way.)
The rotations in $SO(2)$ and reflections in $O(2)\setminus SO(2)$ are respectively given by
$$ \begin{cases} x=\sqrt{2}\cos\theta \\ w=\sqrt{2}\sin\theta \\ y=0 \\ z=0 \end{cases}
\qquad
\begin{cases} x=0 \\ w=0 \\ y=\sqrt{2}\cos\phi \\ z=\sqrt{2}\sin\phi \end{cases} $$
It ought to be obvious, topologically, how a torus is made up of a bunch of disks. But let's get specific.
For solid torus with $SO(2)$ running through it, the disk "centered" at the $\theta$-point on $SO(2)$ is
$$ \begin{cases}
x=\sqrt{2-y^2-z^2}\cos\theta \\
w=\sqrt{2-y^2-z^2}\sin\theta \\
1<y^2+z^2\le 2
\end{cases} $$
The "center" (on $SO(2)$) occurs when $y^2+z^2=2$. Set $y^2+z^2=2\cos^2\rho$ with $\rho\in[0,\frac{\pi}{4})$. Then we can parametrize the geodesic arc from the $\theta$-point on $SO(2)$ and the $(\theta,\phi)$-point on the torus via
$$ \begin{cases}
x=\sqrt{2}\cos\rho \cos\theta \\
w=\sqrt{2}\cos\rho \sin\theta \\
y=\sqrt{2}\sin\rho \cos\phi \\
z=\sqrt{2}\sin\rho \sin\phi
\end{cases} $$
For example when $\cos\theta=0$ these are spherical coordinates for a spherical cap of "diameter" $\frac{\pi}{2}$ (angle between antipodal points) in the $wyz$-plane. Letting $\rho$ run wild parametrizes a great $2$-sphere with poles
$$ \pm\begin{bmatrix} \sqrt{2}\cos\theta \\ \sqrt{2}\sin\theta \\ 0 \\ 0 \end{bmatrix},
\quad \pm\begin{bmatrix} 0 \\ 0 \\ \sqrt{2} \\ 0 \end{bmatrix},
\quad \pm\begin{bmatrix} 0 \\ 0 \\ 0 \\ \sqrt{2} \end{bmatrix}. $$
If we differentiate this parametrization at the $\theta$-point on $SO(2)$, where $\cos\rho=1$ the derivative is $\cos'\rho=0$ so the $xw$-components are $0$. On the other hand, leaving $\cos\rho=1$ and differentiating with respect to $\theta$ leaves the $yz$-components $0$. Thus, at the $\theta$-point on $SO(2)$, the tangent vectors of $SO(2)$ and of the $2$-spherical cap are orthogonal.
I will let you do the same thing for the other solid torus (which, under stereographic projection, is "inside out"). In your $\mathbb{R}^3$ visualization (again, under stereographic projection), if $O(2)\setminus SO(2)$ represents a straight line (which becomes a circle with the "point at infinity"), the great $2$-spheres containing these new caps will be all possible spheres through the circle representing $SO(2)$, plus the plane through said circle (which becomes a $2$-sphere with the "point at infinity").
Moreover, these $2$-spherical caps have orthogonally intersecting boundary circles. That is, in the original parametrization of the torus, let $\theta$ be fixed and differentiate with respect to $\phi$ for one tangent vector, and let $\phi$ be fixed and differentiate with respect to $\theta$ for another, and these tangent vectors will be orthogonal.
Best Answer
Yes, it is closed under composition. That is not hard to check directly. This group acts exactly as you have described on the plane $z=1$.
This comes from the standard way to express the affine group as a linear group. In general, the $n$-dimensional affine group over $k$ is $G=\operatorname{GL_n}(k)\ltimes k^n$ and it isomorphic to the linear group of matrices of the form $\begin{pmatrix}M & v\\ 0\ldots 0 & 1\end{pmatrix}$, where $M\in \operatorname{GL_n}(k)$ and $v\in k^n$. The standard action of the affine group on $k^n$ is the same as the action of the linear group on the plane $x_{n+1}=1$.