Question:
Find the matrix for the transformation which first reflects across the
main diagonal, then projects onto the line $2y+\sqrt{3}x=0$, and then
reflects about the line $\sqrt{3}y=2x$.
Attempt to Question:
Reflection about the line $y=x: T(x,y)=(y,x)$, so the standard matrix for this would be the matrix
$$\begin{pmatrix}0 &1 \\ 1 &0 \end{pmatrix}.$$
However I'm not sure how to deal with equations rather than axis. I assume in the second projection, you can simplify it to $y=\frac{-\sqrt{3}}{2} x$. Can you then separate these into a scalar operation $\frac{-\sqrt{3}}{2}$ and orthogonal operation $y=x$? Even so, I wouldn't know how to go further than this since I only know how to do it among the axis.
Edit: $T_1=\left[\begin{array}{cc}0&1\\1&0\end{array}\right]$ by reflecting across main diagonal
$2y+\sqrt{3}x=0\rightarrow y=\frac{-\sqrt{3}}{2}x$
$T_2=\frac{-\sqrt{3}}{2}\left[\begin{array}{cc}\cos-45&-\sin-45\\\sin-45&\cos-45\end{array}\right]\left[\begin{array}{cc}0&0\\0&1\end{array}\right]\left[\begin{array}{cc}\cos45&-\sin45\\\sin45&\cos45\end{array}\right]$ by rotating $45^\circ$, projecting along the y axis, then rotating $-45^\circ$ (transformations are applied right to left)
$\sqrt{3}y=2x\rightarrow y=\frac{2}{\sqrt{3}}x$
$T_3=\frac{2}{\sqrt{3}}\left[\begin{array}{cc}0&1\\1&0\end{array}\right]$ by separating fraction and then reflecting along $y=x$
Best Answer
Okay, let's start with projections. The projection matrix onto a line $a x + b y = 0$ is a linear transformation expressible by a matrix, mapping the world onto points on that line. A typical point on that line has the form $t [b ;\; -a ]$ for some $t$, as this generates $a(b t) + b (-a t) = 0$. So the unit vector pointing in the direction of that line is $\hat u = [b ;\; -a] / \sqrt{a^2 + b^2}$ and the projection of a vector $\vec v$ is $\operatorname{proj}_{\hat u}~\vec v = \hat u (\hat u \cdot \vec v)$ which we can write as a matrix:$$\operatorname{proj}_{\hat u} = \frac{1}{a^2 + b^2} \begin{bmatrix}b\\-a\end{bmatrix}\begin{bmatrix}b & -a\end{bmatrix} = \frac{1}{a^2 + b^2} \begin{bmatrix}b^2 & -ba\\-ba & a^2\end{bmatrix}.$$So that's the projection matrix.
Once you have projections onto a line, you have reflections about the line. This is because if $\operatorname{proj}_{\hat u} \vec v = \vec v_u $ then we know $\vec v = \vec v_u + \vec c$ for some vector $\vec c$, and then the reflection about that line is just $\vec v_u - \vec c$: you flip the sign of the deviation, but you do not change the projection. Some thinking gives you an explicit construction as: $$\operatorname{flip}_{\hat u} \vec v = \operatorname{proj}_{\hat u} \vec v - (\vec v - \operatorname{proj}_{\hat u} \vec v) = 2 \operatorname{proj}_{\hat u} \vec v - \vec v$$so that the actual matrix which does this is $$\operatorname{flip}_{\hat u} = 2 \operatorname{proj}_{\hat u} - I.$$
Now indeed, if we're flipping around the line $y = x$ then $a = 1, b = -1$, and our projection matrix is $\frac 12 \begin{bmatrix}1&1\\1&1\end{bmatrix}$ and indeed, doubling this and subtracting the identity matrix gives $\begin{bmatrix}0&1\\1&0\end{bmatrix}.$
Your end result is therefore probably not right, since I do not see any factors of $\frac 17,$ which comes naturally from these $(a^2 + b^2)^{-1}$ prefactors.