Let $u$ and $v$ be vectors. Suppose the matrix $M$ is an isometry (preserves distances), takes $u$ to the $y$-axis and $v$ into the $x{-}y$ plane. Then it must take $u\times v$ to the $z$-axis and $u\times(u\times v)$ to the $x$-axis. That is
$$
M\begin{bmatrix}u&u\times v&u\times(u\times v)\end{bmatrix}=\begin{bmatrix}0&0&|u||u\times v|\\|u|&0&0\\0&|u\times v|&0\end{bmatrix}\tag{1}
$$
So we get
$$
M=\begin{bmatrix}0&0&|u||u\times v|\\|u|&0&0\\0&|u\times v|&0\end{bmatrix}\begin{bmatrix}u&u\times v&u\times(u\times v)\end{bmatrix}^{-1}\tag{2}
$$
Thus, $M$ takes three orthogonal vectors to three orthogonal vectors, preserving their lengths, so $M$ is an isometry.
If we let $u=b-a$ and $v=c-a$ and compute $M$ as in $(2)$, then $M(b-a)$ lies on the $y$-axis, and $M(c-a)$ lies in the $x{-}y$ plane. The only thing we can't do is map $a$ to the origin. A linear map sends $0$ to $0$, and since $M$ preserves distances $|Ma|=|M(a-0)|=|a-0|=|a|$. So unless $a=0$, $Ma\not=0$. To map $a$ to the origin, we need to apply a translation as well.
Compute $M$ as above, then the affine-linear map $A(x)=M(x-a)$ satisfies the requirements.
I would guess that you're supposed to use "homogeneous coordinates", and that the "3D" matrix you're supposed to compute is the matrix of a projective transformation. When we use homogeneous coordinates, a coordinate vector for a point in a plane has three components rather than two. This may seem unintuitive or unappealing, but there are advantages to homogeneous coordinates: 1) "points at infinity" can be assigned coordinates as easily as ordinary points ; 2) every projective transformation is given by multiplication by a $3 \times 3$ matrix. (There are other advantages too; mathematicians have discovered that homogeneous coordinates are a natural and elegant way to assign coordinates to points in a projective plane.)
Suppose we have a projective transformation that is represented by a matrix $H \in \mathbb R^{3 \times 3}$, so that point with coordinate vector $x$ maps to a point with coordinate vector $Hx$. If we have four points $x_1,\ldots,x_4$ in "general position" and we know these four points map to $y_1,\ldots, y_4$, then we can use this information to compute $H$.
We cannot simply say that $y_i = H x_i \, \,$, because one quirky thing about homogeneous coordinates is that if $y$ is a coordinate vector for a point in a projective plane, then any nonzero scalar multiple of $y$ is also a coordinate vector for the same point. This might seem weird but it's something we must get used to.
The most we can say is that $y_i$ is a (nonzero) scalar multiple of $H x_i$. An equivalent way to say the same thing is
\begin{equation}
y_i \times H x_i = 0
\end{equation}
where $\times$ denotes the cross product.
Notice that this equation is linear in the components of $H$ ! For each $i$, we have three linear equations for the components of $H$. However, it turns out that one of those three equations is redundant, so each $i$ really only gives us two equations. All together, we have eight linear equations.
$H$ is found by finding a nonzero solution of this system of eight equations. Note that this only determines $H$ up to a scalar multiple. However, that's to be expected, because in homogeneous coordinates, any nonzero scalar multiple of $H$ represents the same projective transformation as $H$.
Best Answer
all you need to do is measure how the vectors vec3(1,0,0) vec3 (0,1,0) vec3(0,0,1) get changed by any matrix.
vec3(1,0,0)'s transformation is equal to the first column of the matrix vec3(0,1,0)'s transformation is equal to the second column of the matrix vec3(0,0,1)'s transformation is equal to the second column of the matrix (only if it has 3 dimensions) vec4(0,0,0,1)'s transformation is equal to the second column of the matrix (only if it has 4 dimensions),...
yes, learn how matrices function on basic core level:
in a 2d to 2d matrix transformation, 2 matrix colums are equal to what the 2 base-vectors vec2(1,0) and vec2(0,1) are changed into by the matrix. any scalar of that is also transformed linearily scaled to that, as a matrix transformation is a linear transformation.
in a 3d to 3d matrix transformation, same goes for 3 base-vectors vec3(1,0,0) vec3 (0,1,0) vec3(0,0,1) being changed by 3 matrix columns.
in a 3d to 2d matrix, 3 columns of 2 lines tell how 3 3d basis vectors as in the above are scamed in 3d space, as a sum of three 2d vectors (that get scaled by the matrix).