$V$ is a two-dimensional subspace of $\mathbb R^3$, so the matrix of the projection $\pi_v:V\to V$, where $v\in V$, will be $2\times2$, not $3\times3$. There are a few ways to approach this problem, several of which I’ll illustrate below.
Method 1: The matrix of $\pi_v$ relative to the given basis will have as its columns the images of the two basis vectors expressed relative to the basis. So, start as you did by computing the image of the two basis vectors under $\pi_v$ relative to the standard basis: $${(1,1,-1)^T\cdot v\over v\cdot v}v=\left(-\frac13,\frac23,-\frac13\right)^T \\
{(5,-4,1)^T\cdot v\over v\cdot v}v = \left(\frac73,-\frac{14}3,\frac73\right)^T.$$ We now need to find the coordinates of the vectors relative to the given basis, i.e., express them as linear combinations of the basis vectors. A way to do this is to set up an augmented matrix and then row-reduce: $$\left(\begin{array}{rr|rr}1&5&-\frac13&\frac73 \\ 1&-4&\frac23&-\frac{14}3 \\ -1&1&-\frac13&\frac73\end{array}\right)\to\left(\begin{array}{rr|rr} 1&0&\frac29&-\frac{14}9 \\ 0&1&-\frac19&\frac79 \\ 0&0&0&0 \end{array}\right).$$ The matrix we seek is the upper-right $2\times 2$ submatrix, i.e., $$\pmatrix{\frac29&-\frac{14}9\\-\frac19&\frac79}.$$
Method 2: Find the matrix of orthogonal projection onto $v$ in $\mathbb R^3$, then restrict it to $V$. First, we find the matrix relative to the standard basis: $${vv^T\over v^Tv}=\pmatrix{\frac16&-\frac13&\frac16\\-\frac13&\frac23&-\frac13\\\frac16&-\frac13&\frac16}.$$ Note that each column of this matrix is a multiple of $v$, as one would expect. Now, we need to apply a change of basis to this matrix. For this, we need a third basis vector not in $V$. A convenient choice is $(1,2,3)^T$ because we know from the definition of $V$ that it is orthogonal to $V$ and hence an element of the kernel of the projection. This gives $$\pmatrix{1&5&1\\1&-4&2\\-1&1&3}^{-1}\pmatrix{\frac16&-\frac13&\frac16\\-\frac13&\frac23&-\frac13\\\frac16&-\frac13&\frac16}\pmatrix{1&5&1\\1&-4&2\\-1&1&3} = \pmatrix{\frac29&-\frac{14}9&0\\-\frac19&\frac79&0\\0&0&0}.$$ The matrix we seek is the upper-left $2\times 2$ submatrix, which agrees with that found by method 1.
Method 3: Relative to a basis of $V$ consisting of an element of the image of $\pi_v$ and an element of its kernel, the matrix of $\pi_v$ is simply $\pmatrix{1&0\\0&0}$. Applying a change of basis to this matrix will produce the matrix we seek.
For an element of $\operatorname{im}\pi_v$, we can obviously take $v$. For an element of the kernel, we can take any non-zero multiple of $v\times n$, where $n=(1,2,3)^T$, which by the definition of $V$ is orthogonal to this subspace. Here, $v\times n=(-8,-2,4)^T$, so we can take $(4,1,-2)^T$ as the second basis vector to keep the arithmetic a bit simpler. For the change of basis, we need either the coordinates of these two vectors relative to the given basis, or vice-versa, which can be found via row-reduction just as in method 1. (Actually, we need both, but inverting a $2\times2$ matrix is quicker than performing a second row-reduction.) Applying this change of basis, we get $$\pmatrix{-\frac23&\frac73\\\frac13&\frac13}\pmatrix{1&0\\0&0}\pmatrix{-\frac13&\frac73\\\frac13&\frac23}=\pmatrix{\frac29&-\frac{14}9\\-\frac19&\frac79}$$ as before. This matrix multiplication is much simpler than that of method 2. First, the matrices are $2\times 2$ instead of $3\times 3$. Furthermore, right-multiplication by $\pmatrix{1&0\\0&0}$ zeroes out the second column of a matrix, while left-multiplication by it zeroes out the second row of the other matrix, so one of the two matrix multiplications involved requires no computation at all.
Best Answer
I'm assuming the projection is from $V$ to $V$, so that we can enter in a $2 \times 2$ matrix.
First, write out the formula for orthogonal projection: $$\mathrm{proj}_v(x) = \frac{v \cdot x}{v \cdot v} v$$ Next, transform the basis vectors: $$\mathrm{proj}_v(-4, 1, 0) = \frac{(-12, 4, -2) \cdot (-4, 1, 0)}{(-12, 4, -2) \cdot (-12, 4, -2)} (-12, 4, -2) = \frac{13}{41} (-12, 4, -2)$$ You can do the other one. Then, write these transformed basis vectors as coordinates of the basis. We can compute the coordinates of the first transformed vector by row-reducing the augmented matrix (the third column is augmented): $$\left(\begin{matrix}-4 & -2 & -\frac{156}{41} \\ 1 & 0 & \frac{52}{41} \\ 0 & 1 & -\frac{26}{41} \end{matrix}\right).$$ This row reduces to: $$\left(\begin{matrix}1 & 0 & \frac{52}{41} \\ 0 & 1 & -\frac{26}{41} \\ 0 & 0 & 0 \end{matrix}\right).$$ Thus, we have: $$\mathrm{proj}_v(-4, 1, 0) = \frac{13}{41} (-12, 4, -2) = \frac{52}{41}(-4, 1, 0) - \frac{26}{41}(-2, 0, 1).$$ Therefore, the first column vector of the matrix is: $$\left(\begin{matrix}\frac{52}{41} \\ \frac{-26}{41}\end{matrix}\right).$$ Now do the same for the other vector, and you are done!