It sounds like what you're asking is: how do you construct a representation of SU(2) in terms of 3x3 matrices on a real 3-dimensional vector space? (This representation is also known as the "spin-1" representation, as it's used to describe the spin of spin-1 particles.)
The H you mention, which appears to be some kind of Hamiltonian, is irrelevant to the above question. I assume it is part of a longer homework question which isn't described fully here, so I'll ignore it.
As your teacher mentions, a simple way to construct $S_x$, $S_y$, and $S_z$ is to start with the raising and lowering operators $S_+$ and $S_-$.
If you work in the $S_z$ basis, then you know what the action of $S_z$ is on each of the 3 $S_z$ eigenstates:
$S_z \begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix} = \hbar \begin{bmatrix}1 \\ 0 \\ 0\end{bmatrix}$
$S_z \begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix} = 0$
$S_z \begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix} = -\hbar \begin{bmatrix}0 \\ 0 \\ 1\end{bmatrix}$
So the 3x3 matrix form of $S_z$ in this basis must be:
$S_z = \hbar\begin{bmatrix}1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & -1 \end{bmatrix}$
And you also know the actions of the raising and lowering operators on these $S_z$ eigenstates (up to an undetermined constant):
$S_+$ $\begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix} = c\begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix}$
$S_+$ $\begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix} = c\begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix}$
$S_+$ $\begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix} = 0$
$S_-$ $\begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix} = 0$
$S_-$ $\begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix} = c\begin{bmatrix}0 \\ 0 \\ 1 \end{bmatrix}$
$S_-$ $\begin{bmatrix}1 \\ 0 \\ 0 \end{bmatrix} = c\begin{bmatrix}0 \\ 1 \\ 0 \end{bmatrix}$
If you take those actions and write them in matrix form, you get:
$S_+$ = c$\begin{bmatrix}0 & 1 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0\end{bmatrix}$
$S_-$ = c$\begin{bmatrix}0 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0\end{bmatrix}$
Then, you can write down $S_x$ and $S_y$ just by taking the right linear combinations of $S_+$ and $S_-$:
$S_x = \frac{1}{2}(S_+ + S_-)$
$S_y = \frac{1}{2i}(S_+ - S_-)$
The only final step required is to determine the constant c. This can be determined by finding the eigenvalues of the $S_x$ and $S_y$ matrices. You want them to be $-\hbar$, $0$, and $\hbar$. You can accomplish this by setting $c = \hbar\sqrt{2}$.
As for the commutation relations $[J_i,J_j] = i\hbar\epsilon_{ijk} J_k$, it just means that:
$[S_x,S_y] = i\hbar S_z$
$[S_y,S_z] = i\hbar S_x$
$[S_z,S_x] = i\hbar S_y$
You can verify these directly using matrix multiplication, for example by showing that $S_x S_y - S_y S_x = i\hbar S_z$
The two representations are unitarily equivalent to each other, except for an overall factor of $i$.
To be clear, I'll write $J$ and $\tilde J$ for the generators in the two different representations. One representation is
$$
J_x = \left(
\begin{matrix} 0&0&0\\ 0&0&-1 \\ 0&1&0\end{matrix}
\right)
\hskip1cm
J_y = \left(
\begin{matrix} 0&0&1\\ 0&0&0 \\ -1&0&0\end{matrix}
\right)
\hskip1cm
J_z = \left(
\begin{matrix} 0&-1&0\\ 1&0&0 \\ 0&0&0\end{matrix}
\right)
$$
and the other is
$$
\tilde J_x = \frac{1}{\sqrt{2}}\left(
\begin{matrix} 0&1&0\\ 1&0&1 \\ 0&1&0\end{matrix}
\right)
\hskip1cm
\tilde J_y = \frac{i}{\sqrt{2}}\left(
\begin{matrix} 0&-1&0\\ 1&0&-1 \\ 0&1&0\end{matrix}
\right)
\hskip1cm
\tilde J_z =
\left(
\begin{matrix} 1&0&0\\ 0&0&0 \\ 0&0&-1\end{matrix}
\right).
$$
The $J$s are anti-hermitian and $\tilde J$s are hermitian. That's just a matter of convention, because we can multiply the $J$s by $i$ to make them hermitian. The unitary matrix
$$
U = \frac{1}{\sqrt{2}}
\left(
\begin{matrix} 1&0&-1\\ i&0&i \\ 0&-\sqrt{2}&0\end{matrix}
\right)
$$
satisfies
$$
i\,J_x U = U\tilde J_x
\hskip2cm
i\,J_y U = U\tilde J_y
\hskip2cm
i\,J_z U = U\tilde J_z,
$$
which proves that the two representations are equivalent except for the overall factor of $i$.
These identities could be written in the form $i\,J=U\tilde J U^{-1}$ instead, but they way I wrote them above makes them easier to check.
Best Answer
The short answer is that $J^2$ is a scalar under all rotations, and the only matrix that has that property is the identity matrix, so you have to have $J^2\propto I$.
I think that the process and physics are a bit easier to understand if we start from the elemental rotation matrices in $3$-d about each axis \begin{align} R_x & = \left[\begin{array}{ccc} 1 & 0 & 0 \\ 0 & \cos\theta & -\sin\theta \\ 0 & \sin\theta & \cos\theta \end{array}\right] & R_y & = \left[\begin{array}{ccc} \cos\theta & 0 & \sin\theta \\ 0 & 1 & 0 \\ -\sin\theta & 0 & \cos\theta \end{array}\right] & R_z & = \left[\begin{array}{ccc} \cos\theta & -\sin\theta & 0\\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1 \end{array}\right]. \end{align} where the pattern of signs is chosen so that a positive rotation gives a rotation around the axis in the sense defined to be positive by the right hand rule. A matrix/operator is said to generate a transformation if, for an infinitesimal transformation, it satisfies $$R(\theta) = I - i \frac{J\theta}{\hbar} + \mathcal{O}(\theta^2),$$ where the transformation is $R$ and the generator is $J$. From that we get that \begin{align} J_x & = \left[\begin{array}{ccc} 0 & 0 & 0 \\ 0 & 0 & -i\hbar \\ 0 & i\hbar & 0 \end{array}\right] & J_y & = \left[\begin{array}{ccc} 0 & 0 & i\hbar \\ 0 & 0 & 0 \\ -i\hbar & 0 & 0 \end{array}\right] & J_z & = \left[\begin{array}{ccc} 0 & -i\hbar & 0 \\ i\hbar & 0 & 0 \\ 0 & 0 & 0 \end{array}\right]. \end{align}
We now look for the eigenvectors and eigenvalues of $J_z$. The defining equation for the eigen-problem is $$J_z \vec{v} = \lambda \vec{v}$$ for some eigenvalue $\lambda$ and its corresponding eigenvector $\vec{v}$. We get that the eigenvalue/eigenvector pairs of $J_z$ are \begin{align} \lambda_{-1} & = -1 \Rightarrow & \vec{v}_{-1} & = \frac{1}{\sqrt{2}}\left[\begin{array}{c} i \\ 1 \\ 0 \end{array}\right], \\ \lambda_0 & = 0 \Rightarrow & \vec{v}_0 & = \left[\begin{array}{c} 0 \\ 0 \\ i \end{array}\right],\ \mathrm{and} \\ \lambda_1 & = 1 \Rightarrow & \vec{v}_1 & = \frac{1}{\sqrt{2}}\left[\begin{array}{c} -i \\ 1 \\ 0 \end{array}\right], \end{align} where I have taken care to normalize the eigenvectors to unit length (i.e. $\vec{v}^* \cdot \vec{v} = 1$), and to set the overall phases of the unit vectors to make the results match certain other conventions.
It is a defining property of the eigenvalue problem for Hermitian matrices that if we define $V \equiv \left[\vec{v}_1,\ \vec{v}_0,\ \vec{v}_{-1}\right]$, the matrix with normalized distinct eigenvectors in the columns, then \begin{align} V & = \left[\begin{array}{ccc} -\frac{i}{\sqrt{2}} & 0 & \frac{i}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} & 0 & \frac{1}{\sqrt{2}} \\ 0 & i & 0 \end{array}\right],\ \mathrm{and} & V^\dagger J_z V &= \hbar \left[\begin{array}{ccc} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & -1 \end{array}\right]. \end{align} Note that $V^\dagger V=I$ and $\operatorname{det}(V)=1$, meaning $V$ is a special unitary matrix. That fact means that $V$ is a transformation from one orthonormal basis to another. If we say that the above matrices are just representations of a more abstract operator in a particular basis, and use a $\rightarrow$ to signify representation, then we can say that in this basis $$J_z\rightarrow \hbar \left[\begin{array}{ccc} 1 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & -1 \end{array}\right].$$ If we examine $J_x$ and $J_y$ in this basis, we get \begin{align} J_x &\rightarrow \frac{\hbar}{\sqrt{2}} \left[\begin{array}{ccc} 0 & 1 & 0 \\ 1 & 0 & 1 \\ 0 & 1 & 0 \end{array}\right],\ \mathrm{and} & J_y \rightarrow i\frac{\hbar}{\sqrt{2}} \left[\begin{array}{ccc} 0 & -1 & 0 \\ 1 & 0 & -1 \\ 0 & 1 & 0 \end{array}\right]. \end{align}
Now you can evaluate $J^2 = J_x^2 + J_y^2 + J_z^2$ explicitly.
Notice how our original basis was in terms of the three states that are invariant under rotations about their respective axes, the $m_x=0$, $m_y=0$, and $m_z=0$ states, respectively.
Note also that the choice of phases made won't effect $J^2$. That was done so that the $J_x$ and $J_y$ constructed will match those generated by following the development in Wikipedia's ladder operator article.
You can make a more general construction if you look at the commutation relations among $J_x$, $J_y$ and $J_z$ as defined above, and rename the $x$ through $z$ axes as $1$ through $3$ to get that $$[J_i,\, J_j] = i\hbar \sum_{k=1}^3 \epsilon_{ijk} J_k, \tag1$$ with $\epsilon_{ijk}$ the Levi-Civita symbol. We then say that any collection of three Hermitian matrices that satisfies the commutation relations in (1) are generators of the symmetry transformation we call rotations in physics, in some particular representation/basis. The next step is to prove that $$[J^2,\, J_i] = 0,$$ which shows that the $J^2$ and $J_z$ can be simultaneously digaonalized. I have forgotten the next step in the process, but this more general approach works for all possible values of orbital angular momentum and spin, making it much more powerful.