It may be easier to treat the direction of the plane as along one axis, like the $x$ axis:
$$\vec{v_0} = 400 \hat{x}.$$
Then the new velocity is
$$\vec{v} = 475.3 (\cos (-2.82^{\circ}) \hat{x} + \sin (-2.82^{\circ}) \hat{y}) \approx 474.72 \hat{x} - 23.38 \hat{y}.$$
So the speed of the wind is
$$\sqrt{74.72^2 + (-23.38)^2} \approx 78.29 \text{ MPH}$$
and its direction is
$$\arctan(-23.38/74.72) + 30^{\circ} \approx -17.37^{\circ} + 30^{\circ} = 12.63^{\circ}.$$
A diagram often helps with problems like these.
When the question says the aircraft is flying due east at $400$ km/hr,
I suppose it means the aircraft itself is pointing due east and
its speed relative to the air mass around it
(what aviators call "true airspeed")
is $400$ km/hr.
This gives us a vector represented by the blue arrow in the figure below.
Often when we see "northwest wind" it means the air mass is coming from
the northwest and moving toward the southeast.
But since the problem statement says "headwind" it seems reasonable to
suppose it means the air mass is moving toward the northwest,
since the other direction would be a tailwind for this aircraft.
The vector for this is represented by the purple arrow in the figure below.
![enter image description here](https://i.stack.imgur.com/Fhkny.png)
The resulting velocity of the aircraft relative to the ground is the vector shown as a red arrow.
As you already determined (by interpreting "headwind" the same way I did),
the direction of the ground velocity is slightly north of east.
The groundspeed is the magnitude of the ground velocity, that is,
the length of the red arrow.
That should make it clear which formula to use.
(Apply the Pythagorean Theorem if there is any remaining doubt.)
By the way,
using $\vec\imath$ as the unit east vector, I would have set $\vec\jmath$
to the unit north vector, so I would have written the wind vector
as $-35.4\vec\imath + 35.4\vec\jmath$ rather than
$35.4\vec\imath + 35.4\vec\jmath$ or even $-(35.4\vec\imath + 35.4\vec\jmath)$.
But in this particular question, as it happens,
the choice of $35.4\vec\jmath$ or $-35.4\vec\jmath$
doesn't actually change the answer.
In another case it might, which is why I mention it.
Best Answer
The velocity of the plane relative to the wind is $$_P\underline{v}_W=\underline{v}_P-\underline{v}_W$$
Therefore, $$\frac{900}{\sqrt{2}}\left(\begin{matrix}1\\1\end{matrix}\right)=\underline{v}_P-\left(\begin{matrix}60\\0\end{matrix}\right)$$
So now you can find the velocity of the plane $\underline{v}_P$.Then you can find the direction and magnitude.
I hope this helps.