I tried to sketch a simple harmonic motion below
At the top (a) the mass is moving towards the left, in this case $v<0$. Eventually it will reach the minimum value of $x$, stop, and start moving toward the right. It will get to position (b) where the location is the same as in the previous case, but the velocity has the oposite sign $v >0$. That is, for the same location $x$, the velocity could be have either positive or negative sign. As the mass keeps moving, it will get to the other side of $x = 0$ (c), in this case both $x>0$ and $v>0$. Once again it will stop at its maximum distance from the origin, change its velocity and start moving towards the left, reaching again the same position as before, but with a different sign in the velocity. I think this should answer your first question
In general for a force of the form $$F = -m\omega^2 x$$ the velocity is
$$
v^2 = \pm \left[2(E - \omega^2 x^2) \right]^{1/2}
$$
for some integration constant $E$. For another point of view, the solution to $F = m\ddot{x}$ is
$$
x(t) = A\cos(\omega t + \phi) ~ v(t) = -A\omega \sin(\omega t + \phi)
$$
where the constants $A$ and $\phi$ are integration constants. As a matter of fact $E = \omega^2 A^2/2$. Due to the oscillatory nature of the $\sin$ and $\cos$ functions, the sign of the position and velocity changes periodically.
This should answer your second question as well, if you write, for exmaple, $v = +\sqrt{(\cdots)}$, you must be certain in which part of the motion cycle. In general you should use $\pm$
You're right to be suspicious; the question is just all sorts of wrong. The question says
A particle of mass $m$ starts from rest at time $t=0$ and is moved along the $x$-axis with constant acceleration $a$ from $x=0$... blablablablablablabla
Once I read this first part of the sentence, without even thinking, I immediately translate this into an initial value problem for a second order ODE:
\begin{align}
\begin{cases}
x''(\cdot)&=a\\
x'(0)&=0\\
x(0)&=0
\end{cases}
\end{align}
The solution is immediately given as $x(t)=\frac{1}{2}at^2$, and so it takes a total of $t_h=\sqrt{\frac{2h}{a}}$ units of time to travel a displacement of $h$. It doesn't matter how many forces or what kinds of forces act on the particle. If the particle has a constant acceleration of $a$, then this is the trajectory of the particle. That's it.
Next, the net work done along the path from $x=0$ to $x=h$ is simply the change in kinetic energy (as you've correctly identified), and in this case, it is
\begin{align}
\frac{m}{2}[x'(t_h)]^2-\frac{m}{2}[x'(0)]^2=\frac{m}{2}[at_h]^2-\frac{m}{2}(0)^2=mah.
\end{align}
If you wish to calculate only the work done by the force $F$, then it is
\begin{align}
\text{work done by $F$ on the particle}&=\int_0^{t_h}F(t)\cdot x'(t)\,dt=\int_0^{t_h}t^2\cdot at\,dt=\frac{at_h^4}{4},
\end{align}
which again is not the "intended" answer. In the current form of how the question is phrased, you're right to introduce a new force $F_1$, but even the work done by $F_1$ is not equal to the "intended" answer. i.e none of the forces $F, F_1, F_1-F$ have work equal to the "intended" answer. This just goes to show how poorly the question is worded.
If one wants to obtain the "intended" answer, here's how the question should have been phrased:
Consider a particle of mass $m$ which undergoes motion along the $x$-axis; let $x(t)$ denote the position of the particle at time $t$. Suppose that the particle is under the influence of a force $F$, such that $F(x(t))=kt^2$ for some constant $k>0$ (to ensure that $F$ has the right units) and that the particle starts off at rest at the origin. Find the work done by the force $F$ on the particle as it is displaced from $0$ to some final displacement $h$.
I'm pretty sure here the answer works out to be $\frac{4}{\sqrt{3}}\sqrt{mh^3k}$ (the same as what you have, up to the factor of $k$).
Best Answer
$F_x=-2x+2y$, $F_y=2x-y^2$
$W=\int\vec{F}\cdot d\vec{x}=(-2x+2y)dx+(2x-y^2)dy=-x^2+2xy-y^3/3|_{(2,3)}^{(1,2)}=?$
Alternatively, $\int F_x dx=-x^2+2xy+g(y)$.
$\frac{d}{dy}\int F_xdx=2x+g'=2x-y^2$
So $g(y)=-y^3/3+C$.
Let potential $V(x,y)= -x^2+2xy-y^3/3$ evaluate at the endpoinds then take the difference, get the same answer.