No need to go back to the definition of limits, use some of the properties instead. If $f(x,y)$ is continuous at $(0,0)$, consider $p(x) = (x, 0)$; $p$ is continuous everywhere (perhaps you need to prove that, but that's easy). So: if $f$ is continuous at $(0,0)$, then $g(x) = f \circ p (x)$ is continuous at $0$. Compute $f \circ p$ and show it is not continuous at $0$ for either $a=0$ or $a=1$, again, this is much easier than the original problem. So the conclusion is that the original $f$ is not continuous at $(0,0)$ for either value of $a$.
I gave the argument in relatively formal terms; but what I am doing is this: If $f$ is continuous as a function of two variables, it should in particular be continuous when I fix $y=0$ - and it just isn't.
One of the ways in which a second-partial-derivative test can fail us is where the Hessian determinant is zero not at an isolated point, but on a line passing through that point. A "slice" through the surface for the function $ \ f(x,y) \ = \ y^2· \left(\sin x - \frac{x}{2}\right) \ $ at, say, $ \ y \ = \ \pm 1 \ $ , looks like this:
Except for $ \ y \ = \ 0 \ $ , where $ \ f(x,0) \ = \ 0 \ $ , all parallel slices are a "vertically-stretched" version of this slice. The consequence of this is that each slice at a specific value of $ \ x \ $ is a parabola $ \ z \ = \ ky^2 \ $ , except for those values of $ \ x \ $ at which $ \ \left(\sin x - \frac{x}{2}\right) \ = \ 0 \ \ . $
So we have "flat" lines in the surface of the function for $ \ f(x,0) \ = \ 0 \ $ , $ \ f(0,y) \ = \ 0 \ $ , $ \ f(\approx 1.895 , y) \ = \ 0 \ $ , and $ \ f(\approx -1.895 , y) \ = \ 0 \ $ . If instead of evaluating the Hessian determinant at each individual critical point, we look at the determinant function
$$ \mathbf{H_f} \ \ = \ \ -2y^2 \ · \ \left[ \ \sin x · \left( \sin x \ - \ \frac{x}{2} \right) \ + \ 2 · \left( \cos x \ - \ \frac{1}{2} \right)^2 \ \right] \ \ , $$
we see that every point along the $ \ x-$ axis on the surface gives a value of zero, so those will be the points that Folland calls "degenerate" (I happen to have a copy of that text and I believe this is the sort of situation he is referring to).
So the short answer to your main question is that all of your critical points are degenerate, but that is not the same as saying that they are all inflection points. It is still possible to characterize them by looking at the behavior of the function in their neighborhoods.
As for recognizing the types of the critical points you've identified, things are a bit complicated then. The graph will be helpful for this, since the expression in brackets in the determinant function is not straightforward to analyze.
Since the slices parallel to the $ \ y-$ axis are parabolic, they "open upward" where $ \left(\sin x - \frac{x}{2}\right) > 0 \ $ and "downward" when this expression is $ < 0 \ $ .
For the critical points $ \ \left(\frac{5\pi}{3} + 2\pi n \ , \ 0 \right) \ , \ n \ge 0 \ , $ the surface is "concave up" in the $ \ x-$ direction and "down" in the $ \ y-$ direction, making these saddle points (negative curvature); for $ \ \left(\frac{\pi}{3} + 2\pi n \ , \ 0 \right) \ , \ n \ge 1 \ , $ the surface is "concave down" in both directions (positive curvature), so these are local maxima.
Since our function has odd symmetry about the $ \ yz-$ plane, the direction of local extrema is reversed, telling us that $ \ \left(-\frac{5\pi}{3} + 2\pi n \ , \ 0 \right) \ , \ n \le 0 \ , $ are saddle points and $ \ \left(-\frac{\pi}{3} + 2\pi n \ , \ 0 \right) \ , \ n \le -1 \ , $ are local minima.
The points $ \ \left( \frac{\pi}{3} \ , \ 0 \right) \ $ has $ \left(\sin x - \frac{x}{2}\right) > 0 \ $ , so it is "concave upward" in the $ \ y-$ direction, but it lies at a local maximum of that function, so it is "concave downward" in the $ \ x-$ direction; hence it is a saddle point. The directions of concavity are both reversed for $ \ \left( -\frac{\pi}{3} \ , \ 0 \right) \ , $ so it is likewise a saddle point.
The two points $ \ (\approx 1.895 \ , \ 0) \ $ and $ \ (\approx -1.895 \ , \ 0) \ $ are zeroes of $ \ f(x,y) \ $ and are sitting on "declining slopes" in the $ \ x-$ direction but on a "flat line" in the $ \ y-$ direction; I don't know what the correct term for such a critical point is.
Finally, near the origin, the points on $ \ (x \ , \ 0) \ $ in the neighborhood have opposed concavities in the perpendicular directions, so those are all saddle points. The orientation of these saddle points reverses at the origin, so perhaps the origin is also characterized as a sort of saddle point. (The term seems a bit inadequate to describe the behavior of the surface there.)
Best Answer
Your computations are correct. On the other hand, the way you express the answer is not very clear. For instance: what is that $n$?
I would say that the set of critical points is$$\bigl\{(0,0)\bigr\}\cup\left\{\left(\frac1y\left(n\pi+\frac\pi4\right),y\right)\,\middle|\,n\in\Bbb Z\wedge y\in\Bbb R\setminus\{0\}\right\}.$$