I'm trying to grasp what's going on at a cusp geometrically. For instance, $y^2=x^3$ is not differentiable at the origin. In $y$ things appear fine: differentiate $y = \pm x^{3/2}$ and we get $y'=0$ regardless of approach from the top or bottom branch. But in $x$, we get the slope $x'$ is $\pm\infty$ depending on approach.
But then again, for the circle $y^2+x^2=1$, we get $y' = \pm\infty$ at (1,0) depending on approach from the top or bottom. Yet it's clear that $dy/dx$ exists since, among many reasons, the Jacobian matrix has full rank everywhere. On the surface the two cases seem the same; at some point the derivative wrt to the other variable is 0, and flipping the variable we get two different slopes for the vertical tangent. Either there is some subtle difference or I'm missing something glaringly obvious. I think given any parameterization of the cuspidal cubic that tangent vectors at the origin only point in the negative x direction, while we can get opposite pointing tangent vectors everywhere on the circle depending on our parameterizations, but I'm not sure how to interpret this formally.
Best Answer
We need to clearly indicate which part of calculus we are discussing here. If we are talking about differentiability of a function of single variable $$y=f(x)$$ then the question is ill-posed since $y^2=x^3$ is not a function and the standard rules of differentiation do not apply (i.e. you cannot derive using $y=\pm x^{3/2}$ since it's a couple of functions)
If we are talking here about differentiability of a curve (or more generally about a manifold) then a tangent line (or a tangent space, more generally) does not exist. It can be shown rigorously.
If you would like to gain "feeling" of what it means geometrically, imagine that you have a magnifying glass that can zoom in to any scale you want. No matter how close you zoom in for the curve you mention at $(0,0)$, it will never remind you of a line. Look at the circle instead: if at any point you zoom in super closely, you will not be able to tell if you are looking at a line or a piece of a circle.
This ability or rather inability to distinct between a piece of curve or a straight line is what makes a curve differentiable at a point. If a curve can be approximated with an arbitrary precision within a certain neighborhood (however small) of this point with a straight line, then the curve is differentiable (smooth, or a manifold)
This definition can be made precise using the "$\epsilon-\delta$" language.
To blow your mind, there is so called "subdifferential culculus" that defines derivatives and subdifferentials of objects like $y^3=x^{3/2}$ at $(0,0)$.