Yes, that's a fair description of what happens though of course from the ball's perspective it isn't moving - the rest of the universe is moving around it.
However statements like this, while true, give little feel for what's going on. Actually it's extraordinarily difficult to get an intuitive feel for the way spacetime curvature works (or at least I find it so!). The notorious rubber sheet analogy gives a fair description of the effect of spatial curvature, but neglects the curvature in the time coordinate and the time curvature is usually dominant since $dt$ gets multiplied by $c$ in the metric.
The motion of the ball is described by the geodesic equation, but a quick glance at the article I've linked will be enough to persuade you this is not an approach for the non-nerd. I have never seen an intuitive description of how the geodesic equation predicts the motion of a thrown ball.
Suppose you and I start on the equator, a kilometre apart, and we both head exactly due North in a straight line, so we head off in exactly parallel directions:
Now we know that in Euclidean geometry parallel lines remain the same distance apart. But if you and I measure the distance, $d$, between us we find that $d$ starts off at 1km but decreases as we head North and we eventually meet at the North Pole.
Se we have a paradox: we started out parallel but we moved together. The only explanation is that there is some force pulling us together. But we know there is no force really, it's just that we are moving on a curved surface.
This is what happens in general relativity, though as I'm sure you'd expect it's a lot more complicated (principally because time is curved as well). If you see a freely falling body accelerating towards the earth you'd say there must be a force acting between the body and the Earth, and you'd call that force gravity. But the general relativist would say the Earth and the object are both moving along geodesics, i.e. in a straight line, and it's just that because spacetime is curved the two straight lines converge just as we saw for motion on a sphere. There isn't really a force acting even though it looks like a force to us. That's why gravity is sometimes described as a fictitous force.
Actually there is an accelerating object involved in this, and it's you standing on the Earth's surface. How do you know you're accelerating? Well the Earth is pushing at the soles of your shoes and accelerating you upwards. Where there's a force there's an acceleration, so the conclusion must be that the surface of the Earth is accelerating you outwards while the freely falling object you're watching is not accelerating.
If you're interested, twistor59's answer to What is the weight equation through general relativity? explains how to calculate this acceleration, though you may find the maths involved a bit hard going.
Best Answer
There are two answers to this. The simplest is that the curvature is small if you're far from any masses, so motion will be approximately in a straight line at a constant velocity.
The second answer is far more important, but also far harder to explain. Basically it's that we define a straight line as the trajectory followed by a freely moving particle. So for example a thrown object actually follows a straight line - it just looks like a parabola to us.
This may seem like playing with words, but actually if the geometry is non-Euclidean then there is no simple definition of a straight line. We are only used to having an intuitive grasp of what a straight line is because the geometry around the Earth's surface is approximately Euclidean. In a non-Euclidean geometry the principle we use is that if no force is acting on an object then that object will travel in a straight line. So a freely falling object travels in a straight line because, well, freely falling means no force is acting on it. The catch is that the straightness of a line is observer dependant, so different observers will disagree about whether a line is straight or not.
Jan's comment to your question refers to a geodesic, and this is the term for a straight line line in a non-Euclidean geometry. You may be interested to Google for geodesic to learn more, but to get any further than the simplified description I've given above will involve you getting stuck into the maths.