The most important thing to take away is that you don't want to approach physics in terms of a long list of equations, where the goal is to choose the right one for the question you have at hand. (That's why I edited the title to your question, so it no longer simply asks for an equation.) Physics is a method, and the equations are a tool for answering the question. They aren't all that physics is about. I'll try to illustrate this below.
When you hit a golf ball, there are a few different parts involved. First, there's the impact of the club with the ball. Next, there's the path of the ball through the air, which is affected by both gravity and aerodynamics.
Let's do the easiest part first - gravity. We want to know what happens if you take a bunch of golf balls of different sizes and shoot them on the same trajectory. It's a known fact that everything falls at the same speed, ignoring air drag. Heavy things and light things take the same amount of time to fall, so if you shoot them out at the same angle and speed (i.e. the same "velocity vector"), they will take the same amount of time to come back to the ground. (This time depends only on their initial vertical speed because gravity is a force that pulls only vertically.) They also have the same horizontal speed because we set it up that way, so all the balls follow the same path. Thus, the mass of your golf ball doesn't really matter for gravity.
Next we'll look at air drag. Suppose we take a bunch of balls of different sizes and shoot them all into the air at the same speed and angle, as before. They'll start slowing down because they're hitting the air and the air they slam into slows them down. When is this important, though? Maybe the air drag only slows them down by 1% before they land again, or maybe it slows them down by 50%. We should only worry about air drag if it's significant; if it's only 1% we can ignore it.
Air drag works because golf balls have momentum, and as they travel they give some of that momentum to the air. Momentum is mass*velocity, and we can assume that when a golf ball hits some air, the velocity it gives the air is pretty similar to its own velocity. That means that in order to give away a significant amount of its momentum, the golf ball needs to interact with about the same amount of air as its own mass. If the ball weighs 100g, it needs to interact with around 100g of air before it slows down too much.
Golf balls are about the same density as water (I'm not sure if they sink or float, but they don't do either very aggressively), which is about 1000 times as dense as air. Therefore, a golf ball needs to be hit around 1000 times its own distance before air stops it. The size of a golf ball is a few centimeters, so air drag is important for hits of tens or hundreds of meters. 100 meters is a good, long golf stroke. (edit: actually it's not so long, my mistake) We see that air drag is very important for such a strike; in fact it is a limiting factor. Bigger golf balls will be able to go further, since they all go about 1000 times their own length. We could say that in the air-drag limited regime, golf balls go about $l \propto m^{1/3}$. $l$ is the length, how far they go. $m^{1/3}$ is the cube root of the mass, which tells us the size of the golf ball.
Finally, we need to think about striking the golf ball. What happens is that a golf club comes in and smacks the stationary ball. If the ball is very light, it just bounces right off the golf club at something a little less than twice the golf club's speed. (Roughly this is because it runs away from the club at the same speed the club is going, but I don't want to get into too much detail; the factor of 2 isn't extremely important.) All very small balls behave roughly the same way because they aren't heavy enough to affect the club much. This lets us come to our first result: if all our golf balls are very small, the largest ones will go the furthest because they will all be hit the same speed by the club, but the larger ones go further before air drag stops them.
When the mass of the golf ball becomes comparable to the mass of the club, it doesn't go flying off so quickly any more. If the mass of the club and ball were equal, we might expect the club to come to a dead stop and the ball to go flying off at the same speed - all the motion would be transferred to the ball. (This isn't exactly true since it's never a perfect collision, but good enough for now.) Such a ball would shoot away half as fast as a small ball.
When you shoot out half as fast, how much does that hurt your flight? Not much if air drag was just going to stop you anyway, but a lot more otherwise. If you shoot out half as fast, you'll only be in the air half as long, and going at half speed for half the time go one quarter as far.
For golf balls heavier than the club, the club transfers most of its momentum to the ball. This means its velocity is inversely-proportional to its mass, and the distance it goes is $l \propto m^{-2}$
Thus, we're trying to find a balance. Make the golf ball too big and it won't pop off the club as fast. Make it too small and air drag will stop it. We could try to work out a bunch of equations, but it's usually more useful just to start off with an estimate. Let's try to estimate a distance where air drag and slower-initial speed are just becoming important. Balancing these effects should give a rough estimate of the optimal distance.
The ball should go about 1000 times its own diameter
$$l \approx 1000 d$$
This distance is also the square of the initial velocity, as we already worked out. That's not actually a distance though. We need to divide by gravity to get the units right.
$$l \approx \frac{v^2}{g}$$
Where $v \approx v_{club} \frac{m_{club}}{m_{ball}}$, since we might as well make the ball heavier until being heavier is slowing down its initial speed, and this is the approximate relation we found for that regime.
Converting from $d$ to the mass $m_{ball}$ by $d^3 \rho = m$ ($\rho$ is the density), we get
$$l \approx \frac{v_{club}^2 (m_{club}/m_{ball})^2}{g} \approx 1000 \left(\frac{m}{\rho}\right)^{1/3}$$
which gives
$$ m_{ball} \approx \left(\frac{m_{club}^2 \rho v_{club}^2}{1000 g}\right)^{3/7}$$
We've already approximated the density as roughly that of water, $\rho \approx 1 g/cm^3$, and $g = 1000 cm/s^2$ The velocity of the club shouldn't be wildly different from the faster speeds in ball sports, around 100 mph or 50 m/s or 5000 cm/s. Club heads are around 200g. Plugging these in gives
$$m_{ball} \approx 370g$$
about twice as much as the golf club itself. This would make golf rather awkward, since the club would bounce off the ball. We get a distance of around 80 meters out of this, suggesting that although a golf ball heavier than the 45g actual ball wouldn't really fly all that much further. The effects of air drag and slower takeoff speed fight each other, so that a decent-sized range of golf balls do about the same thing. The 45g ball plays much better without sacrificing too much range, but the weight of golf balls is regulated to prevent getting a distance bonus from heavier balls. In fact, golfers regularly hit balls further than our estimated range, which is largely because it was just an approximation, but could also be partially because the aerodynamics of a golf ball are more complicated than just simple air drag, and the lift generated as they fly and spin can boost their range a bit.
This answer wasn't intended to be the final word; I guess it's pretty likely that someone in the comments will find a significant mistake. My main point for you is that physics isn't about just asking "what's the equation for this? What's the equation for that?" over and over every time a new situation pops up. Instead, it's about understanding the various interactions in a system and finding ways to model their various importances and influences, which can be done either with a great deal of formulas and computation or with very little.
When the player hits the ball with top spin, it makes the ball, well, spin.
By spinning, the ball will modify the airflow around itself and thus create an air pressure profile which will deflect the ball : this is the Magnus effect.
So by applying top spin on the ball the way tennis players do, the ball is rotating in the direction of the trajectory. This will bend the trajectory downwards. If you look at the ball's speed as a vector, the vertical component of the top-spun ball's velocity is greater than the normal served ball.
You simply direct the ball more vertically into the ground with a top spin. So after contact with the ground, the ball with top spin will leave the ground more vertically than a normal ball.
Since you are familiar with the sport, you might also have noticed that after contact, the top-spun ball will slightly accelerate towards you. This comes from the fact that part of its rotation has transfered itself into horizontal momentum. If you let a spinning ball fall vertically on the ground, after contact, it will fly out flat in some direction. I think this is also one of the reason why top spin serves are so hard to handle.
Best Answer
1. Frame of reference:
The golf ball is at the origin of an $x,y,z$ coordinate system (see Fig.1). $x$ and $y$ are in the horizontal plane. The golfer wants to project the ball in the $y$ direction.
2. Rotational and translational motion:
The ball is hit in such a way that it acquires an initial speed $v_0$, at an angle $\theta$ (see Fig.2). The ball will now follow a more or less parabolical trajectory (in the absence of air drag it would be perfectly parabolical). This motion is known as tranlational motion.
Now the golfer may want to impart on the ball an additional rotational motion, known as spin. A rotating object always rotates about an axis, so we can distinguish three possibilities.
a. Rotation around the $x$ axis, which is perpendicular to the axis of translation. Backspin: bottom moving towards the golfer. Topspin: top moving towards the golfer.
b. Rotation around the $z$ axis, in golf known as slice or hook.
c. Rotation around the $y$ axis (axis of translation): not important is sport but this spin is imparted by the rifles of a gun barrel to a bullet and increases stability of the bullet in flight.
3. Spin in flight:
Due to the Magnus effect, in air the rotation of the ball causes a drag force acting on it, perpendicularly to both the air flow and to the axis of rotation, because the airflow has been deflected in the direction of spin. Back spin thus causes extra lift and extends the flight path somewhat (see Fig.4).
Similarly, slice or hook causes a horizontal force to act on the ball, creating (left or right pointing) curvature of the trajectory.
4. Spin on impact:
Due to conservation of angular (rotational) momentum, spin also has consequences for the ball’s post-impact trajectory. It’s well known that tennis ball with backspin will travel faster and at lower angle (bounce) after hitting the floor, compared to a spin-less ball. This is similar for a golf ball.
A ball with spin/hook will also experience a slight change in horizontal direction on impact with the floor.
5. Optimal projectile angle:
A golf ball launched as indicated in Fig.2, in the absence of drag and spin, will reach a horizontal distance (see Fig.3) of: $d=\frac{{v_0}^2}{g} \sin (2 \theta)$.
It can derived that $d$ is maximum for $\theta=\frac{\pi}{4}$ (45 degrees).
6. From an energy perspective:
One could argue that imparting spin on a ball is a form of loading it with extra kinetic energy.
A spinning ball that’s also translating through the air has a translational kinetic energy at the start of the trajectory:
$E_{k,T}=\frac{m{v_0}^2}{2}$, with $m$ the mass of the ball and $v_0$ the initial speed.
But a rotating body also has kinetic energy:
$E_{k,R}=\frac{I \omega ^2}{2}$, with $I$ the inertial moment of the ball and $\omega$ its angular velocity ($\omega = 2 \pi f$ with $f$ the frequency of rotation).
The total initial kinetic energy is $E_k=\frac{m{v_0}^2}{2} + \frac{I \omega ^2}{2}$.
During flight, part of the rotational kinetic energy is converted to work by the Magnus force.