I hope this doesn't confuse you, but in one sense, yes, heavier bodies do fall faster than light ones, even in a vacuum. Previous answers are correct in pointing out that if you double the mass of the falling object, the attraction between it and the earth doubles, but since it is twice as massive its acceleration is unchanged. This, however, is true in the frame of reference of the center of mass of the combined bodies. It is also true that the earth is attracted to the falling body, and with twice the mass (of the falling body), the earth's acceleration is twice as large. Therefore, in the earth's frame of reference, a heavy body will fall faster than a light one.
Granted, for any practical experiment I don't see how you'd measure a difference that small, but in principle it is there.
We also know that in reality a lead feather falls much faster than a
duck's feather with exactly the same dimensions/structure etc
No, not in reality, in air. In a vacuum, say, on the surface of the moon (as demonstrated here), they fall at the same rate.
Is there a more formal mathematical explanation for why one falls
faster than the other?
If the two objects have the same shape, the drag force on the each object, as a function of speed $v$, is the same.
The total force accelerating the object downwards is the difference between the force of gravity and the drag force:
$$F_{net} = mg - f_d(v)$$
The acceleration of each object is thus
$$a = \frac{F_{net}}{m} = g - \frac{f_d(v)}{m}$$
Note that in the absence of drag, the acceleration is $g$. With drag, however, the acceleration, at a given speed, is reduced by
$$\frac{f_d(v)}{m}$$
For the much more massive lead feather, this term is much smaller than for the duck's feather.
Best Answer
When you are lifting an object, you are exerting a force that balances the force of gravity on the object. By $$ F = m g$$ where g is the acceleration due to gravity, you see that a greater mass causes a greater gravitational force that has to be balanced by the force you apply to the object by holding it or lifting it at a constant velocity.
Using the more general Newtonian law of gravitation, $$F = G \frac{M m}{r^2}$$ with $G$ being the gravitational constant, $r$ the separation and $M$ and $m$ two masses, we can rearrange for an object in the gravitational pull of Earth: $$F = G \frac{M_{Earth}}{r^2} m$$
If we approximate this by saying that the object is very close to Earth and its mass is nearly zero, the first part of the formula depends only on Earth's physical properties and therefore becomes a constant: $$g = G \frac{M_{Earth}}{r^2}$$
We see that $$g = 6.67 \times 10^{-11} \frac{5.97 \times 10^{24}}{(6.37\times 10^6)^2} \frac{N m^2}{kg^2} \frac{kg}{m^2} = 9.81 \frac{N}{kg} = 9.81 \frac{m}{s}$$
So, the acceleration due to gravity only depends on the mass and the radius of the planet (with our assumptions), but the force that you need to exert in order to balance gravity depends on the mass of the object you are trying to hold.