Using your definition of "falling," heavier objects do fall faster, and here's one way to justify it: consider the situation in the frame of reference of the center of mass of the two-body system (CM of the Earth and whatever you're dropping on it, for example). Each object exerts a force on the other of
$$F = \frac{G m_1 m_2}{r^2}$$
where $r = x_2 - x_1$ (assuming $x_2 > x_1$) is the separation distance. So for object 1, you have
$$\frac{G m_1 m_2}{r^2} = m_1\ddot{x}_1$$
and for object 2,
$$\frac{G m_1 m_2}{r^2} = -m_2\ddot{x}_2$$
Since object 2 is to the right, it gets pulled to the left, in the negative direction. Canceling common factors and adding these up, you get
$$\frac{G(m_1 + m_2)}{r^2} = -\ddot{r}$$
So it's clear that when the total mass is larger, the magnitude of the acceleration is larger, meaning that it will take less time for the objects to come together. If you want to see this mathematically, multiply both sides of the equation by $\dot{r}\mathrm{d}t$ to get
$$\frac{G(m_1 + m_2)}{r^2}\mathrm{d}r = -\dot{r}\mathrm{d}\dot{r}$$
and integrate,
$$G(m_1 + m_2)\left(\frac{1}{r} - \frac{1}{r_i}\right) = \frac{\dot{r}^2 - \dot{r}_i^2}{2}$$
Assuming $\dot{r}_i = 0$ (the objects start from relative rest), you can rearrange this to
$$\sqrt{2G(m_1 + m_2)}\ \mathrm{d}t = -\sqrt{\frac{r_i r}{r_i - r}}\mathrm{d}r$$
where I've chosen the negative square root because $\dot{r} < 0$, and integrate it again to find
$$t = \frac{1}{\sqrt{2G(m_1 + m_2)}}\biggl(\sqrt{r_i r_f(r_i - r_f)} + r_i^{3/2}\cos^{-1}\sqrt{\frac{r_f}{r_i}}\biggr)$$
where $r_f$ is the final center-to-center separation distance. Notice that $t$ is inversely proportional to the total mass, so larger mass translates into a lower collision time.
In the case of something like the Earth and a bowling ball, one of the masses is much larger, $m_1 \gg m_2$. So you can approximate the mass dependence of $t$ using a Taylor series,
$$\frac{1}{\sqrt{2G(m_1 + m_2)}} = \frac{1}{\sqrt{2Gm_1}}\biggl(1 - \frac{1}{2}\frac{m_2}{m_1} + \cdots\biggr)$$
The leading term is completely independent of $m_2$ (mass of the bowling ball or whatever), and this is why we can say, to a leading order approximation, that all objects fall at the same rate on the Earth's surface. For typical objects that might be dropped, the first correction term has a magnitude of a few kilograms divided by the mass of the Earth, which works out to $10^{-24}$. So the inaccuracy introduced by ignoring the motion of the Earth is roughly one part in a trillion trillion, far beyond the sensitivity of any measuring device that exists (or can even be imagined) today.
it is because the Force at work here (gravity) is also dependent on the mass
gravity acts on a body with mass m with
$$F = mg$$
you will plug this in to $$F=ma$$ and you get
$$ma = mg$$
$$a = g$$
and this is true for all bodies no matter what the mass is. Since they are accelerated the same and start with the same initial conditions (at rest and dropped from a height h) they will hit the floor at the same time.
This is a peculiar aspect of gravity and underlying this is the equality of inertial mass and gravitational mass (here only the ratio must be the same for this to be true but Einstein later showed that they're really the same, i.e. the ratio is 1)
Best Answer
As long as the mass that we aren't dropping is very large and is kept constant, then the mass of the object we are dropping has no considerable effect on its acceleration. This is because of Newton's 2nd Law:
$$F = ma$$
Where $m$ is the mass that is accelerating, i.e. the smaller mass we are dropping. So, if $F = G\frac{Mm}{r^2}$, where $m$ is the mass we dropped, and $M$ is the big mass that the object we dropped is fall to, then:
$$a = \frac{F}{m} = G\frac{M}{r^2}$$
So, while acceleration is dependent in $M$, it does not depend on the mass of the dropped object.
The constant value $g$ is actually only true on the earth's surface, and is appropriately defined as:
$$g_{earth} = G\frac{M}{(R_{earth})^2}$$
Where $R_{earth}$ is the radius of the Earth.
Notice that I said the bigger mass, $M$ (or, the mass that is causing the gravitational field) is, indeed, big. If it were not that big, the object of the mass we dropped (by Newton's 3rd Law) would cause a force on $M$ that results in a significant acceleration of $M$. This means that both masses are significantly accelerating towards each other, so that the effective acceleration of the dropped object (i.e. the acceleration in a reference frame where $M$ is stationary) would be the sum of the magnitudes of the accelerations of each object in a non-accelerating frame of reference:
$$a_{effective} = a_m + a_M = G\frac{M}{r^2} + G\frac{m}{r^2} = G\frac{M+m}{r^2}$$
In which case, the mass of the dropped object does indeed affect the effective acceleration.
(Note that the "effective acceleration" is the acceleration of the dropped object when $M$ appears stationary to the observer. This does not appear to obey Newton's 2nd Law (i.e. $a_{effective} \neq a_m$ because this frame of reference is accelerating, and accelerating reference, otherwise know as non-inertial reference frames, do not obey Newton's 2nd Law! In fact, this question assumes that, in the inertial frame of reference, the two masses are heading towards each other in a straight line...)