Correct. If you split the earth up into spherical shells, then the gravity from the shells "above" you cancels out, and you only feel the shells "below" you. When you are in the middle there is nothing "below" you.
Refrence from Wikipedia Gauss & Shell Theorem.
{I am using some simplistic terms, but I don't want to break out surface integrals and radial flux equations}
Edit: Although the inside of the shell will have zero gravity classically, it will also have non zero gravity relativistically. At the perfect center the forces may balance out, yielding an unstable solution, meaning that a small perturbation in position will result in forces that exaggerate this perturbation.
Both you and your ancestors are wrong. But I bet you would never guess the real answer!
Assuming the base of the ladder doesn't slide you have a rotating system. Just like a freely falling man you convert potential energy to kinetic energy, but for a rotating system the kinetic energy is given by:
$$ T = \frac{1}{2}I\omega^2 $$
where $I$ is the angular momentum and $\omega$ is the angular velocity. Note that $v = r\omega$, where $r$ is the radius (i.e. the length of the ladder). We'll need this shortly.
Let's start by ignoring the mass of the ladder. In that case the moment of inertia of the system is just due to the man and assuming we treat the man as a point mass $I = ml^2$, where $m$ is the mass of the man and $l$ the length of the ladder. Setting the change in potential energy $mgl$ equal to the kinetic energy we get:
$$ mgl = \frac{1}{2}I\omega^2 = \frac{1}{2}ml^2\omega^2 = \frac{1}{2}mv^2 $$
where we get the last step by noting that $l\omega = v$. So:
$$ v^2 = 2gl $$
This is exactly the same result as we get for the man falling straight down, so you hit the ground with the same speed whether you fall straight down or whether you hold onto the ladder.
But now let's include the mass of the ladder, $m_L$. This adds to the potential energy because the centre of gravity of the ladder falls by $0.5l$, so:
$$ V = mgl + \frac{1}{2}m_Lgl $$
Now lets work out the kinetic energy. Since the man and ladder are rotating at the same angular velocity we get:
$$ T = \frac{1}{2}I\omega^2 + \frac{1}{2}I_L\omega^2 $$
For a rod of mass $m$ and length $l$ the moment of inertia is:
$$ I_L = \frac{1}{3}m_Ll^2 $$
So let's set the potential and kinetic energy equal, as as before we'll substitute for $I$ and $I_L$ and set $\omega = v/l$. We get:
$$ mgl + \frac{1}{2}m_Lgl = \frac{1}{2}mv^2 + \frac{1}{6}m_Lv^2$$
and rearranging this gives:
$$ v^2 = 2gl \frac{m + \frac{1}{2}m_L}{m + \frac{1}{3}m_L} $$
and if $m_L \gt 0$ the top of the fraction is greater than the bottom i.e. the velocity is greater than $2gl$. If you hold onto the ladder you actually hit the ground faster than if you let go!
This seems counterintuitive, but it's because left to itself the ladder would rotate faster than the combined system of you and the ladder. In effect the ladder is accelerating you as you and the ladder fall. That's why the final velocity is higher.
Best Answer
If you are falling you feel weightless even though you are still being affected by gravity - the sensation of weightlessness is just that there is nothing around you to press against you.
An astronaut in orbit is effected by gravity almost as much as you are (being 10% further from the centre of the earth doesn't make gravity much less)