The difference between the "mouth hole" and the "needle hole" is that the mouth hole is reinforced and under low tension, while the needle hole is not reinforced and under high tension.
The mouth piece of a balloon, some might call it the "neck" of the balloon, generally includes a rolled section of material. This section is what I call it being "reinforced," since it is stronger than the rest of the balloon. Additionally, the neck is under low tension compared to the stretched fabric of the rest of the balloon.
So, why should any of that matter?
The tension on the surface of the balloon determines the response the balloon material will provide under a given load. When you poke a hole in, or open, the neck, not very much happens, because there is low tension (low force). However, when you poke a hole in the body of the balloon, the much higher tension there provides a larger response.
You can make an analogy to a stretched rubber band. If you stretch it a small amount, it is easy to perturb the rubber band a great distance. However, if the rubber band is stretched to near breaking, then even a small perturbation results in a snapping of the material, and the subsequent release of energy (both in motion of the pieces and in generation of sound).
A typical human can exert an over-pressure of ~$9.8kPa$ with their lungs. Given that balloons are designed so that a human can inflate them, I'd say go for a pump that can deliver $10kPa$ of pressure (that's around $1.5psi$ in case you were wondering).
Best Answer
I think that most of the answers here are incorrect since it has nothing to do with decreasing resistance of rubber. In fact, the force required to stretch the balloon increases, not decreases while inflating. It's similar to stretching a string, ie. the reaction force is proportional to the increase in length of the string - this is why there is a point when you can no longer stretch a chest expander.
The real reason that initially it's hard to inflate the balloon is that in the beginning, ie. with the first blow, you increase the total surface of the balloon significantly, thus the force (pressure on the surface) increases also significantly. With each subsequent blow, the increase of the total surface is smaller and so is the increase of force. This is the result of two facts:
For a sphere you have:
$$ A={4}\pi R^2 \\ V={4\over3}\pi R^3 $$ The equations says that the amount of work required to increase the volume of the balloon by one unit is smaller if the balloon is already inflated.