You will be punching the feather with a really small force. That doesn't mean your arm is punching lightly. Your arm can have a lot of momentum and internal tension due to internal forces that makes you punch hard, but the actual force is defined on the interaction with another object, in this case: the feather, so the force you apply to the feather is pretty small compared with the maximum force your arm would be capable of doing.
Update: In a comment Jim says:
Unless the collision is elastic, in which case the feather will experience the full force and accelerate to an enormous speed
This is incorrect (although it depends on the definition of "enormous"). Here I will show that the "enormous" speed that you can get for the feather is at most twice that of the arm. If you calculate the final speed for an elastic collision (which I'll leave as homework), you will obtain:
$$v_{feather}=\frac{2v_{arm}}{1+\mu}$$
where $\mu=\frac{m_{feather}}{m_{arm}}$
Thus, in the limit of a negligible feather mass, and in an elastic collision, the speed of the feather will be at most twice that of the arm. The speed will be still less if the collision is not elastic.
UPDATE:
For those who find it difficult to get the equation.
From conservation of momentum: $m_{arm}v_{arm.ini}=m_{arm}v_{arm.final}+m_{feather}v_{feather}$
From conservation of energy: $.5m_{arm}v_{arm.ini}^2=.5m_{arm}v_{arm.final}^2+.5m_{feather}v_{feather}^2$
if you eliminate $v_{arm.final}$ you get
$$v_{feather.initial}=\frac{2v_{arm}}{1+\mu}$$
The assumption that you are making here is that with the same motion of a punch, that you are applying $\text{100 N}$ of force to both a wall and to the air. However, you should think about the most fundamental equation of Newton's laws, namely,
$F=ma$
The most important part of this in relation to what you are talking about is that the force applied, $F$, is proportional to the acceleration, $a$. When you hit the wall, your hand goes from full speed to a complete stop rather quickly. This is a fast deceleration, or a high value of $a$, so the value of the force is high.
However, when you punch through the air, the air molecules hardly slow down your hand at all. This means that your deceleration is low, or a low value of $a$, meaning that the force, $F$ is also low.
In the absence of a wall to stop your fist, what is really stopping your punch is your own body, not the air, as your arm socket will have to pull back on your arm to keep your fist from flying away. This also will slow down your arm more slowly than a wall however, since the tendons and ligaments in your arm tend to stretch, reducing the deceleration compared to the wall, and thus the force.
So, what I'm trying to say here, is that yes, the forces are always equal and opposite which is in line with Newton's laws. However, your assumption that force from hitting a wall with a punch is the same as a force from swinging your fist through the air is incorrect.
I hope this clears things up for you.
Best Answer
Despite 11 answers to this question already, I don't feel that any have answered the question well.
(Note: This answer is simplified and assumes the punch is slow enough to ignore inertia and relativity)
Firstly, let's look at force at the atomic level. This is where the force is really happening. The forces that we feel in everyday life are generally the forces between atoms and molecules (intermolecular forces). I'll use Helium atoms as an example, because they're easy to draw. When two He atoms get close together, their electron shells overlap and cause them to repel each other. Note that you never get a situation where one atom repels, and the other does nothing, or one repels and one attracts. Always they both repel each other, or both attract each other, and both atoms feel the same magnitude force, in exactly opposite directions.
The force they feel is a function of the distance between them. The force between them behaves basically like a spring. In the illustration above, the two atoms are repelling each other, and will accelerate away from each other. As they move apart, the force decreases, until at a certain point, it reaches zero, and we consider them not to be 'touching' any more.
Now imagine we start with one atom stationary, and throw another atom at it. When the moving atom gets close enough to the stationary one, they will feel the force of repulsion. Both will accelerate based on the force between them. They accelerate in opposite directions, so the stationary atom accelerates and flies off, while the moving one decelerates to a stop.
Molecules behave in a similar way towards each other.
Since a wall is made up of molecules, it behaves pretty much like the force between molecules, except in a solid object, neighboring molecules are bonded together, meaning that when you push them closer together, they repel, and when you pull them further apart, they attract. The wall is basically a very stiff spring. When you push on a wall, it bends.
Bending is the only way it can push back on you. Bending means that some of the molecules in the wall are pushed closer together, and some are pulled further apart. The harder you push, the more it bends. It bends just so that it's pushing back on you as hard as you're pushing. If you're pushing with a constant force, everything is in equilibrium, and all the force vectors acting on each molecule add up to zero, so nothing is accelerating.
If you push hard enough, you'll manage to stretch some molecules far enough apart that their bond breaks. At that point the force between them drops to zero. Now those molecules are not in equilibrium, and they will accelerate away from each other.
If you push hard enough, and the wall breaks, it's no longer bending, it's accelerating away from your hand, just like the atoms in the example above. As it accelerates away, the force between your hand and the wall decreases and reaches zero when your hand and the wall are no longer 'touching'.
When you punch a wall, the forces you and the wall are feeling are entirely made up of the forces between atoms and molecules. So whether the wall stands or falls, Newton's 3rd law holds the whole time. The wall can only push back on your hand to the extent that it can bend without breaking.
But what if I push really hard on the wall?
The answer is you can't. You can put a lot of effort into the punch, but if you were to measure the actual force applied to the wall, it would increase up to the point, then the wall would break, then the force would drop back down to zero.
Newton's 3rd law doesn't mean that everything is indestructible.
Added:
If you haven't already discovered Veritasium's excellent YouTube channel, you should. He has a good video helping us to understand Newton's Third Law: