Despite 11 answers to this question already, I don't feel that any have answered the question well.
(Note: This answer is simplified and assumes the punch is slow enough to ignore inertia and relativity)
Firstly, let's look at force at the atomic level. This is where the force is really happening. The forces that we feel in everyday life are generally the forces between atoms and molecules (intermolecular forces). I'll use Helium atoms as an example, because they're easy to draw. When two He atoms get close together, their electron shells overlap and cause them to repel each other. Note that you never get a situation where one atom repels, and the other does nothing, or one repels and one attracts. Always they both repel each other, or both attract each other, and both atoms feel the same magnitude force, in exactly opposite directions.
The force they feel is a function of the distance between them. The force between them behaves basically like a spring. In the illustration above, the two atoms are repelling each other, and will accelerate away from each other. As they move apart, the force decreases, until at a certain point, it reaches zero, and we consider them not to be 'touching' any more.
Now imagine we start with one atom stationary, and throw another atom at it. When the moving atom gets close enough to the stationary one, they will feel the force of repulsion. Both will accelerate based on the force between them. They accelerate in opposite directions, so the stationary atom accelerates and flies off, while the moving one decelerates to a stop.
Molecules behave in a similar way towards each other.
Since a wall is made up of molecules, it behaves pretty much like the force between molecules, except in a solid object, neighboring molecules are bonded together, meaning that when you push them closer together, they repel, and when you pull them further apart, they attract. The wall is basically a very stiff spring. When you push on a wall, it bends.
Bending is the only way it can push back on you. Bending means that some of the molecules in the wall are pushed closer together, and some are pulled further apart. The harder you push, the more it bends. It bends just so that it's pushing back on you as hard as you're pushing. If you're pushing with a constant force, everything is in equilibrium, and all the force vectors acting on each molecule add up to zero, so nothing is accelerating.
If you push hard enough, you'll manage to stretch some molecules far enough apart that their bond breaks. At that point the force between them drops to zero. Now those molecules are not in equilibrium, and they will accelerate away from each other.
If you push hard enough, and the wall breaks, it's no longer bending, it's accelerating away from your hand, just like the atoms in the example above. As it accelerates away, the force between your hand and the wall decreases and reaches zero when your hand and the wall are no longer 'touching'.
When you punch a wall, the forces you and the wall are feeling are entirely made up of the forces between atoms and molecules. So whether the wall stands or falls, Newton's 3rd law holds the whole time. The wall can only push back on your hand to the extent that it can bend without breaking.
But what if I push really hard on the wall?
The answer is you can't. You can put a lot of effort into the punch, but if you were to measure the actual force applied to the wall, it would increase up to the point, then the wall would break, then the force would drop back down to zero.
Newton's 3rd law doesn't mean that everything is indestructible.
Added:
If you haven't already discovered Veritasium's excellent YouTube channel, you should. He has a good video helping us to understand Newton's Third Law:
In both cases and at all times, the force from the (wall/tire) on the hammer equals the force from the hammer on the (wall/tire) : total momentum must be conserved.
However, in the first case, the initial energy is dissipated in the wall (as heat and/or damage), so at the end the hammer is stopped.
In the second case the initial energy is stored as mechanical deformation (potential energy). As soon as the hammer is stopped, this energy does work on the hammer and that sends the hammer backwards. This "follow-up" doubles the momentum exchange between hammer and tire, that's what you feel different.
Best Answer
You've caught a non-intuitive part of Newton's 3rd law. It's actually applying in the case you mention, but because the objects involved are of dissimilar hardness it's easy to perceive the impact as a violation of the law.
Impacts are actually really complicated. Consider this slow motion video of a punch to the gut. We won't be able to cover all of the complexities we see here, but we can layer a few of them together to try to explain why the non-inutitive results you get are actually correct applications of Newton's 3rd law.
The key thing which makes impacts so complicated is that we have to pay attention to momentum. When you punch the brick wall or the drywall, your hand has quite a lot of momentum. When you punch the brick wall, that momentum has to be stopped. The only way to do this is through the reactionary force of the wall pushing back on your hand. The more momentum your hand has, the more reactionary force you deal with. In your brick example, that reactionary force is 50lbs, and the corresponding force of your hand on the wall is also 50lbs.
In the drywall case, we need to make a few adjustments. The first is to note that your hand goes through the drywall. It does not have to be stopped by the wall. This points out that the reactionary force will be less than it was in the brick wall case, because the brick wall had to stop the fist.
Well, almost. I cheated slightly, and that cheat may be a source of non-intuitive behavior. The more correct statement is that the brick wall had to impart more impulse to your hand, because it had to stop your hand. Impulse is force*time, and is a measure of change in momentum. The key detail here is that the distance can change. If you drop a superball on the ground, it rebounds almost back to where you threw it from. The impulse applied to the ball by the ground is very high. Contrast that with a steel ball bearing with the same mass as the superball, which does not rebound as much. The impulse applied to the bearing is lower. However, the superball deforms a great deal on impact, so it has a longer time to apply that impulse over. It is reasonable that superball could be subjected to less force than the ball bearing, and yet bounce higher because that lower force was applied for a longer distance.
In the case of the punch, we're lucky that 99% of the deformation in your punch occurs in your hand. Your skin and fat squish out of the way until your bones start to have to move. The shock in theory works its way all the way up your arm. However, we can ignore all of that for now, because we're just doing comparisons. It's the same hand in both the brick wall punch and the drywall punch, so it can be expected to deform in similar ways over similar distances and similar times. This is how we can claim that the brick wall punch must have a higher force. We know the impulse must be higher (because it stopped your hand, and the drywall punch didn't), and the times are the same for the reaction to both punches, so the brick wall punch must have more force.
Thus, the truth is that you did not punch the drywall with 50lbs. You actually supplied less force than that. In fact, you supplied just enough force to break the internal bonds that were keeping the drywall solid. Intuitively, we like to measure punches in forces (claiming a 50lb punch), but it's actually not possible to punch that hard unless the thing being punched is capable of providing a corresponding reactionary force! If you layered enough pieces of drywall to have the structural integrity to provide 50lbs of force, you would find that you don't break through, and it hurts almost as much as the bricks did (the first sheet of drywall will deform a little, so it wont hurt as much as the brick)
The issue of breaking through the wall is actually a very important thing for martial artists. Those who break boards or bricks in demonstrations all know that it hurts far more if you fail to break the board or brick. That's because the board stopped all of your forward momentum, meaning you had a lot of impulse over a short time, meaning a lot of force. If you break the brick, the reactionary forces don't stop your hand, so they are less. I would wager that the greatest challenge of breaking bricks with a karate chop is not breaking them, but in having conditioned your body and mind such that you can withstand the impulse when you fail to break them.