a) Cuold someone briefly explain to me why the block on top can accelerate, but the one hanging on the pulley does not?
Both blocks do accelerate, and in fact they have the same acceleration. You'll see that if you write out the full set of both the x and y components of Newton's second law for each system (i.e. for each block). In the given solution, they only wrote out the components that are directly needed to solve the problem.
Notice how the 3kg block is actually "attached" tot he 8kg block, yet the solution here didn't include it in its free-body diagram and they even included the reaction force from the 3kg on the 8kg.
Compared with the cart problem where the hanging mass is also touching the cart, no reaction force was drawn and they even treated the three mass as a single mass.
It's not attached. The only interactions between the 8kg and 3kg blocks are the normal force and static friction, neither of which qualifies as "attachment." This is exactly analogous to the interaction between $M$ and $m_2$ in the first problem. The reason no reaction force was drawn in the first case (by "reaction force" I assume you are referring to the reaction to the normal force acting on $m_2$) was that they didn't draw a free body diagram for the object the reaction force acts on, namely $M$. Again, the given solution omits some pieces (in this case, a free body diagram) which aren't necessary for solving the problem. If you draw a free body diagram for $M$, you will see the reaction force.
Could you have solved the first problem with the cart WITHOUT taking all three mass into the cart's FBD? Is there another way of applying Newton's Second Law to the cart problem?
If by this you are asking whether you could have solved the first problem by drawing separate free body diagrams for each of the three masses $m_1$, $m_2$, and $M$, then yes, you certainly can.
The centripetal force isn't a separate force. It is the vector sum of the forces that are present. In the case of a roller coaster, the normal force and the force of gravity will add vectorically to produce a net force, and that net force must apply the centripetal force to keep you moving in a circle. Because the force of gravity is constant, the normal force will change (both in magnitude and direction) to provide the remainder of the centripetal force.
When you are at the top of the loop, both the normal force and gravity point in the same direction. When you are at the bottom of the loop, the normal force points up, the opposite direction as gravity (down). So the normal force will have to be larger in magnitude when you are at the bottom of the loop than it is when you are at the top of the loop.
Best Answer
The vector you see is not the centrifugal force, which would appear in a rotating frame. The book is working consistently in an inertial frame. This is just the normal force, and they are just noting that it is equal to $mv^2/r$.
What the answer did not do is include a left-pointing centrifugal force in the diagram to cancel out the right-pointing normal. Including a centrifugal force in a free-body diagram is considered a mistake in elementary mechanics, although as Peter Shor points out, it is not a mistake at all if you are working in a non-inertial frame (which is consistent, just considered slightly more advanced).
The reason that the normal is not cancelled by anything is that you need an acceleration to keep something moving in a circle. This is counterintuitive, because we tend to shift reference frame to move along with the object, so that if you have a force pointing to the left, intuition suggests that it must be balanced by a force to the right. This is true in the rotating frame, but not in the inertial frame description.