Given a system of $N$ point-particles with positions
${\bf r}_1, \ldots , {\bf r}_N$; with corresponding virtual displacements $\delta{\bf r}_1$, $\ldots $, $\delta{\bf r}_N$; with momenta ${\bf p}_1, \ldots , {\bf p}_N$; and with applied forces ${\bf F}_1^{(a)}, \ldots , {\bf F}_N^{(a)}$. Then D'Alembert's principle states that
$$\tag{1} \sum_{j=1}^N ( {\bf F}_j^{(a)} - \dot{\bf p}_j ) \cdot \delta {\bf r}_j~=~0. $$
The total force
$${\bf F}_j ~=~ {\bf F}_j^{(a)} +{\bf F}^{(ec)}_j+{\bf F}^{(ic)}_j
+ {\bf F}^{(i)}_j + {\bf F}_j^{(o)}$$
on the $j$'th particle can be divided into five types:
applied forces ${\bf F}_j^{(a)}$ (that we keep track of and that are not constraint forces).
an external constraint force ${\bf F}^{(ec)}_j$ from the environment.
an internal constraint force ${\bf F}^{(ic)}_j$ from the $N-1$ other particles.
an internal force ${\bf F}^{(i)}_j$ (that is not an applied or a constraint force of type 1 or 3, respectively) from the $N-1$ other particles.
Other forces ${\bf F}_j^{(o)}$ not already included in type 1, 2, 3 and 4.
Because of Newton's 2nd law ${\bf F}_j= \dot{\bf p}_j$, D'Alembert's principle (1) is equivalent to$^1$
$$\tag{2} \sum_{j=1}^N ( {\bf F}^{(ec)}_j+{\bf F}^{(ic)}_j+{\bf F}^{(i)}_j+{\bf F}_j^{(o)}) \cdot \delta {\bf r}_j~=~0. $$
So OP's question can essentially be rephrased as
Are there examples in classical mechanics where eq. (2) fails?
Eq. (2) could trivially fail, if we have forces ${\bf F}_j^{(o)}$ of type 5, e.g. sliding friction, that we (for some reason) don't count as applied forces of type 1.
However, OP asks specifically about internal forces.
For a rigid body, to exclude pairwise contributions of type 3, one needs the strong Newton's 3rd law, cf. this Phys.SE answer. So if these forces fail to be collinear, this could lead to violation of eq. (2).
For internal forces of type 4, there is in general no reason that they should respect eq. (2).
Example: Consider a system of two point-masses connected by an ideal spring. This system has no constraints, so there are no restrictions to the class of virtual displacements. It is easy to violate eq. (2) if we count the spring force as a type 4 force.
Reference:
H. Goldstein, Classical Mechanics, Chapter 1.
--
$^1$It is tempting to call eq. (2) the Principle of virtual work, but strictly speaking, the principle of virtual work is just D'Alembert's principle (1) for a static system.
1) According to usual terminology we wouldn't call a sliding friction force a constraint force as it doesn't enforce any constraint. (No pun intended.) In other words, a sliding friction does not by itself constrain the particles to some constraint subsurface, i.e., the particles can still move around everywhere.
On the other hand, rolling friction and static friction may actually impose a constraint, so they can be constraint forces.
2) In more detail, Goldstein says on p.17 in Chapter 1 of the book Classical Mechanics that
[The principle of virtual work] is no longer true if sliding friction forces are present [in the tally of constraint forces], ...
Goldstein goes on to say that
this restriction is not unduly hampering.
What he has in mind is, that we can still at least analyze and study many systems of fundamental/microscopic point particles (which is anyway the most important case!) with the principle of virtual work, because there are often no sliding friction forces involved down at these scales.
In particular, Goldstein does not imply that sliding friction forces are not important in macroscopic systems.
3) Later in chapter 1, Goldstein confronts us with another problem with sliding friction forces. They cannot be described with the help of a velocity-dependent potential $U$ but only in terms of Rayleigh's dissipation function ${\cal F}$. This is related to the fact that there is no action principle for systems with sliding friction forces.
Best Answer
The author of the excellent book where I found this problem (The Lazy Universe) explains in another part of the book:
I feel this is the answer to my question, but I can't say I understand how this translates to the Atwood machine problem. This is my guess:
The important thing is not to confuse virtual displacements with actual displacements. Sometimes, the actual physical displacements cannot be chosen as virtual displacements. Virtual displacements occur without the passage of time. If we think of the example above, while time is frozen the finger cannot actually move in the direction the block moves when time is flowing. For it do so, it would have to compress the block, but the block is inelastic. So that's not an allowed virtual displacement. But with things frozen in time, the finger is allowed to move within the face of the block (like up and down across the face), and this motion is at right-angles to the reaction (constraint) force from the block on the finger.
As @alephzero stated in another answer, the real constraint in the Atwood machine "is that the string has constant length". This means that, like the block in the previous example, the string can't be compressed. Therefore, if we freeze time, we'll find that we can't take the virtual displacement of the blocks to be the same as the actual displacement, i.e, "up and down", i.e, in the direction of the strings. For them to do so with time frozen they would have to compress the strings, and the strings can't be compressed. So we are once again "getting distracted by the actual motions". This is another case where we can't take the actual displacements as the virtual displacements. The virtual displacements allowed for the blocks are actually at right angles to the string, not up and down, but sideways!
Am I onto something?