For your first question: obtaining equation (2) from (1) is basically just a matter of taking the derivative. Consider this expression that relates an infinitesimal change in the value of a function $f(x,b)$ to the infinitesimal changes in the values of its arguments:
$$\delta f(x,b) = \frac{\partial f(x,b)}{\partial x}\delta x + \frac{\partial f(x,b)}{\partial b}\delta b$$
If you've ever done anything with uncertainty analysis, in particular error propagation, you should be familiar with this sort of thing, but if not, it's still not too complicated - it's just the chain rule of multivariable calculus. Try plugging in the left side of equation (1) for $f(x,b)$ and see that it works out.
A different way to do the same derivation is to consider the right triangle whose legs are formed by the rod and the "ceiling" and whose hypotenuse is formed by the string. Originally, the triangle has side lengths $a$, $b$, and $l - x$; after an infinitesimal displacement of the system, it has side lengths $a$, $b + \delta b$, and $l - x - \delta x$. Using the Pythagorean theorem on the first set of lengths, you get
$$a^2 + b^2 = (l - x)^2$$
and on the second set of lengths, you get
$$a^2 + (b + \delta b)^2 = (l - x + \delta x)^2$$
Try combining these to see that you get equation (2) out of it. Remember that because the $\delta$ quantities are infinitesimal, you can ignore terms involving $\delta x^2$ and $\delta b^2$.
As for the second question: probably the easiest way to think about it is that for a static system, the principle of virtual work is equivalent to potential energy minimization. To find the state of the system, you find what values of the coordinates minimize the potential energy, which in this case is $-Mgx - mgb$. Ordinarily, when you want to minimize a function, you take the derivative and set it equal to zero, but you can just as well calculate an infinitesimal change using the formula above and set that equal to zero. Try setting $f(x,b)$ in that formula equal to the potential energy.
If you're still concerned about tension, though, you could make the following argument, staying true to the spirit of virtual work. As the ring moves downward by a small amount $\delta b$, the work done on it is
$$\delta W_m = mg\delta b - T\sin\theta\,\delta b$$
where the factor $\sin\theta = \frac{b}{\sqrt{a^2 + b^2}}$ picks out the vertical component of the tension.
As this happens, the string gets pulled through the pulley, thus raising the larger mass by a displacement $\delta x$. The work done on the larger mass is going to be
$$\delta W_M = -Mg\delta x + T\delta x$$
When the system is in equilibrium, it will be in a state such that the no work gets done if it shifts a tiny amount in either direction. That means that the total work for an infinitesimal displacement should be zero,
$$\delta W_m + \delta W_M = 0$$
Using this and equation (2), namely $\delta x = -\frac{b}{\sqrt{a^2 + b^2}}\delta b$, you can show that the two tension terms, one from $\delta W_m$ and one from $\delta W_M$, cancel out. Any work done by the tension on one mass is canceled out by work done by tension on the other mass. So even if you don't actually take the tension into account, you still wind up getting the right result.
Best Answer
Choose $\alpha$ as the generalized coordinate, so $y_\text{COM}=\frac{1}{2}L \sin\alpha$, and $x_B=L\cos\alpha$.
Then $\delta y_\text{COM}=\frac{1}{2}L \cos\alpha \,\delta\alpha$, and $\delta x_B=-L\sin\alpha \,\delta\alpha$. Substitute into the equation.