Your approach is all right but the solution given by the textbook is wrong :), at least if no approximation is to be made.
Let's go the other way around: start from $$P(V,T) = \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2V_0}\left[\left(\frac{V_0}{V}\right)^5 - \left(\frac{V_0}{V}\right)^3\right]$$ and then derive the values of $\kappa_T$ and $\beta$ (by the way, shouldn't it rather be $\chi_T$ (see here) and $\alpha$ (here)?). As you mentioned, to do this, we need to compute partial derivatives of $P$:
$$
\left(\frac{\partial P}{\partial T}\right)_V = \frac{\Gamma c_v}{V}
$$
$$
\left(\frac{\partial P}{\partial V}\right)_T =
-\left(\frac{\Gamma c_vT}{V^2} + \frac{\varepsilon}{2{V_0}^2}\left[5\left(\frac{V_0}{V}\right)^6 - 3\left(\frac{V_0}{V}\right)^4\right]\right)
$$
which give
$$
\frac{1}{\kappa_T} = -V \left(\frac{\partial P}{\partial V}\right)_T
= \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2{V_0}}\left[5\left(\frac{V_0}{V}\right)^5 - 3\left(\frac{V_0}{V}\right)^3\right]
$$
and
$$
\beta = \kappa_T \left(\frac{\partial P}{\partial T}\right)_V
= \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[5\left(\frac{V_0}{V}\right)^4 - 3\left(\frac{V_0}{V}\right)^2\right]
\right)^{-1}
$$
Clearly, that's not what the textbook gives in the first place, but it's close enough to understand what they did: a Taylor expansion at order 3 in $V_0/V$ in booth cases, which gives back the expressions
$$
\frac{1}{\kappa_T}
\approx \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2{V_0}}\left[ - 3\left(\frac{V_0}{V}\right)^3\right]
$$
and
$$
\beta
\approx \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[ - 3\left(\frac{V_0}{V}\right)^2\right]
\right)^{-1}
\approx \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[ 3\left(\frac{V_0}{V}\right)^2\right]
\right)
$$
It really should have been clearer in the text that you could suppose $V\gg V_0$ (and not only $V>V_0$) and I don't see how you could derive the term in $\left(\frac{V_0}{V}\right)^5$ from this as it is completely neglected to find back $\kappa_T$ and $\beta$
Are you sure that the first block of code is correct? V2 should have been multiplied by V. You get correct values here only because you assumed V=1.
No, the difference in volumes isn't equal (or even in linear proportion) to difference in densities because they have an inverse relation. In terms of densities you have: $\alpha = {m (\frac 1 \rho - \frac 1 \rho_1) \over dT}$
Best Answer
Presumably because ...