Your approach is all right but the solution given by the textbook is wrong :), at least if no approximation is to be made.

Let's go the other way around: start from $$P(V,T) = \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2V_0}\left[\left(\frac{V_0}{V}\right)^5 - \left(\frac{V_0}{V}\right)^3\right]$$ and then derive the values of $\kappa_T$ and $\beta$ (by the way, shouldn't it rather be $\chi_T$ (see here) and $\alpha$ (here)?). As you mentioned, to do this, we need to compute partial derivatives of $P$:

$$
\left(\frac{\partial P}{\partial T}\right)_V = \frac{\Gamma c_v}{V}
$$

$$
\left(\frac{\partial P}{\partial V}\right)_T =
-\left(\frac{\Gamma c_vT}{V^2} + \frac{\varepsilon}{2{V_0}^2}\left[5\left(\frac{V_0}{V}\right)^6 - 3\left(\frac{V_0}{V}\right)^4\right]\right)
$$

which give

$$
\frac{1}{\kappa_T} = -V \left(\frac{\partial P}{\partial V}\right)_T
= \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2{V_0}}\left[5\left(\frac{V_0}{V}\right)^5 - 3\left(\frac{V_0}{V}\right)^3\right]
$$
and

$$
\beta = \kappa_T \left(\frac{\partial P}{\partial T}\right)_V
= \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[5\left(\frac{V_0}{V}\right)^4 - 3\left(\frac{V_0}{V}\right)^2\right]
\right)^{-1}
$$

Clearly, that's not what the textbook gives in the first place, but it's close enough to understand what they did: a Taylor expansion at order 3 in $V_0/V$ in booth cases, which gives back the expressions

$$
\frac{1}{\kappa_T}
\approx \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2{V_0}}\left[ - 3\left(\frac{V_0}{V}\right)^3\right]
$$

and

$$
\beta
\approx \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[ - 3\left(\frac{V_0}{V}\right)^2\right]
\right)^{-1}
\approx \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[ 3\left(\frac{V_0}{V}\right)^2\right]
\right)
$$

It really should have been clearer in the text that you could suppose $V\gg V_0$ (and not only $V>V_0$) and I don't see how you could derive the term in $\left(\frac{V_0}{V}\right)^5$ from this as it is completely neglected to find back $\kappa_T$ and $\beta$

In general, you could write V as a function of p and T, so:

$$dV = \frac{\partial V}{\partial p} dp + \frac{\partial V}{\partial T} dT$$

Because the transformation is isothermal, you have $dT = 0$, so:

$$dV = (\frac{\partial V}{\partial p})_T ~ dp $$

The work is $W = -\int_i^f p dV = - \int_i^f p (\frac{\partial V}{\partial p})_T ~ dp$

Suppose that $(\frac{\partial V}{\partial p})_T, \kappa, \rho$, are constant, we would have :

$$W = - (\frac{\partial V}{\partial p})_T \int_i^f p ~ dp = - \frac{1}{2} (\frac{\partial V}{\partial p})_T (p_f^2 - p_i^2) = -\frac{m{ \kappa}}{2{\rho}} (p_f^2 - p_i^2)$$

If the quantities $(\frac{\partial V}{\partial p})_T, \kappa, \rho$ are varying slowly with p, an acceptable approximation of W is :
$$W \approx -\frac{m{\bar \kappa}}{2{\bar \rho}} (p_f^2 - p_i^2)$$, where $\bar \kappa$ and $\bar \rho$ are mean quantities (based on the interval variation of the pressure).

## Best Answer

Hint: use the chain rule:$$\beta_S=-\rho \frac{\partial \frac{1}{\rho}}{\partial p}=-\rho\frac{\partial \frac{1}{\rho}}{\partial \rho} \frac{\partial {\rho}}{\partial p}$$

No need to use $PV=nRT$, which does not hold for non-ideal gases.