Your approach is all right but the solution given by the textbook is wrong :), at least if no approximation is to be made.
Let's go the other way around: start from $$P(V,T) = \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2V_0}\left[\left(\frac{V_0}{V}\right)^5 - \left(\frac{V_0}{V}\right)^3\right]$$ and then derive the values of $\kappa_T$ and $\beta$ (by the way, shouldn't it rather be $\chi_T$ (see here) and $\alpha$ (here)?). As you mentioned, to do this, we need to compute partial derivatives of $P$:
$$
\left(\frac{\partial P}{\partial T}\right)_V = \frac{\Gamma c_v}{V}
$$
$$
\left(\frac{\partial P}{\partial V}\right)_T =
-\left(\frac{\Gamma c_vT}{V^2} + \frac{\varepsilon}{2{V_0}^2}\left[5\left(\frac{V_0}{V}\right)^6 - 3\left(\frac{V_0}{V}\right)^4\right]\right)
$$
which give
$$
\frac{1}{\kappa_T} = -V \left(\frac{\partial P}{\partial V}\right)_T
= \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2{V_0}}\left[5\left(\frac{V_0}{V}\right)^5 - 3\left(\frac{V_0}{V}\right)^3\right]
$$
and
$$
\beta = \kappa_T \left(\frac{\partial P}{\partial T}\right)_V
= \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[5\left(\frac{V_0}{V}\right)^4 - 3\left(\frac{V_0}{V}\right)^2\right]
\right)^{-1}
$$
Clearly, that's not what the textbook gives in the first place, but it's close enough to understand what they did: a Taylor expansion at order 3 in $V_0/V$ in booth cases, which gives back the expressions
$$
\frac{1}{\kappa_T}
\approx \frac{\Gamma c_vT}{V} + \frac{\varepsilon}{2{V_0}}\left[ - 3\left(\frac{V_0}{V}\right)^3\right]
$$
and
$$
\beta
\approx \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[ - 3\left(\frac{V_0}{V}\right)^2\right]
\right)^{-1}
\approx \frac1T \left(1 + \frac{\varepsilon }{2\Gamma c_V T}\left[ 3\left(\frac{V_0}{V}\right)^2\right]
\right)
$$
It really should have been clearer in the text that you could suppose $V\gg V_0$ (and not only $V>V_0$) and I don't see how you could derive the term in $\left(\frac{V_0}{V}\right)^5$ from this as it is completely neglected to find back $\kappa_T$ and $\beta$
The internal energy of a mono-atomic gas is given by:
$E_{\text{int}}=\dfrac{3}{2}nRT$.
Where $n$ is the number of moles and $R$ is the gas constant and $T$ is the temperature.
The statement of conservation of energy is given by:
$E_{\text{int}}=Q+W$
The work done by the gas is given by $W=-\int PdV$.
For a gas undergoing temperature variation at constant volume, the work $W$ done by it, is of course zero, therefore
$\Delta E_{\text{int}}=Q=\dfrac{3}{2}nR\Delta T$.
Now we define the molar heat capacity at constant volume $C_V$ as the amount of heat $Q$ required to raise the temperature of one mole of a gas by $1$ degree at constant volume. Therefore it follows from the last equation that:
$C_V=\dfrac{3}{2}R$ for one mole of any monoatomic gas.
For an adiabatic process applied to one mole of a gas, by definition $Q=0$ therefore it follows from $E_{\text{int}}=Q+W$ that
$\Delta E_{\text{int}}=W=\dfrac{3}{2}R\Delta T=-P\Delta V$.
Since $\dfrac{3}{2}R=C_V$.
Therefore
$C_V\Delta T = -P\Delta V$.
For any given mono-atomic gas, since it's always the case that $C_V=\dfrac{3}{2}R$, therefore you can use it whenever you like, whether the process under question is isovolumetric or not.
Best Answer
To leading order in $\alpha$, the volume expansion of your container does not depend on its shape, and is equal to $\Delta V = 3V\alpha\Delta T$. It is fairly simple to verify this for a cylinder, cube or sphere. Also the expansion coefficient for aluminum is going to be quite a bit smaller than glycerol, so you may be intended to simply neglect the expansion of the container.