The possible right-hand-sides for this method are, themselves, solutions of linear homogeneous differential equations with constant coefficients. User8268 provided another description of this class in a comment.
If your equation is $\mathsf Ly=f$, where $\mathsf L$ is a linear differential operator with constant coefficients, and $\mathsf Mf=0$ where $\mathsf M$ is another such operator, then your desired solution is also a solution of $\mathsf{ML}y=0$, and knowledge of solutions of such (homogeneous differential equations with constant coefficients) can then be used to show what the form of $y$ must be.
The_lost is right in his comment, we show $xe^{rx}$ solves the equation
$Ay''(x) + By'(x) + Cy(x) = 0, \; A \ne 0, \tag 0$
by simple substitution (here I assume $A \ne 0$ to ensure (0) is a bona-fide second order equation); however, it helps to know something about $r$ first, to wit:
According to the quadratic formula, the roots of
$Ar^2 + Br + C = 0, \; A \ne 0, \tag 1$
are
$r_1, r_2 = \dfrac{-B \pm \sqrt{B^2 - 4AC}}{2A}; \tag 2$
we see that there is one root
$r = r_1 = r_2 = -\dfrac{B}{2A} \tag 3$
precisely when
$B^2 - 4AC = 0, \tag 4$
or, since $A \ne 0$,
$C = \dfrac{B^2}{4A}, \tag 5$
or
$\dfrac{C}{A} = \dfrac{B^2}{4A^2} = \left ( \dfrac{B}{2A} \right )^2; \tag 6$
for then,
$Ar^2 + Br + C = A \left (r^2 + \dfrac{B}{A} r + \dfrac{C}{A} \right )$
$= A \left ( r^2 + \dfrac{B}{A} + \left ( \dfrac{B}{2A} \right )^2 \right ) = A \left ( r + \dfrac{B}{2A} \right )^2; \tag 7$
now if
$y(x) = x e^{rx}, \tag 8$
we calculate:
$y'(x) = e^{rx} + rxe^{rx}; \tag 9$
$y''(x) = re^{rx} + re^{rx} + r^2xe^{rx} = r^2xe^{rx} + 2re^{rx}, \tag{10}$
whence, via (3),
$y''(x) + \dfrac{B}{A}y'(x) + \dfrac{C}{A} y(x) = r^2xe^{rx} + 2re^{rx} + \dfrac{B}{A}(rxe^{rx} + e^{rx}) + \dfrac{C}{A} x e^{rx}$
$= \left (r^2 + \dfrac{B}{A}r + \dfrac{C}{A} \right ) xe^{rx} + \left (2r + \dfrac{B}{A} \right )e^{rx}$
$= \left ( r + \dfrac{B}{2A} \right )^2 xe^{rx} + 2 \left (r + \dfrac{B}{2A} \right ) e^{rx} = 0; \tag{11}$
it then follows that
$Ay''(x) + By'(x) + Cy(x) = A \left (y''(x) + \dfrac{B}{A}y'(x) + \dfrac{C}{A} y(x) \right ) = 0 \tag{12}$
as well.
Best Answer
Since your root $-5$ is double, the right form of $y_p$ is $$ Y_p=x^2(ax+b)e^{-5x}. $$