You're right, just take derivatives and plug in.
$$
\begin{align}
h&=L\cosh{k(x-a)}+M\sinh{k(x-a)}\\
h'&=Lk\sinh{k(x-a)}+Mk\cosh{k(x-a)}\\
h''&=Lk^2\cosh{k(x-a)}+Mk^2\sinh{k(x-a)}
\end{align}
$$
so then notice that, indeed,
$$h''-k^2h=0$$
so your proposed $h$ is a solution to this differential equation.
I can help withthe second part, expressing $h$ as an exponential function, but I want to see what you've tried so far, so we can help the most with what you don't understand.
Hint:
Let $r=\sinh(x)$ ,
Then $\dfrac{du}{dx}=\dfrac{du}{dr}\dfrac{dr}{dx}=\cosh(x)\dfrac{du}{dr}$
$\dfrac{d^2u}{dx^2}=\dfrac{d}{dx}\left(\cosh(x)\dfrac{du}{dr}\right)=\cosh(x)\dfrac{d}{dx}\left(\dfrac{du}{dr}\right)+\sinh(x)\dfrac{du}{dr}=\cosh(x)\dfrac{d}{dr}\left(\dfrac{du}{dr}\right)\dfrac{dr}{dx}+\sinh(x)\dfrac{du}{dr}=\cosh(x)\dfrac{d^2u}{dr^2}\cosh(x)+\sinh(x)\dfrac{du}{dr}=\cosh^2(x)\dfrac{d^2u}{dr^2}+\sinh(x)\dfrac{du}{dr}$
$\therefore\sinh^2(x)\left(\cosh^2(x)\dfrac{d^2u}{dr^2}+\sinh(x)\dfrac{du}{dr}\right)+\cosh^2(x)\sinh(x)\dfrac{du}{dr}+(\lambda-\sinh^2(x))u=0$
$\sinh^2(x)\cosh^2(x)\dfrac{d^2u}{dr^2}+\sinh(x)(\sinh^2(x)+\cosh^2(x))\dfrac{du}{dr}+(\lambda-\sinh^2(x))u=0$
$r^2(r^2+1)\dfrac{d^2u}{dr^2}+r(2r^2+1)\dfrac{du}{dr}+(\lambda-r^2)u=0$
Let $s=r^2$ ,
Then $\dfrac{du}{dr}=\dfrac{du}{ds}\dfrac{ds}{dr}=2r\dfrac{du}{ds}$
$\dfrac{d^2u}{dr^2}=\dfrac{d}{dr}\left(2r\dfrac{du}{ds}\right)=2r\dfrac{d}{dr}\left(\dfrac{du}{ds}\right)+2\dfrac{du}{ds}=2r\dfrac{d}{ds}\left(\dfrac{du}{ds}\right)\dfrac{ds}{dr}+2\dfrac{du}{ds}=2r\dfrac{d^2u}{ds^2}2r+2\dfrac{du}{ds}=4r^2\dfrac{d^2u}{ds^2}+2\dfrac{du}{ds}$
$\therefore r^2(r^2+1)\left(4r^2\dfrac{d^2u}{ds^2}+2\dfrac{du}{ds}\right)+2r^2(2r^2+1)\dfrac{du}{ds}+(\lambda-r^2)u=0$
$4r^4(r^2+1)\dfrac{d^2u}{ds^2}+2r^2(3r^2+2)\dfrac{du}{ds}+(\lambda-r^2)u=0$
$4s^2(s+1)\dfrac{d^2u}{ds^2}+2s(3s+2)\dfrac{du}{ds}+(\lambda-s)u=0$
$\dfrac{d^2u}{ds^2}+\dfrac{3s+2}{2s(s+1)}\dfrac{du}{ds}+\dfrac{\lambda-s}{4s^2(s+1)}u=0$
$\dfrac{d^2u}{ds^2}+\left(\dfrac{1}{s}+\dfrac{1}{2(s+1)}\right)\dfrac{du}{ds}+\left(\dfrac{\lambda}{4s^2}-\dfrac{\lambda+1}{4s(s+1)}\right)u=0$
Let $u=s^av$ ,
Then $\dfrac{du}{ds}=s^a\dfrac{dv}{ds}+as^{a-1}v$
$\dfrac{d^2u}{ds^2}=s^a\dfrac{d^2v}{ds^2}+as^{a-1}\dfrac{dv}{ds}+as^{a-1}\dfrac{dv}{ds}+a(a-1)s^{a-2}v=s^a\dfrac{d^2v}{ds^2}+2as^{a-1}\dfrac{dv}{ds}+a(a-1)s^{a-2}v$
$\therefore s^a\dfrac{d^2v}{ds^2}+2as^{a-1}\dfrac{dv}{ds}+a(a-1)s^{a-2}v+\left(\dfrac{1}{s}+\dfrac{1}{2(s+1)}\right)\left(s^a\dfrac{dv}{ds}+as^{a-1}v\right)+\left(\dfrac{\lambda}{4s^2}-\dfrac{\lambda+1}{4s(s+1)}\right)s^av=0$
$\dfrac{d^2v}{ds^2}+\dfrac{2a}{s}\dfrac{dv}{ds}+\dfrac{a(a-1)}{s^2}v+\left(\dfrac{1}{s}+\dfrac{1}{2(s+1)}\right)\dfrac{dv}{ds}+\left(\dfrac{a}{s^2}+\dfrac{a}{2s(s+1)}\right)v+\left(\dfrac{\lambda}{4s^2}-\dfrac{\lambda+1}{4s(s+1)}\right)v=0$
$\dfrac{d^2v}{ds^2}+\left(\dfrac{2a+1}{s}+\dfrac{1}{2(s+1)}\right)\dfrac{dv}{ds}+\left(\dfrac{4a^2+\lambda}{4s^2}+\dfrac{2a-\lambda-1}{4s(s+1)}\right)v=0$
Choose $4a^2+\lambda=0$ , i.e. $a=\pm i\dfrac{\sqrt\lambda}{2}$ , the ODE becomes
$\dfrac{d^2v}{ds^2}+\left(\dfrac{\pm i\sqrt\lambda+1}{s}+\dfrac{1}{2(s+1)}\right)\dfrac{dv}{ds}+\dfrac{\pm i\sqrt\lambda-\lambda-1}{4s(s+1)}v=0$
Which reduces to Gaussian hypergeometric equation.
Best Answer
Just let $y = \cosh x$. Then
$$\frac{d}{dx}\frac{d(\cosh x)}{dx} = \frac{d}{dx} \sinh x = ?$$
Similarly for $\sinh x$.