The usual way to obtain a generating function from the Catalan number is to start with a different recurrence: $$C_n = C_{n-1} C_0 + C_{n-2} C_1 + \dots + C_1 C_{n-2} + C_0 C_{n-1},$$ valid when $n\ge 1$. (If $g(x)$ is the generating function for the sequence $(C_n)$, then the right-hand side is a coefficient of $g(x)^2$.)
The recurrence relation you've found is true but not very well-known, so I'm assuming the person who wrote the problem you're trying to solve didn't expect you to solve it this way.
If you do intend to go on, you can rewrite the recurrence as $(n+2)C_{n+1} = (4n+2)C_n$ and use it to get a differential equation for $g(x)$, using the fact that $$g(x) = \sum_{n \ge 0} C_n x^n \implies g'(x) = \sum_{n \ge 0} n C_n x^{n-1}.$$ The differential equation I get by doing this is $(4x^2-x)g'(x) + (2x-1)g(x) + 1 = 0$, which (together with the initial condition $g(0)=1$) does indeed produce the generating function of the Catalan numbers.
I find it helpful to rewrite the identity as
$$\begin{align*}
C_{n+1}&=\sum_{k\ge 0}(-1)^k\binom{n+1-k}{k+1}C_{n-k}\tag{1}\\
&=(n+1)C_n-\binom{n}2C_{n-1}+\binom{n-1}3C_{n-2}-+\ldots\;;
\end{align*}$$
No upper limit is needed on the summation, since the binomial coefficients are eventually $0$.
$C_n$ is the number of rooted binary trees with $n$ nodes; each of these has $n+1$ open slots at which a leaf can be added. Thus, to build one of the $C_{n+1}$ trees on $n+1$ nodes we can add a leaf in any one of $n+1$ places to any of the $C_n$ trees on $n$ nodes. This produces $(n+1)C_n$ trees on $n+1$ nodes, but some of them are duplicates.
For example, consider one of the $C_{n-1}$ trees on $n-1$ nodes. There are $\binom{n}2$ ways to pick two of the vacant slots at which a leaf can be added, and each such pair produces a tree on $n+1$ nodes in two ways, one for each of the two possible orders in which the two leaves can be added. Thus, the figure $(n+1)C_n$ overcounts by $\binom{n}2C_{n-1}$, and $(n+1)C_n-\binom{n}2C_{n-1}$ is a better estimate of $C_{n+1}$.
This analysis rapidly gets a bit awkward, however. Instead of trying to pursue it directly, let $T$ be a binary tree on $n+1$ nodes, and let $\ell$ be the number of leaves of $T$. $T$ is an extension of $\ell$ different $n$-trees (i.e., binary trees on $n$ nodes) by addition of a single leaf. It is an extension of $\binom\ell2$ different $(n-1)$-trees by the simultaneous addition of $2$ leaves. And in general for $k=1,\ldots,\ell$ it is the extension of $\binom\ell k$ different $(n+1-k)$-trees by the simultaneous addition of $k$ leaves.
For $k=0,\ldots,\ell-1$ let $\mathscr{T}_k$ be the set of $(n-k)$-trees of which $T$ is an extension by simultaneous addition of leaves, so that $|\mathscr{T}_k|=\binom\ell{k+1}$. Each member of $\mathscr{T}_0$ is counted once in $(n+1)C_n$, the $k=0$ term of $(1)$. Each member of $\mathscr{T}_1$ is counted once in $\binom{n}2C_{n-1}$, the absolute value of the $k=1$ term of $(1)$. And in general each member of $\mathscr{T}_k$ is counted once in $\binom{n+1-k}{k+1}C_{n-k}$. Thus, the summation in $(1)$ counts $T$
$$\sum_{k=0}^{\ell-1}(-1)^k\binom\ell{k+1}=\sum_{k=0}^{\ell}(-1)^{k+1}\binom\ell k-\left(-\binom\ell0\right)=0+1=1$$
time. This is the case for each binary tree on $n+1$ nodes, so the summation must indeed yield $C_{n+1}$.
Best Answer
You have made great progress so far, here is how you conclude. We want to derive a formula for $M_n$, where $M(x)=\sum_{n\ge 0}M_nx^n$. For any $n\ge 0$, $$ \begin{align} M_n &=[x^n]\frac{1-x-\sqrt{1-2x-3x^2}}{2x^2} \\&=[x^n]\frac{-\sqrt{1-2x-3x^2}}{2x^2} \\&=-\frac12[x^{n+2}](1-2x-3x^2)^{1/2}. \end{align} $$ As you calculated, $(1-2x-3x^2)^{1/2}=\sum_{k\ge 0}\binom{1/2}k(-2)^k\cdot x^k(1+\tfrac32x)^k$. Therefore, $$ \begin{align} M_n&=-\frac12[x^{n+2}](1-2x-3x)^{1/2} \\&=\sum_{k\ge 0}\binom{1/2}k(-2)^{k-1}[x^{n+2}]x^k(1+\tfrac32x)^k \\&=\sum_{k\ge 0}\binom{1/2}k(-2)^{k-1}\binom{k}{n+2-k}(3/2)^{n+2-k} \\&=(3/2)^{n+2}\sum_{k\ge 0}\color{#00a}{\binom{1/2}k(-1)^{k-1}2^{-(2k-1)}}\binom{k}{n+2-k}3^{-k}, \\M_n&=(3/2)^{n+2}\sum_{k\ge 1}\color{#00a}{\frac1k\binom{2k-2}{k-1}}\binom{k}{n+2-k}3^{-k}. \end{align} $$ This formula appears in the OEIS page for the Motzkin numbers.
You asked for alternate methods, so here is a different method which gives a different formula.
Recall that you can use a combinatorial argument to show that the Catalan generating function satisfies the equation $C(x)=1+xC(x)^2$, which is where the $C(x)=(1-\sqrt{1-4x})/2x$ formula comes from. In the same way, a combinatorial argument implies the equation $$M(x)=1+xM(x)+(xM(x))^2.$$ Using this equation alone, together with the method of Lagrange inversion, we will derive a formula for the $n^\text{th}$ Motzkin number.
Let $f(x)=xM(x)$. The previous equation can be written $$ \frac{f(x)}{1+f(x)+f(x)^2}=x. $$ Now, let $g(x)=\frac x{1+x+x^2}$. The above equation says that $g(f(x))=x$, so $f(x)$ and $g(x)$ are functional inverses of each other, with $f(0)=g(0)=0$. Therefore, we can use the Lagrange inversion formula to extract the coefficient of $f(x)$.
$$ \begin{align} [x^n]f(x) &=\tfrac1n[x^{-1}]g(x)^{-n}, \\&= \tfrac1n [x^{-1}]\;x^{-n}(1+x+x^2)^n, \\&= \tfrac1n [x^{n-1}]\;(1-x^3)^n(1-x)^{-n}, \\&= \frac1n\sum_{k}\binom{n}{k}(-1)^{k}\cdot \binom{-n}{n-1-3k}(-1)^{n-1-3k}, \\&= \frac1n\sum_{k}\binom{n}{k}(-1)^{k}\cdot \binom{2n-2-3k}{n-1-3k}. \end{align} $$ Finally, since $M(x)=xf(x)$, the coefficient of $x^n$ in $M(x)$ is equal to the coefficient of $x^{n+1}$ in $f(x)$. Therefore, letting $M_n=[x^n]M(x)$, we have derived $$ \boxed{M_n=\frac1{n+1}\sum_{k=0}^{\lfloor n/3\rfloor }(-1)^k\binom{n+1}k \binom{2n-3k}{n}.} $$