Since
$$
\frac{\binom nk}{k+1}=\frac{\binom{n+1}{k+1}}{n+1}\;,
$$
we have
\begin{eqnarray}
\sum_{k=0}^n\frac{(-1)^k\binom nk^2}{(k+1)^2}
&=&
\frac1{(n+1)^2}\sum_{k=0}^n(-1)^k\binom{n+1}{k+1}^2
\\
&=&
\frac1{(n+1)^2}\left(1-\sum_{k=0}^{n+1}(-1)^k\binom{n+1}k^2\right)\;.
\end{eqnarray}
With
$$
\sum_{k=0}^nq^k\binom nk^2=(1-q)^nP_n\left(\frac{1+q}{1-q}\right)\;,
$$
where $P_n$ is the $n$-th Legendre polynomial, this is
$$
\frac1{(n+1)^2}\left(1-2^{n+1}P_{n+1}(0)\right)\;,
$$
where
$$
P_l(0)=
\begin{cases}
\frac{(-1)^m}{4^m}\binom{2m}m&l=2m\\0&l=2m+1
\end{cases}
$$
(see Wikipedia).
The factor $4^m$ cancels, so the result is
$$
\sum_{k=0}^n\frac{(-1)^k\binom nk^2}{(k+1)^2}=
\begin{cases}
\frac{1-(-1)^m\binom{2m}m}{(2m)^2}&n=2m-1\;,\\
\frac{1}{(2m+1)^2}&n=2m\;.
\end{cases}
$$
An expression of this double sum in terms of a special function can be obtained as
\begin{align}
f(t,r)&=\sum_{k=0}^{\infty} \sum_{m=0}^{\infty} r^m t^k \binom{m+k}{k} \binom{m+k+1}{k} \\
&=\sum_{k=0}^{\infty} \sum_{m=0}^{\infty} \frac{(1)_{m+k}(2)_{m+k}}{(1)_{k}(2)_{m}}\frac{ t^kr^m}{k!m!}
\end{align}
Using the definition of the fourth Appell function
\begin{equation}
{F_{4}}\left(\alpha,\beta;\gamma,\gamma^{\prime};x,y\right)=\sum_{m,n=0}^{%
\infty}\frac{{\left(\alpha\right)_{m+n}}{\left(\beta\right)_{m+n}}}{{\left(%
\gamma\right)_{m}}{\left(\gamma^{\prime}\right)_{n}}m!n!}x^{m}y^{n}
\end{equation}
valid for $\sqrt{\left|x\right|}+\sqrt{\left|y\right|}<1$,
we identify
\begin{equation}
f(t,r)={F_{4}}\left(1,2;1,2;t,r\right)
\end{equation}
when $\sqrt{\left|r\right|}+\sqrt{\left|t\right|}<1$. This function cannot be expressed as the product of two hypergeometric functions, in general. A list of its properties (including the OP expressions (2) and (3)) can be found in an article by Brychkov and Saad (under a paywall).
Best Answer
Use Binomial identity: $$(1+t)^n=\sum_{k=0}^{n} {n \choose k}t^n~~~(1)$$ Integration of (1) from $t=0$ to $t=x$,gives $$\frac{(1+x)^{n+1}-1}{n+1}= \sum_{k=0}^n {n \choose k}\frac{x^{k+1}}{k+1}~~~(2)$$ Let $t=-1/x$ in (1), then $$(-1)^n x^{-n} (1-x)^n=\sum_{k=0}^{n} (-1)^k {n \choose k} x^{-k}~~~~(3)$$ Multiplying (2) and (3) and collecting the terms of $x^1$, we get $$\frac{(-1)^n}{n+1} [(1-x^2)^{n}(1+x)-(1-x)^n]= x^n\sum_{k=0}^{n} \frac{(-1)^k {n \choose k}^2}{k+1} x^1+...+...$$ $$\implies S_n=\sum_{k=0}^n \frac{(-1)^k {n \choose k}^2}{k+1}=[x^{n+1}] \left((-1)^n \frac{ (1-x^2)^{n}(1+x)-(1-x)^n}{n+1}\right)$$ if $m=n/2]$, then $$S_n=(-1)^{m} \frac{{n \choose m}}{n+1}.$$