I think your solution is basically correct.
Part (a)
To find the missing transformations of the momenta, we first try to find a generating function $\cal F_2(q, P)$ that generates the known transformations of the coordinates. Then, we use this generating function $\cal F_2(q, P)$ to compute the relations regarding the momenta.
The transformation of coordinates $Q^i = Q^i(q)$
can be conveniently generated by the generating function of type 2 as
\begin{align}
\cal F_2(q, P)
&=\sum_i P_i \, Q^i(q) + F(q),
\end{align}
where $F(q)$ is arbitrary function of $q$.
In this way, the requirement
\begin{align}
\frac{ \partial \cal F_2(q, P) }{ \partial P_i} = Q^i(q).
\end{align}
is automatically satisfied.
In our case
\begin{align}
\cal F_2(q, P)
&= P_1 \, Q^1(q^1, q^2)
+ P_2 \, Q^2(q^1, q^2)
- F \\
&= P_1 \, (q^1)^2
+ P_2 \, (q^1 + q^2)
- F,
\end{align}
where $F \equiv F(q^1, q^2)$ is an arbitrary function of $q^1$ and $q^2$.
So
\begin{align}
p_1 &= \frac{ \partial \cal F_2(q, P) }{ \partial q^1 } = 2 P_1 \, q^1 + P_2
- \frac{\partial F }{\partial q^1}, \\
p_2 &= \frac{ \partial \cal F_2(q, P) }{ \partial q^2 }
= P_2 - \frac{\partial F }{\partial q^2}.
\end{align}
Or
\begin{align}
P_1 &= \frac{1}{2q^1} \left(
p_1 + \frac{ \partial F } { \partial q^1 }
-p_2 - \frac{ \partial F } { \partial q^2 }
\right)
\tag{1}
\\
P_2 &= p_2 + \frac{ \partial F } { \partial q^2 }.
\tag{2}
\end{align}
Part (b)
Basically we need to find an $F$ such that $K$ matches $H$, because
$$
d{\cal F}_2 = p \, dq + Q dP + (K - H) \, dt,
$$
and our $F_2$ does not depend on time explicitly (so $K-H$ must vanish).
Now by the solution of part (a), we have
\begin{align}
H &= \left( \frac{p_1 - p_2}{2q^1} \right)^2 + p_2 + (q^1 + q^2)^2, \\
K &= P_1^2 + P_2 \\
&= \left( \frac{p_1 - p_2 + \partial F/\partial q^1 - \partial F/\partial q^2}{2q^1} \right)^2 + p_2 + \partial F / \partial q^2.
\end{align}
It would be nice if
\begin{align}
\partial F/\partial q^1 &= \partial F/\partial q^2,
\\
\partial F/\partial q^2 &= (q^1 + q^2)^2.
\end{align}
A simple solution would be
\begin{align}
F = \frac{1}{3} (q^1 + q^2)^3.
\end{align}
Then Eq. (1) and (2) means
\begin{align}
P_1 &= \frac{1}{2q^1} \left(
p_1 - p_2
\right)
\tag{1}
\\
P_2 &= p_2 + (q^1 + q^2)^2.
\tag{2}
\end{align}
The result $P_1 = (p_1 + p_2)/(2q^1)$ doesn't make sense, because it implies $\dot P_1 \ne 0 = -\partial K/\partial Q^1$.
So the plus sign might be a typo.
Best Answer
Cool question!
Thanks to user lionelbrits for his answer that prompted me to pull out my mechanics books and check the definitions of "canonical transformation" given by different authors.
If you look in Goldstein's classical mechanics texts in the section on canonical transformations, then you'll find that canonical transformations are essentially defined as follows (I paraphrase)
This is essentially the definition given by lionelbrits in his answer.
On the other hand, if you look, for example, in Spivak's mechanics text, then you'll find the following definition:
In more concrete terms (namely in canonical coordinates), Spivak's definition can be stated as follows:
The transformation $f(q,p) = (f^q(q,p), f^p(q,p))$ is canonical if and only if its Jacobian (derivative) matrix preserves the symplectic matrix $J$, namely \begin{align} f'(p,q)\,J\,f'(p,q)^t = J \end{align} where \begin{align} J=\begin{pmatrix} 0 & I_n \\ -I_n & 0 \\ \end{pmatrix},\qquad f' = \begin{pmatrix} \frac{\partial f^q}{\partial q} & \frac{\partial f^q}{\partial p} \\ \frac{\partial f^p}{\partial q} & \frac{\partial f^p}{\partial p} \\ \end{pmatrix} \end{align} where $2n$ is the dimension of phase space and $I_n$ is the $n\times n$ identity matrix.
It also turns out that
but the converse is not true. In fact, this example you brink up is a counterexample to the converse! What lionelbrit showed in his answer is that the example you have written is a canonical transformation in the sense of Goldstein, but, as you should try to convince yourself (I did), the function $K = H\circ f^{-1}$ that you wrote down by inverting the transformation and plugging back into $H$ leads to Hamilton's equations that are not satisfied by $(Q(t), P(t)) = f(q(t), p(t))$. You can show this directly by writing down the equations of motion. You can also show this by computing the Jacobian of the transformation and showing that it does not preserve the symplectic matrix. In fact, you should find that the Jacobian is given by \begin{align} f'(q,p)=\begin{pmatrix} 1 & 0 \\ -\frac{1}{2\sqrt{q}} & \frac{1}{2\sqrt{p}} \\ \end{pmatrix} \end{align} and that \begin{align} f'(q,p) J f'(q,p)^t = \frac{1}{2\sqrt{p}} J \end{align} In other words, the Jacobian of the transformation preserves the symplectic matrix up to a multiplicative factor.
Speculation. I'm going to go out on a limb and guess that your professor calls Goldstein's definition a "local canonical transformation" and Spivak's definition a "canonical transformation." If we adopt this terminology, then it's clear from our remarks that the $K$ he gives shows that your example is a local canonical transformation, but that the transformation is not canonical.