I think your solution is basically correct.
Part (a)
To find the missing transformations of the momenta, we first try to find a generating function $\cal F_2(q, P)$ that generates the known transformations of the coordinates. Then, we use this generating function $\cal F_2(q, P)$ to compute the relations regarding the momenta.
The transformation of coordinates $Q^i = Q^i(q)$
can be conveniently generated by the generating function of type 2 as
\begin{align}
\cal F_2(q, P)
&=\sum_i P_i \, Q^i(q) + F(q),
\end{align}
where $F(q)$ is arbitrary function of $q$.
In this way, the requirement
\begin{align}
\frac{ \partial \cal F_2(q, P) }{ \partial P_i} = Q^i(q).
\end{align}
is automatically satisfied.
In our case
\begin{align}
\cal F_2(q, P)
&= P_1 \, Q^1(q^1, q^2)
+ P_2 \, Q^2(q^1, q^2)
- F \\
&= P_1 \, (q^1)^2
+ P_2 \, (q^1 + q^2)
- F,
\end{align}
where $F \equiv F(q^1, q^2)$ is an arbitrary function of $q^1$ and $q^2$.
So
\begin{align}
p_1 &= \frac{ \partial \cal F_2(q, P) }{ \partial q^1 } = 2 P_1 \, q^1 + P_2
- \frac{\partial F }{\partial q^1}, \\
p_2 &= \frac{ \partial \cal F_2(q, P) }{ \partial q^2 }
= P_2 - \frac{\partial F }{\partial q^2}.
\end{align}
Or
\begin{align}
P_1 &= \frac{1}{2q^1} \left(
p_1 + \frac{ \partial F } { \partial q^1 }
-p_2 - \frac{ \partial F } { \partial q^2 }
\right)
\tag{1}
\\
P_2 &= p_2 + \frac{ \partial F } { \partial q^2 }.
\tag{2}
\end{align}
Part (b)
Basically we need to find an $F$ such that $K$ matches $H$, because
$$
d{\cal F}_2 = p \, dq + Q dP + (K - H) \, dt,
$$
and our $F_2$ does not depend on time explicitly (so $K-H$ must vanish).
Now by the solution of part (a), we have
\begin{align}
H &= \left( \frac{p_1 - p_2}{2q^1} \right)^2 + p_2 + (q^1 + q^2)^2, \\
K &= P_1^2 + P_2 \\
&= \left( \frac{p_1 - p_2 + \partial F/\partial q^1 - \partial F/\partial q^2}{2q^1} \right)^2 + p_2 + \partial F / \partial q^2.
\end{align}
It would be nice if
\begin{align}
\partial F/\partial q^1 &= \partial F/\partial q^2,
\\
\partial F/\partial q^2 &= (q^1 + q^2)^2.
\end{align}
A simple solution would be
\begin{align}
F = \frac{1}{3} (q^1 + q^2)^3.
\end{align}
Then Eq. (1) and (2) means
\begin{align}
P_1 &= \frac{1}{2q^1} \left(
p_1 - p_2
\right)
\tag{1}
\\
P_2 &= p_2 + (q^1 + q^2)^2.
\tag{2}
\end{align}
The result $P_1 = (p_1 + p_2)/(2q^1)$ doesn't make sense, because it implies $\dot P_1 \ne 0 = -\partial K/\partial Q^1$.
So the plus sign might be a typo.
Here is how I understand it: let's say you have some coordinates $p_i$ and some momenta $q_i$. You want to find transformations
$$Q_i\equiv Q_i(q_i,\,...,\,q_n,\,p_i,...,\,p_n,\,t)\qquad P_i\equiv P_i(q_i,\,...,\,q_n,\,p_i,...,\,p_n,\,t) $$
These variables $p_i$, $q_i$, $Q_i$, and $P_i$ must satisfy the Hamilton's equations of motion:
$$\dot{q}_i=\frac{\partial H}{dp_i},\quad \dot{p}_i=-\frac{\partial H}{dq_i},\quad \dot{Q}_i=\frac{\partial K}{dP_i},\quad \dot{P_i} = -\frac{\partial K}{dQ_i}$$
where $K$ is the transformed Hamiltonian $K\equiv K(Q,\,P,\,t)$.
To perform the canonical transformation, we may be able to introduce the function $F=F_1$, where
$$P_i \dot{Q}_i-K+\frac{dF_1}{dt}=p_i \dot{q}_i-H$$
or we could introduce $F=F_2-Q_iP_i$, and we thus have
$$-\dot{P}_i Q_i-K+\frac{dF_2}{dt}=p_i \dot{q}_i-H $$
Similarly for the generating functions for $F_3$ and $F_4$.
Now that we understand their differences, we have to ask how we use them. What you use is based upon what you know. If I want to find $F_1$, I integrate $p_i$ with respect to $q_i$ and $-P_i$ with respect to $Q_i$, and combine the result to find the whole value for $F_1$. A similar process follows for the identities for $F_2$, $F_3$, and $F_4$. Now, if I have the opposite problem--namely, I have $F_1$, and I want to find the coordinates and/or momenta--then I take the partial with respect to $q_i$ to find $p_i$ and the partial with respect to $Q_i$, take the negative, and I find $P_i$.
If I understand your question correctly, you have a set of $Q$'s and $P$'s, and you want to find $F$. You can pick whatever $F$ you want--you just need to change the definition of the generating function as given above.
Hope this helps. These slides might also be of assistance:
http://users.physics.harvard.edu/~morii/phys151/lectures/Lecture20.pdf
Best Answer
Since you know the new $q(Q)$, it’s easier to use another generating function $F(p,Q)=pq(Q)$ (Legendre transform of your function) which gives: $$ q = \frac{\partial F}{\partial p}(=q) \\ P = \frac{\partial F}{\partial Q}(=p \frac{\partial q}{\partial Q}) \\ $$ so all you have to is invert the jacobian according to the second equation.
Note that in this case, the Hamiltonian formalism is clearly overkill, as you could have found the result using Lagrangian formalism. The convenience of generating functions is more apparent when there is a more intricate mixing of the variables in $F$.
Hope this helps and tell me if you find a mistake.
Edit (detail of the calculation):
The second equation gives the system: $$ P_1 = p_1\frac{\partial q_1}{\partial Q_1}+p_2\frac{\partial q_2}{\partial Q_1} \\ = p_1+p_2\\ P_2 = p_1/2-p_2/2 $$
which is a linear system that you can easily solve (the matrix that you are inverting is precisely the jacobian of the transformation $Q\to q$, which you could have figured out using Lagrangian formalism):
$$ p_1 = P_1/2+P_2 \\ p_2 = P_1/2-P_2 $$
thus completing your coordinate change.
Finally, plugging it back in your Hamiltonian: $$ H = \frac{(p_1+p_2)^2}{4}+\frac{(p_1-p_2)^2}{8}+V(q_1-q_2) $$ which worsened your expression, normally, you would go the other way round, I guess it was proposed for pedagogic purposes.