You're on the right track. I would view things as follows.
Let $Q=f(q,p)$ and $p=g(q,Q)$. Then, we have both
$$dQ=f_1(q,p)dq+f_2(q,p)dp \tag 1$$
and
$$dp=g_1(q,Q)dq+g_2(q,Q)dQ \tag 2$$
Substitution of $dQ$ as given in $(1)$ into $(2)$ yields
$$\begin{align}
dp&=g_1(q,Q)dq+g_2(q,Q)\left(f_1(q,p)dq+f_2(q,p)dp \right)\\\\
&=\left(g_1(q,Q)+g_2(q,Q)f_1(q,p)\right)dq +f_2(q,p)g_2(q,Q)\,dp\tag 3
\end{align}$$
Inasmuch as the differential terms on both sides of $(3)$ must be equal, we obtain
$$\bbox[5px,border:2px solid #C0A000]{f_2(q,p)g_2(q,Q)=\frac{\partial Q}{\partial p}\times \frac{\partial p}{\partial Q}=1}$$
which was to be shown along with
$$g_1(q,Q)+g_2(q,Q)f_1(q,p)=\frac{\partial p}{\partial q}+\frac{\partial p}{\partial Q}\times \frac{\partial Q}{\partial q}=0$$
The notation here is not very clear, so I have slightly changed it:
Let $X$ and $Y$ be the functions $X(s,t)=e^{2s}\cos(5t)$ and $Y(s,t)=e^{2s}\sin(5t)$, and $u$ be the function
$$u(s,t)=f[X(s,t),Y(s,t)]$$
What you have been asked to find is the $g$ and $h$ in:
$$f_x^2+f_y^2=g(s,t)u_s^2+h(s,t)u_t^2 \tag{1}$$
where I have suppressed the arguments of all the derivatives.
Here
$$u_s(s,t)=f_xX_s+f_yY_s\qquad \text{and}\qquad u_t(s,t)=f_xX_t+f_yY_t$$
where
$$X_s(s,t)=2e^{2s}\cos(5t)=2X\qquad \text{and}\qquad Y_s(s,t)=2e^{2s}\sin(5t)=2Y$$
while
$$X_t(s,t)=-5e^{2s}\sin(5t)=-5Y\qquad \text{and}\qquad Y_t(s,t)=5e^{2s}\cos(5t)=5X$$
So
$$u_s(s,t)=2(f_xX+f_yY)\qquad \text{and}\qquad u_t(s,t)=-5(f_xY-f_yX)$$
You can solve this system of equations to get
$$f_x=\frac{5Xu_s-2Yu_t}{10(X^2+Y^2)}\qquad \text{and}\qquad \frac{2Xu_t+5Yu_s}{10( Y^{2}+X^{2})}$$
Then
$$f_x^2+f_y^2=\frac{25u_s^2+4u_t^2}{100(X^2+Y^2)}=\frac{25u_s^2+4u_t^2}{100e^{4s}}$$
and so $g(s,t)=\frac{1}{4e^{4s}}$ and $h(s,t)=\frac{1}{25e^{4s}}$.
I suspect there is probably a quicker way...
Best Answer
These are known as the reciprocal and reciprocity theorems. I feel these are perhaps not stated as clearly as they should be in introductory courses. I always had trouble with them until I read Andrew Steane's fantastic book on Thermodynamics (pretty much the only book on the subject that I like), which I will be following in this answer.
The idea is to start with the well known theorem of partial derivatives. If $f$ is a function of two variables $x$ and $y$, then we can relate a change in $f$ to changes in $x$ and $y$ using the partial derivatives: $$\text{d}f = \left(\frac{\partial f}{\partial x}\right)_{y} \text{d}x + \left(\frac{\partial f}{\partial y}\right)_{x} \text{d}y.\label{1}\tag{1}$$
In the above formula, the notation $(\cdot)_x$ means that the partial derivative is taken while keeping the subscript (here, $x$) constant. Note that in general (I encourage you to try to come up with an example for this)$$\left(\frac{\partial f}{\partial x}\right)_{y} \neq \left(\frac{\partial f}{\partial x}\right)_{z}.$$
We can now use the above relation carefully, while realising that we can't quite be as cavalier with partial derivatives as we could with total derivatives. For example, we can't talk about quantities like $\text{d}f/\text{d}x$, since $f$ is a function of two variables, $x$ and $y$. You might naively call the quantity in question the "rate of change of $f$ with $x$", but since it depends on $y$ as well, we need to specify along which path we choose to describe this variation completely.
The Reciprocal Theorem:
let's start by dividing Equation (\ref{1}) by $\text{d}x$, while keeping $y$ constant. The second term in the equation would trivially be zero, since $\text{d}y=0$, and we'd get the trivial result: $$\left(\frac{\partial f}{\partial x}\right)_{y} = \left(\frac{\partial f}{\partial x}\right)_{y}.$$
A more interesting result appears if we carefully divide Equation (\ref{1}) by $\text{d}f$, but we still keep $y$ constant. In this case, it reduces to: $$1 = \left(\frac{\partial f}{\partial x}\right)_{y} \left(\frac{\partial x}{\partial f}\right)_{y},$$ from which you get the relation
$$\boxed{\left(\frac{\partial f}{\partial x}\right)_{y} = \left(\frac{\partial x}{\partial f}\right)_{y}^{-1}}.$$
The Reciprocity Theorem:
Let's go back to Equation (\ref{1}) again, and this time we divide throughout by $\text{d}x$ again, except we don't keep $y$ constant, but rather we do it while keeping $f$ constant. It should be easy to show that $$0 = \left(\frac{\partial f}{\partial x}\right)_{y} + \left(\frac{\partial f}{\partial y}\right)_{x} \left(\frac{\partial y}{\partial x}\right)_{f}.$$
Using the reciprocal theorem above, we can easily see that this means that $$\boxed{\left(\frac{\partial f}{\partial y}\right)_{x} \left(\frac{\partial y}{\partial x}\right)_{f} \left(\frac{\partial x}{\partial f}\right)_{y} = -1.}$$