The hyperbola given by the equation $xy=c^2>0$ is your prime example of a rectangular hyperbola. Let's rotate it by some arbitrary angle $\phi$! This rotation maps points $(x,y)$ to new points $(x',y')$, whereby
$$(x,y)^\top=\left[\matrix{\cos\phi&-\sin\phi\cr\sin\phi&\cos\phi\cr}\right](x',y')^\top\ .$$
This means that the coordinates of the preimage $(x,y)$ are obtained from the coordinates $(x',y')$ of the image by
$$x=\cos\phi\> x'-\sin\phi\> y',\qquad y=\sin\phi\> x'+\cos\phi\> y'\ .$$
If $(x,y)$ is lying on the model hyperbola then $xy=c^2$. This implies that the coordinates of the image point $(x',y')$ have to satisfy
$$(\cos\phi\> x'-\sin\phi\> y')(\sin\phi\> x'+\cos\phi\> y')=c^2\ .$$
This can be rewritten as
$$\cos\phi\sin\phi(x'^2-y'^2)+(\cos^2\phi-\sin^2\phi)x'y'=c^2\ .$$
Leaving away the primes here gives the result of the question.
$$\Rightarrow (x^2-y^2)\sin\theta+2xy\cos\theta=2c^2$$
As pointed out in the comment by Jan-MagnusĂkland, solving the problem requires using the property that the axis bisects the angle between the two asymptotes of the hyperbola. We are given one asymptote $y=0$ thus we can find the second one by reflecting $y=0$ across the axis $y=2x+2$. Let's rewrite it in the standard form $2x-y+2=0$.
Let's pick the point $(0,0)$ to reflect. The distance between the axis and the point $(0,0)$ is $$d=\left| \frac{Ax_0 + By_0 +C}{\sqrt{A^2 + B^2}} \right| = \left| \frac{2x_0 -y_0 +2}{\sqrt{(2)^2 + (-1)^2}} \right| = \left| \frac{2\cdot0 -1\cdot0 +2}{\sqrt{5}} \right| = \left| \frac{2}{\sqrt{5}} \right|$$
Therefore, the distance from the symmetric point $(x_0, y_0)$ is also $\frac{2}{\sqrt{5}}$. This gives the equation:
$$ \left| \frac{2x_0 -y_0 +2}{\sqrt{5}} \right| = \frac{2}{\sqrt{5}} \implies \left| {2x_0 -y_0 +2} \right| = 2 \qquad \qquad \qquad (*)$$
Since the point symmetric to $(0,0)$ with respect to the axis $2x-y+2=0$ lies on the perpendicular to $2x-y+2=0$, we can find the gradient of the perpendicular $$ m_1m_2=-1 \implies m_2=\frac{-1}2$$
Thus, together with $(*)$ we have our second (or third) equation $$\frac{y_2-y_1}{x_2-x_1} = \frac{y_0 - 0}{x_0 - 0} = \frac{-1}2 \implies x_0 = -2y_0 \qquad \qquad \qquad (**)$$.
The equation in $(*)$ gives us two cases:
$$\left| {2x_0 -y_0 +2} \right| = 2 \implies \begin{cases}
2x_0 -y_0 +2 = 2,\\
2x_0 -y_0 +2 = -2
\end{cases}$$
Solving the fist case of $(*)$ simultaneously with $(**)$, we get $(x_0,y_0)$ = $(0,0)$ which is our initial point. So the second case gives us our symmetric point:
$$\begin{cases}
2x_0 -y_0 +2 = -2\\
x_0 = -2y_0
\end{cases} \implies (x_0,y_0) = \left(\frac{-8}{5}, \frac45 \right)$$
That gives us the reflection of the point $(0,0)$ across the line $2x-y+2=0$.Next, we know that the asymptote intersects the axis at the centre of the hyperbola which gives us $c=(-1,0)$. Using $c$ and the symmetric point we just calculated, we find that the equation of the second asymptote is $$y=\frac{-4}{3}x - \frac43 \equiv 3y+4x+4=0 $$
Using the property that the equation of a hyperbola can be given by its asymptotes $$(Ax+By+C)(A_1x+B_1y+C_1)=k$$
We have $$y(3y+4x+4)=k$$
Since the point $(1,1)$ lies on the hyperbola, we get that $k=11$ giving the final answer to be $$4xy+3y^2+4y-11=0$$
Best Answer
Comparing the given quadratic with $$Ax^2+2Hxy+By^2+2Gx+2Fy+C=0~~~(1)$$ We get $$A=2, H=3/2, B=-2, G=-3, F=13/2, C=-36~~~~(2)$$ The combined eq. of the asymptotes of the given hyperbola would be $$2x^2+3xy-2y^2-6x+13y+D=0~~~(3)$$, provided (3) represents a pair of straight line. The condition for this is $$ABD+2FGH-AF^2-BG^2-DH^2=0~~~~(3)$$ $$\implies -4D-117/2-169/2+18-9D/4=0 \implies D=-20.~~~~(4)$$ So the combined Eq. of asymptotes is $$2x^2+3xy-2y^2-6x+13y-20=0 \implies~(x+2y-5)(2x-y+4)=0~~(5)$$ Finally, we get $\lambda=4.$