The "topograph" for $x^2 - 13 y^2$ is definitely more complicated than the previous ones, because the continued fraction for $\sqrt {13}$ has period 5, your two previous examples had period 1. Confirming the "automorph" matrix, which just preserves the quadratic form:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
gp-pari
?
?
? form = [ 1,0; 0,-13]
%1 =
[1 0]
[0 -13]
?
? a = [649, 2340; 180, 649]
%2 =
[649 2340]
[180 649]
?
? atranspose = mattranspose(a)
%3 =
[649 180]
[2340 649]
?
? atranspose * form * a
%5 =
[1 0]
[0 -13]
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
The pairs of numbers in green are vectors in the plane. Two basic properties. First, each shows its value for $x^2 - 13 y^2.$ For example, in the first occurrence of 4, we see the (column) vector $(11,3),$ and we can easily confirm that $11^2 - 13 \cdot 3^2 = 4. $ Next, around any point where three purple line segments meet (even if two are parallel), one of the three green vectors is the sum of the other two. For example, $$ (4,1) + (7,2) = (11,3). $$
As long as we just continue to the right, we can continue getting all positive entries in green.
Oh: you said you can do continued fractions. It happens that you can find all representations of 4 and 1 using the continued fraction of $\sqrt {13},$ so you can confirm a good deal of the Conway diagram, the vectors in green, whatever.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
jagy@phobeusjunior:~/old drive/home/jagy/Cplusplus$ ./indefCycle
Input three coefficients a b c for indef f(x,y)= a x^2 + b x y + c y^2
1 0 -13
0 form 1 0 -13 delta 0
1 form -13 0 1 delta 3
2 form 1 6 -4
-1 -3
0 -1
To Return
-1 3
0 -1
0 form 1 6 -4 delta -1
1 form -4 2 3 delta 1
2 form 3 4 -3 delta -1
3 form -3 2 4 delta 1
4 form 4 6 -1 delta -6
5 form -1 6 4 delta 1
6 form 4 2 -3 delta -1
7 form -3 4 3 delta 1
8 form 3 2 -4 delta -1
9 form -4 6 1 delta 6
10 form 1 6 -4
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Multiplying with 4 (I first completed the square and then multiplied to get rid of fractions), the equation becomes
$$(2y+1)^2-(2x+21)^2+456=0$$
Thus
$$(2x+21)^2-(2y+1)^2=456$$
Factor the LHS
$$(2x-2y+20)(2x+2y+22)=456\\
(x-y+10)(x+y+11)=114$$
Now factor $114=2*3*19$ in all the $24$ possible ways each of them gives you exactly one solution.
P.S. The solution $x=19, y=27$ corresponds to
$$2 * 57=114$$
The solution $x=19, y=-28$ corresponds to
$$57 * 2=114$$
P.S. In general, by this method any equation of the form
\begin{align*}
y^2+ay-x^2-bx+c=0
\end{align*}
Can be reduced to an equation of teh form
\begin{align*}
(2y+a)^2-(2x+b)^2=\alpha
\end{align*}
which by factoring the LHS leads to finitely many solutions.
Equations of the form
\begin{align*}
y^2+ay-dx^2-bx+c=0
\end{align*}
can by multiplication by $4d^2$ be reduced to the Pell equation.
Best Answer
Here’s one way you might move forward with your approach.
You found that \begin{align} \tag{$\star$} 4x^3-27x^2+44x-12 &= (x-2)(4x^2-19x+6) \end{align} must be a square. Note that $$4x^2-19x+6 = (4x-11)(x-2)-16.$$ This can be handled in two cases: $x$ odd [in which case the absolute value of each factor on the right-hand side of $(\star)$ must be the square of an integer], and $x$ even [where the two factors have a common factor].
I hope that’s enough of a hint without solving the problem for you!