$\xi$ and $\eta$ are null coordinates. In the $\xi,\eta$ coordinates the $1+1$ linear hyperbolic PDE is always equivalent to $u_{\xi\eta}(\xi,\eta) = 0$. (If you do the explicit change of coordinates, you will end up with a conformal factor $\Omega(\xi,\eta)$ multiplying $u_{\xi\eta}(\xi,\eta)$. But as long as your change of coordinates $(x,y) \to (\xi,\eta)$ is regular, the conformal factor $\Omega \neq 0$ and you can divide it away.)
That is sort of the point of the canonical form. Once you found the correct change of variables, you can just solve the equation $u_{\xi\eta} = 0$ and do the reverse coordinate transform at the end.
Why do you say that $B = 1/2$? If you identify the coefficients from the $PDE$:
$$Au_{xx} + B u_{xy} + C u_{yy} =0, \quad u = u(x,y), \quad (x,y)\in \mathbb{R}^2,$$
then you have $A = B = C = 1$ and the discriminant is:
$$\Delta = B^2-4AC = -3 < 0, $$
so the PDE is elliptic, as you already know. The characterstics are to be found from the equation:
$$A \xi_x^2 + B \xi_x \xi_y + C \xi_y^2 = 0,$$
so it yields (since $A,B,C$ are not zero):
$$\left( \frac{\xi_x}{\xi_y} \right)^2 + B \left( \frac{\xi_x}{\xi_y} \right) + C = 0,$$
and we have the characteristics given by:
$$\frac{\xi_x}{\xi_y} = -\frac{1}{2} \pm \frac{\sqrt{3}}{2} i = - \frac{dy}{dx}.$$
Integrating the equation above, you obtain the two (complex) characteristics as follows:
$$\xi = \phi_+ x + y, \quad \eta = \phi_- x + y,$$
where $\phi_{\pm}$ are the two different roots of the previous equation.
I hope this is useful to you.
Cheers!
Notice that the result that is expected to be obtained holds. Indeed, if you expand $\phi_+$ and $\phi_-$, you will obtain:
$$-x+2y+\sqrt{3} i\, x = 2 \xi = \overline{\xi}, \quad -x+2y - \sqrt{3}i \, x = 2 \eta = \overline{\eta},$$
where the overlined characteristics remain equally constant (even if you multiply them by $-1$).
Edit: notice that you can obtain two real characteristics by performing the following trick:
$$\begin{array}{l} \overline{\xi}+\overline{\eta} = 2x+4y \equiv u, \\ i(\overline{\xi} - \overline{\eta}) = -2 \sqrt{3}x \equiv v.\end{array} $$
Best Answer
The characteristic curves satisfy:
$$\frac{dy}{dx}=\frac{b\pm\sqrt{b^2-ac}}{a}$$
where $a,b,c$ are coefficients of $U_{xx}, U_{xy}, U_{yy}$, respectively.
With your example, you should get
$$\frac{dy}{dx}=\pm\sqrt{-xy}$$
Solving these two ODE's you should see the following change of variables:
$$\xi=y^{1/2}+\frac{1}{3}(-x)^{3/2}\\ \eta=y^{1/2}-\frac{1}{3}(-x)^{3/2}$$
In the integration, the right hand side with respect to $x$ is not in the denominator. It is $\sqrt{-x}$ instead.