$\xi$ and $\eta$ are null coordinates. In the $\xi,\eta$ coordinates the $1+1$ linear hyperbolic PDE is always equivalent to $u_{\xi\eta}(\xi,\eta) = 0$. (If you do the explicit change of coordinates, you will end up with a conformal factor $\Omega(\xi,\eta)$ multiplying $u_{\xi\eta}(\xi,\eta)$. But as long as your change of coordinates $(x,y) \to (\xi,\eta)$ is regular, the conformal factor $\Omega \neq 0$ and you can divide it away.)
That is sort of the point of the canonical form. Once you found the correct change of variables, you can just solve the equation $u_{\xi\eta} = 0$ and do the reverse coordinate transform at the end.
The equation
$$a(x,y)u_{xx}+2b(x,y)u_{xy}+c(x,y)u_{yy}=\Phi(x,y,u,u_x,u_y)\qquad(1)$$
is
$\mathbf {hyperbolic }\quad {\mathrm if}\quad b^2-ac>0$,
$\mathbf {parabolic }\quad {\mathrm if}\quad b^2-ac=0$,
$\mathbf {elliptic }\quad {\mathrm if}\quad b^2-ac<0$.
The charateristic equation
$$a\,dy^2-2b\,dx\,dy+c\,dx^2=0\qquad(2)$$
splits into two equations
$$a\,dy-(b+\sqrt{b^2-ac})\,dx=0,\qquad(3)$$
$$a\,dy-(b-\sqrt{b^2-ac})\,dx=0.\qquad(4)$$
Elliptic case. Let $\quad\phi(x,y) + i\psi(x,y)=c\quad$ solution of $(3)$ or $(4)$.
Then change variables
$$\xi=\phi(x,y),\quad\eta=\psi(x,y)$$
reduces equation $(1)$ to canonical form
$$u_{\xi\xi}+u_{\eta\eta}=\Phi_1(x,y,u,u_\xi,u_\eta).$$
In our case $a=1,\;b=1,\;c=5,\; b^2-ac=-4<0$. From $(4)$ we get
$$\mathit{dy}-\left( 1-2 i\right) \, \mathit{dx}=0,$$
$$y-x+2x\,i=c,$$
$$\xi=y-x,\quad\eta=2x.\qquad(5)$$
Finaly canonical form of equation $u_{xx}+2u_{xy}+5u_{yy}+3u_x+u=0\quad$ is
$$4u_{\xi\xi}+4u_{\eta\eta}-3u_\xi+6u_\eta+u=0$$
or
$$u_{\xi\xi}+u_{\eta\eta}=\frac{3 u_\xi }{4}-\frac{3 u_\eta}{2}-\frac{u}{4}.$$
Best Answer
Why do you say that $B = 1/2$? If you identify the coefficients from the $PDE$:
$$Au_{xx} + B u_{xy} + C u_{yy} =0, \quad u = u(x,y), \quad (x,y)\in \mathbb{R}^2,$$
then you have $A = B = C = 1$ and the discriminant is:
$$\Delta = B^2-4AC = -3 < 0, $$
so the PDE is elliptic, as you already know. The characterstics are to be found from the equation:
$$A \xi_x^2 + B \xi_x \xi_y + C \xi_y^2 = 0,$$
so it yields (since $A,B,C$ are not zero):
$$\left( \frac{\xi_x}{\xi_y} \right)^2 + B \left( \frac{\xi_x}{\xi_y} \right) + C = 0,$$
and we have the characteristics given by:
$$\frac{\xi_x}{\xi_y} = -\frac{1}{2} \pm \frac{\sqrt{3}}{2} i = - \frac{dy}{dx}.$$
Integrating the equation above, you obtain the two (complex) characteristics as follows:
$$\xi = \phi_+ x + y, \quad \eta = \phi_- x + y,$$
where $\phi_{\pm}$ are the two different roots of the previous equation.
I hope this is useful to you.
Cheers!
Notice that the result that is expected to be obtained holds. Indeed, if you expand $\phi_+$ and $\phi_-$, you will obtain:
$$-x+2y+\sqrt{3} i\, x = 2 \xi = \overline{\xi}, \quad -x+2y - \sqrt{3}i \, x = 2 \eta = \overline{\eta},$$
where the overlined characteristics remain equally constant (even if you multiply them by $-1$).
Edit: notice that you can obtain two real characteristics by performing the following trick:
$$\begin{array}{l} \overline{\xi}+\overline{\eta} = 2x+4y \equiv u, \\ i(\overline{\xi} - \overline{\eta}) = -2 \sqrt{3}x \equiv v.\end{array} $$