Take $ax^2+2hxy+by^2=0$. Multiply by $4a$ $(a\neq 0)$ and complete the square to obtain the equivalent equation $$(2ax+hy)^2+(4ab-h^2)y^2=0$$
If $4ab-h^2\gt 0$ then both terms on the left-hand side are non-negative, and must therefore be zero. If $4ab-h^2=0$ you get the single line $2ax+hy=0$ (the two lines coincide to give a degenerate case). And if $4ab-h^2\lt 0$ you get two lines by factoring the left-hand side as the difference of two squares.
You tried to find the second point by solving the system of equations, which could be a bit messy because of the quadratic equation involved.
A cleaner approach is to find the mid-point along the chord that connects the two points $(x_1,y_1)$ and $(x_2,y_2)$. The line of the chord is
$$y = \frac{y_1}{x_1} $$
And the line perpendicular to the chord and going through the center of the circle is
$$y-\frac 12 = -\frac{x_1}{y_1} \left( x - \frac 12 \right)$$
Next, find the intersection point of the two lines,
$$x_m = \frac 12 \frac{y_1+x_1}{y_1^2+x_1^2}x_1, \>\>\>
y_m = \frac 12 \frac{y_1+x_1}{y_1^2+x_1^2}y_1,$$
Since $(x_m,y_m)$ is the midpoint of $(x_1,y_1)$ and $(x_2,y_2)$, simply take the average below,
$$x_m = \frac {x_1+x_2}{2}, \>\>\> y_m = \frac {y_1+y_2}{2}$$
The second point $(x_2,y_2)$ can then be obtained,
$$x_2= 2x_m - x_1 = \left( \frac{y_1+x_1}{y_1^2+x_1^2}-1 \right)x_1 $$
$$y_2= 2y_m - x_1 = \left( \frac{y_1+x_1}{y_1^2+x_1^2}-1 \right)y_1 $$
Recognizing that $(x_1,y_1)$ satisfies
$$x_1^2+y_1^2-x_1-y_1+\frac 14=0$$
The above expressions for $(x_2,y_2)$ could be further simplified.
Best Answer
An arbitrary line is given by the equation $$ a \, x + b\, y = c \quad (*) $$ The circle through the origin with radius $r$ has the equation $$ x^2 + y^2 = r^2 $$ The requirements on the line is to go through the origin $(0,0)$ which happens to be the center of the circle and some point $P=(x_1, y_1)$.
This line can be written in parametric form as $$ x = (1-t)\, 0 + t\, x_1 = t \, x_1 \\ y = (1-t)\, 0 + t\, y_1 = t \, y_1 $$ for $t \in \mathbb{R}$.
Using $(*)$ we insert the origin and get $$ a\, 0 + b\, 0 = c \Rightarrow c = 0 $$ Then we insert $P$ and get $$ a\, x_1 + b\, y_1 = 0 $$ If $P$ is different from the origin, this gives another condition. E.g. if $x_1 \ne 0$ we have $$ a = - \frac{y_1}{x_1} b $$ and $$ -\frac{y_1}{x_1} b x + b y = 0 $$ which means $b = 0$ or $y = (y_1/x_1) x$.