In even more general cases than converting a line to and from polar, you can use the substitutions
$$r=\sqrt{x^2+y^2}, \theta=\tan^{-1}\left(\frac{y}{x}\right), x=r\cos\theta, y=r\sin\theta$$
Note that because $\arctan$ only takes values in $\left(-\frac{\pi}{2},\frac{\pi}{2}\right)$, the identity given for $\theta$ doesn't do all the work: you need to decide between the fourth and the second, and between the first and the third, quadrants based on the signs of $x$ and $y$.
I haven't checked the formula, but you have ${850000 \over 6371000}$ instead of
either ${8500000 \over 6371000}$ or ${8500 \over 6371}$.
The point for the -180 bearing in the data above is about 750 km south of the center location.
(I believe the correct coordinates are about $(-96.1953, -75.7800)$.)
It is not clear to me how you are computing your new coordinates. The Haversine formula is used to compute the distance given two sets of coordinates and does
not have a trivial inverse. Perhaps you can provide a link?
Here is one way to do it, I am sure there must be better ways:
I am performing the computations in the x-y-z plane rather than using
latitude/longitude. The mapping between the two is straightforward.
Given a point $p \in \mathbb{R}^3$ on the surface of the earth excluding the poles, a bearing $\beta$ (measured clockwise from north) and a distance $L$,
compute a new point $p'$ a distance $L$ away from $p$ at bearing $\beta$ (from $p$).
Note that the poles are excluded so that the bearing makes sense, at either pole a bearing to/from North makes no sense.
I am assuming a perfectly spherical Earth. To reduce notational clutter,
define
$u: \mathbb{R}^3 \setminus \{0\} \to \mathbb{R}^3$ by $u(x) = {1 \over \|x\|} x$ (that is, normalise $x$).
Let $R= \|p\|$, be the radius (of the idealised Earth), and let $N=(0,0,R)$ be the porth pole.
Note that a distance $L$ corresponds to an angle (at the Earth's center) of
$\alpha = {L \over R}$ (radians).
Compute a 'local north direction' $\nu = u(N-{1 \over R^2}\langle p, N \rangle p)$. This is a direction contained in the intersection of the tangent plane at $p$ with the subspace containing $p$ and $N$.
Compute a 'local east direction' $e = u(\nu \times p)$. This is a
direction perpendicular to both $\nu$ and $e$ with the appropriate
orientation.
(Note that $\nu, p, e$ are mutually orthogonal.)
Compute a 'local direction' $d = (\sin \beta) e + (\cos \beta) \nu$.
This is a unit vector in the direction we want to go in on the tangent
plane at $p$.
Then we have the new point $p' = (\cos \alpha) p + (\sin \alpha) R d$.
Best Answer
First, use $r$ and $\theta$ to find an equation for the line you are describing in slope-intercept form.
In particular, convert your given point from polar coordinates to its Cartesian coordinates $(x,y)$, using $x = r\cos\theta$ and $y = r\sin\theta$.
Note also that the line from the origin through this point has slope $y/x$.
Now, to find the line this point describes (as in your question) note that you have a point on the line, namely, $(x,y)$, and know the slope will be $-x/y$. A single point and the slope are enough to find an equation for the line. (I'll leave that to you.)
Now the question becomes: given a line in slope-intercept form and a point's Cartesian coordinates, how do we find the shortest distance between them?
This last question has a standard formula whose derivation I assume you are familiar with.