Consider the general form of equation of straight line.
$$ax + by + c = 0$$
$$\Rightarrow ax + by = -c$$
$$\Rightarrow \frac{ax}{-c} + \frac{by}{-c} = 1$$
$$\Rightarrow \frac{x}{\frac {-c}{a}} + \frac{y}{\frac {-c}{b}} =1 $$
this is equation of line in intercept form with x-intercept $\frac {-c}{a}$ and y-intercept $ \frac {-c}{b}$
In above fig. $\overline {OD}\bot\overline {AB}$
Now consider $\Delta$OAD
$$Cos\alpha =\frac {\overline{OD}}{\overline{OA}} = \frac{P}{\frac {-c}{a}} =\frac {-aP}{c} $$
In $\Delta$OBD
$$Cos(90-\alpha)=\frac {\overline{OD}}{\overline{OB}} = \frac{P}{\frac {-c}{b}} =\frac {-bP}{c} $$
Since Cos($90-\alpha$)= Sin $\alpha$ Therefore,
$$Sin\alpha = \frac {-bP}{c}$$
Now
$$tan\alpha =\frac {sin \alpha}{cos\alpha} = (\frac {-bP}{c})(\frac{-c}{aP})$$
$$tan \alpha = \frac {b}{c}=> \frac {perpendicular}{base}$$
$$\Rightarrow hyp = \pm\sqrt{a^2+b^2} (by Pythagorean theorem)………(a)$$
$$\Rightarrow cos\alpha = \frac {a}{\pm\sqrt {a^2+b^2}},sin\alpha = \frac {b}{\pm\sqrt {a^2+b^2}}……………(1)$$
Now for transforming general equation of straight line to normal form:
Consider the general form of equation of straight line
$$ax+by+c=0$$
$$\Rightarrow ax+by=-c$$
Now in order to to have Normal form of equation of straight line it is said that divide both sides by $\pm\sqrt{a^2+b^2}$, then we get
$$\frac {a}{\pm\sqrt{a^2+b^2}}x+\frac {b}{\pm\sqrt{a^2+b^2}}y=\frac {-c}{\pm\sqrt{a^2+b^2}}$$
Using equations (1) in above we get
$$xcos\alpha+ysin\alpha=\frac {-c}{\pm\sqrt{a^2+b^2}}$$
Now comparing with normal equation of straight line i.e.$xcos\alpha+ysin\alpha=P$
$$\implies P=\frac {-c}{\pm\sqrt{a^2+b^2}}$$
Now the question is:
1: In equation (a) $hyp = \pm\sqrt{a^2+b^2}$ but according to figure in $\Delta OAD hyp=\overline{OA}= \frac{-c}{a}$
How $\pm\sqrt{a^2+b^2}=\frac{-c}{a}$?
Best Answer
The general equation $ax+by+c=0$ of a straight line in $\mathbb R^2$ can be rewrtten as $\mathbf n\cdot\mathbf x=-c$, with $\mathbf n\ne0$. This form of the equation tells us that a line can be characterized as the set of points that have the same dot product with some fixed vector $\mathbf n$. If we normalize $\mathbf n$ by dividing both sides of the equation by its length, we have $${\mathbf n\over\|\mathbf n\|}\cdot\mathbf x=-{c\over\|\mathbf n\|}.$$ This is the so-called normal form of the equation for the line, characterized by $\mathbf n$ being a unit vector. The left-hand side of this equation is the (signed) length of the orthogonal projection of $\mathbf x$ onto $\mathbf n$, so an equivalent way to charactere the line is as the set of vectors $\mathbf x$ that have the same projection $\mathbf x_\parallel$ onto $\mathbf n$. The remainder $\mathbf x-\mathbf x_\parallel$ is perpendicular to $\mathbf n$, from which we can see that $\mathbf n$ is perpendicular to the line, and so the (perpendicular) distance of the line from the origin is ${|c|\over\|\mathbf n\|}$. Substituting the coefficients from the original equation yields the normal equation $${a\over\sqrt{a^2+b^2}}x+{b\over\sqrt{a^2+b^2}}y={-c\over\sqrt{a^2+b^2}}.$$ Since we have a unit normal, we can replace the coefficients on the left-hand side: $$x\cos\alpha+y\sin\alpha={-c\over\sqrt{a^2+b^2}},$$ where $\alpha$ is the angle made by the normal vector with the $x$-axis.
Now, as far as your equation (a) is concerned, remember that in computing $\tan\alpha$, you canceled a constant of proportionality that appeared in both the numerator and denominator, so the most that you can conclude from it is that the length of the hypotenuse of $\triangle{OAD}$ is proportional to $\sqrt{a^2+b^2}$. Indeed, in your diagram you have two other triangles that are similar to $\triangle{OAD}$: $\triangle{BOD}$ and $\triangle{BOA}$, so we have $$\tan\alpha = {\overline{AD}\over\overline{OD}} = {\overline{OD}\over\overline{BD}}={\overline{OA}\over\overline{OB}}.$$ The legs of the largest of these are the hypotenuses of the smaller ones, so there are at least two different lengths of hypotenuse in play. Moreover, it might be that none of the lengths is equal to $\sqrt{a^2+b^2}$!