The "magnitude" of $\cos \theta, \sin \theta$ of an obtuse angles can be measure in terms of the acute angle its terminal arm forms with the $x$-axis, positive or negative. So an angle of, say, $\theta = \frac 34 \pi$ has its terminal arm pointing in the direction of Quadrant II, and the absolute values of $\cos\theta$, $\;\sin \theta$ are the same as the values of $\theta' = \pi/4$, the angle formed by the terminal arm of $\theta$ with respect to the negative $x$-axis: $\theta' = \pi - \frac{3\pi}{4}$.
The sign of that magnitude can be determined by the quadrant in which the angle's terminating ray is directed. (Recall that $\cos \theta$ corresponds with the $x$ coordinate, $\sin \theta$ with $y$):
Quadrant I: $\cos \theta, \sin \theta \gt 0$
Quadrant II: $\cos \theta \lt 0$, $\;\sin\theta > 0$
Quadrant III: $\cos \theta \lt 0,\;\;\sin\theta \lt 0$.
Quadrant IV: $\cos \theta \gt 0,\;\;\sin\theta \lt 0$.
Extending to other quadrants
If trigonometric identity formulas hold true so long as we are within the first quadrant, shouldn't we be able to extend our trig functions to all quadrants in this manner?
Mainly, you'd want the following identities.
$$\cos(\alpha+\theta)=\cos(\alpha)\cos(\theta)-\sin(\alpha)\sin(\theta)$$
$$\sin(\alpha+\theta)=\sin(\alpha)\cos(\theta)+\cos(\alpha)\sin(\theta)$$
Since we know $\sin\left(\frac\pi4\right)=\cos\left(\frac\pi4\right)=\frac{\sqrt2}2$, we can derive $\sin\left(\frac\pi2\right)=1$, $\cos\left(\frac\pi2\right)=0$, and so forth.
Similarly, we can derive what $\sin(-\theta)$ is by using $\sin(\theta-\theta),$ $\cos(\theta-\theta)$, and $\cos^2+\sin^2=1$, the Pythagorean identity. Two equations, and you can solve for $\cos(-\theta),\sin(-\theta)$ by substitution. Use Pythagorean identity for simplifying the answer.
It just happens to be that on the unit circle, the hypotenuse is by definition $1$, so...
$$\sin=\frac{\text{opp}}{\text{hyp}}=\text{opp}=y$$
$$\cos=\frac{\text{adj}}{\text{hyp}}=\text{adj}=x$$
And since both this definition and the one above derived by trig identities come out the same for $\theta>90\deg$ or $\pi$, then both are equally correct.
Best Answer
You can use either method to analyse trigonometric functions (with unit circles, or with triangles), but just keep in mind that (really) trigonometric functions have nothing to do with either. trigonometric functions are not functions of triangles, nor are they functions of circles. Trigonometric functions are functions of angles.
I guess my point is that you can use any geometric definition that is convenient, but in the end, we are left with a function and an argument.
I could just as easily have defined $\sin x$ as $$\sin:R\rightarrow R\:|\sin x = \sum_{k=0}^{\infty} \frac{(-1)^kx^{1+2k}}{(1+2k)!}$$ and i wouldn't be wrong.
I could also define $\sin x$ as $Im(e^{ix})$
None of these definitions have anything to do with geometry; they're numeric definitions. But practically, $\sin$ has a geometric interpretation. An angle is the spread between 2 rays, and may have a one to one correspondence with arc length on a unit circle, but still is not the arc length of a unit circle. it's an angle.
Unfortunately, my answer has become way too abstract. What I initially intended to say is that $\sin$ is a function that can be analysed via either triangles or unit circles, or any other way that the student sees fit.