Because of the geometry of the cutting rack, the intersection point should happen when the point of contact between the two teeth reach the end point of the rack.
This happens when the generating point intersects the line of action, as shown here. (The black line is the line of action, which rotates but does not slide, while the rack gear rotates and slides.)
This yields the following expression:
$$ addendum/(\Delta Y + y_0) = tan(\phi) $$
where $\Delta Y$ is the distance the rack has traveled, equivalent to $t\cdot r_p$, and $y_0$ is the inital location of the rack point.
So this gives an answer for where to end the trochoid generation. To figure out where to start the involute generation, you first need the radius at which the trochoid generation stops. Then you can use the following to find the involute parameter $\theta$ that corresponds to that given radius:
$$ r = r_b*\sqrt{1 + \theta^2} $$
$$ \theta = \sqrt{(r/r_b)^2 - 1} $$
Unfortunately, below a certain pressure angle, this solution stops giving a correct answer. This is because at small $\phi$, the intersection point between the line of action and generating rack is far enough out on the gear that it can not possibly cut the tooth, like is the case here. ($n_1=10, \phi=13$). Here's an example of this error in action. ($n_1 = 41$, $n_2 = 10$, $\phi=28$)
We can solve for when this solution stops working by finding the $t$ at which the generating rack comes inside the base circle, and comparing the two values:
$$ r_b^2 = (y_0+\Delta Y)^2 + (r_p - addendum)^2 $$
$$ \Delta Y = \sqrt{r_b^2 - (r_p - addendum)^2} - y_0 = t \cdot r_p $$
When this value of $t$ is less than the first, the first is inaccurate. Below the crossover point where they are equal, this value progressively underestimates the intersection point, because we actually want to find when the rack hits the intersection radius, which is only $r_b$ at the crossover point. So I'm still not sure how to find an exact answer for this case. It might be solvable by setting it up as a differential equation with the intersection point on both sides, with the boundary condition that the intersection point is $r_b$ at the crossover. Otherwise, it might have to be done numerically.
I hope this helps.
Best Answer
Going off of @Peter Foreman's answer, you can rewrite the second equation as $$\dfrac{2bx+d\pm\sqrt{4b^2x^2+4bdx+d^2+4a^2x^2+4acx}}{2a}=y$$. Then, rewrite the first equation as $$y=\pm\sqrt{1-x^2}$$ Make the two equations equal to each other and solve.