Find a conformal mapping that maps the region $|z| > 1$ onto the region $\operatorname{Re} (w) < 0$.
I know I need to pick three points on $|z|=1$ and three points on the imaginary axis.
I need to them plug them into the formula $\frac{(z-z_1)(z_2-z_3)}{(z-z_3)(z_2-z_1)}=\frac{(w-w_1)(w_2-w_3)}{(w-w_3)(w_2-w_1)}$.
But there is more to the story. I need to pick the points such that orientation is preserved. What does that mean? Why and how do I do this?
I also need to make sure $\infty$ is included as one of the points. Which point? And again, why and how?
Best Answer
For reference, I wrote a detailed answer here about how to find conformal mappings between various domains $\subseteq \mathbb{C} \cup \{\infty\}$. I don't see this question as a duplicate though, since this time you're focusing on how to find the conformal mapping using "three points to three points" thinking and several specific questions you had along the way. I'll respond in the form of responding to each question individually.
"I need to pick the points such that orientation is preserved"
If you pick any 3 points on circle 1 and any 3 points on circle 2 (note lines count as circles in complex analysis!) then the formula you gave will produce a conformal mapping from $\mathbb{C} \cup \{\infty\}$ to itself which takes circle 1 to circle 2. In your case, if you randomly select points $(1,i,-1)$ on the unit circle and $(0, i, 2i)$ on the imaginary axis, then your formula gives some mapping $T : $ unit circle $\to$ imaginary axis. That also means it must take $|z|<1$ to one of the two half-planes and $|z|>1$ to the other one, but which is which? You could plug in a specific value to find out, like just compute $T(0)$ and see whether $\operatorname{Re}(T(0))>0$ or $\operatorname{Re}(T(0))<0$, but the "orientation preserving" idea is a useful shortcut.
Think about walking along the circle and passing by the 3 points in order. If I walk along the unit circle and I pass $(1,i,-1)$ in that order, then I must be walking counterclockwise, so the region you care about $(|z|>1)$ is on my RIGHT. If I walk along the imaginary axis and I pass $(0,i,2i)$ in that order, then the region $\operatorname{Re}(w) < 0$ is on my LEFT. Those don't match, which means the lists of points I picked are oriented wrong: I'll end up mapping $|z|>1$ to $\operatorname{Re}(w) > 0$ instead.
To fix the orientation, I could instead use $(0, -i, -2i)$ for my points on the imaginary axis. That way the source and target regions will both be on my right as I walk around the boundary circles. I could also fix it in other ways: I could instead change the target space points to $(0, 2i, i)$, or I could leave the target space points alone and change the source space points to $(1,-i,-1)$.
I also need to make sure $\infty$ is included as one of the points.
You definitely can obtain valid conformal maps without using $\infty$ as one of the points. But anyway let's just pretend you want to use $\infty$ as one of your points for some reason. $\infty$ is not on the boundary of the source space so we'd have to choose it as one of the target space points. So let's use $(1,i,-1)$ in the source space and $(0,-i,\infty)$ in the target space.
This is all fine except that it's confusing trying to plug $w_3 = \infty$ into your formula. The fix is to take the limit, like
$$\lim_{w_3 \to \infty} \frac{(w-w_1)(w_2-w_3)}{(w-w_3)(w_2-w_1)} = \frac{w-w_1}{w_2-w_1}$$
In these situations, "taking the limit" always just amounts to deleting the two terms that involve the point that equals $\infty$.
It doesn't matter which point $w_j$ was $\infty$. For example, for my 3 points in $w$-space I could have instead used $(-i, \infty, 0)$ or $(\infty, 0, -i)$. Notice how I'm still making sure the orientation stays correct as I move the points around though.
Final note on this problem: convenience
If I needed to solve this problem with no restrictions on how I did it, I would choose $(1,-i,-1)$ in the source space and $(0,i,\infty)$ in the target space. The resulting formulas usually end up looking nicer if you choose $0$ or $\infty$ whenever possible, and otherwise "simple" points like $\pm 1, \pm i$. In this case, my chosen points give the transformation $\frac{(z-1)(-i+1)}{(z+1)(-i-1)} = \frac{w}{i}$ which just simplifies to $w = \frac{1-z}{1+z}$. Sometimes the source/target domains will force you to pick "messier" points, but might as well pick "easy to work with" points if you can!
When you can/can't use this strategy
The "three points to three points" thinking can help you find any Mobius transformation, which is a subset of conformal maps. If you want all circles to map to circles then Mobius transformations will work great. Otherwise you'll need to use some other function. My other answer (here's the link again) has a couple of examples, but the easiest is that if you want a conformal mapping from the first quadrant to the upper half-plane then you should pick $f(z) = z^2$. You can't use a Mobius transformation because the boundary start out "bent" and ends up as a straight line, which a Mobius transformation could never do.