You and Mosteller are calculating slightly different quantities.
Let's look at what you are calculating for case 1:
You write $P(\text{A survives in case A shots B})$ which means $P\text{(A survives given A kills B}) \equiv P(\text{A survives|A kills B})$
But in the right hand side of your equation you are calculating $P(\text{A survives}\ \cap\ \text{A kills B})$ which is equal to $P(\text{A survives|A kills B})\cdot P(\text{A kills B}) = \frac{3}{13}\cdot 0.3$
So you are calculating the intersection of two events (event 1 and event 2) instead of the conditional (event 1 given event 2). Note that if you use the right name/description for the probability you are calculating then your calculations are agreeing with Mosteller.
The same goes for case 2. You are calculating $P(\text{A survives}\ \cap \ \text{A shoots at B and misses})$
Now that we have named the probabilities more accurately, you can ask yourself: How do they help me in answering the question? Isn't it more useful if I know the conditional probabilities instead? Yes it is. This is what will help you decide on the optimal action for A.
So as Monteller says: If A has to pick a target, he has to go for B. If A misses B, then B kills C on the next round, and on the round after that A has a single chance at B. So A survives with probability $0.3$.
If A kills B then we find that the probability of surviving a duel with C (where C goes first) is $\frac{3}{13}$. So we see it's better to miss. And indeed A can choose to deliberately miss, and this is what he should do.
Finally here's another way to find A's survival probability against a duel with C.
Let's define two probabilities, $P_A$ and $P_C$:
$$
P_A \equiv P(\text{A shoots first and A survives at the end}) \\
P_C \equiv P(\text{C shoots first and C survives at the end})
$$
Now note the relationship between the two, that creates a $2\times 2$ system that can be easily solved:
$$
\left.
\begin{array}{}
P_A = 0.3 + 0.7\cdot (1-P_C)\\
P_C = 0.5 + 0.5\cdot (1-P_A)
\end{array}
\right\}
\iff
\begin{array}{}
P_A = \frac{6}{13}\\
P_C = \frac{10}{13}
\end{array}
$$
What we care about in our scenario is person A surviving given that person C goes first, which is equal to $1-P_C = \frac{3}{13}$
Best Answer
To maximise their chances the duellists prefer to be left with a weaker opponent. So Bob would not shoot at Alice in preference to Carol, and Carol will not shoot at Alice in preference to Bob. Therefore Alice will not be shot at until Bob or Carol is dead and she will either be left standing with Bob or Carol, with or without the shot.
Probability of Alice, with shot, surviving against Bob is given by:
p = Pr(A hits B) + Pr(A misses B) * Pr (B misses A) * p
p = 1/3 + 2/3 * 1/3 * p
p = 3/7
Probability of Alice, without shot, surviving against Bob is given by:
p = Pr(B misses A) * (Pr(A hits B) + Pr (A misses B) * p)
p = 1/3 * (1/3 + 2/3 * p)
p = 1/7
Probability of Alice, with shot, surviving against Carol is given by:
p = Pr(A hits C) + Pr(A misses C) * Pr (C misses A) * p
p = 1/3 + 2/3 * 0 * p
p = 1/3
Probability of Alice, without shot, surviving against Carol is given by:
p = Pr(C misses A) * (Pr(A hits C) + Pr (A misses C) * p)
p = 0 * (1/3 + 2/3 * p)
p = 0
So, her probability of surviving from each position is:
Bob, with shot: 3/7
Carol, with shot: 1/3
Bob, without shot: 1/7
Carol, without shot: 0
So Alice is best off not killing anyone since the advantage she gains by having the first shot exceeds any possible benefit of facing Bob rather than Carol. She should shoot into the air.
Given that Alice is neither going to shoot at them, or be shot at by them until one is dead, Bob and Carol are essentially in a two person duel, the winner to face Alice. They cannot improve their chances by forgoing a shot, so they shoot. Bob wins that 2/3 of the time, Carol 1/3.
Alice wins 2/3 * 3/7 + 1/3 * 1/3 = 25/63.
Bob wins 2/3 * 4/7 = 24/63.
Carol wins 1/3 * 2/3 = 14/63.