Let $X$ be the number of hits. Then $X$ can take on values $0$, $1$, $2$, or $3$.
We want to find the probability of $k$ hits, that is, $\Pr(X=k)$, for $k=0,1,2,3$.
To save space, write for example $hmm$ for hit then miss then miss.
The probability that $X=0$ is the probability of $mmm$. This is $\frac{2}{3}\cdot\frac{3}{4}\cdot\frac{3}{4}$, that is, $\frac{9}{24}$.
There are three ways we can have $1$ hit. They are $hmm$, $mhm$, and $mmh$.
The probabilities are $\frac{1}{3}\cdot\frac{1}{2}\cdot\frac{3}{4}$, $\frac{2}{3}\cdot \frac{1}{4}\cdot\frac{1}{2}$ and $\frac{2}{3}\cdot\frac{3}{4}\cdot\frac{1}{4}$. These add up to $\frac{8}{24}$.
The probability that $X=3$ is $\frac{1}{3}\cdot\frac{1}{2}\cdot\frac{1}{2}$, which is $\frac{2}{24}$.
We skipped $X=2$, because the probabilities must add up to $1$. So the probability that $X=2$ is $\frac{5}{24}$. It might be a good idea to find $\Pr(X=2)$ the long way, by listing cases, as a check.
In summary, $\Pr(X=0)=\frac{9}{24}$ and $\Pr(X=1)=\frac{8}{24}$ and $\Pr(X=2)=\frac{5}{24}$ and $\Pr(X=3)=\frac{2}{24}$.
Now finding the mean is simple:
$$E(X)=\frac{9}{24}\cdot 0+\frac{8}{24}\cdot 1+\frac{5}{24}\cdot 2+\frac{2}{24}\cdot 3.$$
For the variance, if is easiest to use the fact that $\text{Var}(X)=E(X^2)-(E(X))^2$. And we have
$$E(X^2)=\frac{9}{24}\cdot 0^2+\frac{8}{24}\cdot 1^2+\frac{5}{24}\cdot 2^2+\frac{2}{24}\cdot 3^2.$$
You and Mosteller are calculating slightly different quantities.
Let's look at what you are calculating for case 1:
You write $P(\text{A survives in case A shots B})$ which means $P\text{(A survives given A kills B}) \equiv P(\text{A survives|A kills B})$
But in the right hand side of your equation you are calculating $P(\text{A survives}\ \cap\ \text{A kills B})$ which is equal to $P(\text{A survives|A kills B})\cdot P(\text{A kills B}) = \frac{3}{13}\cdot 0.3$
So you are calculating the intersection of two events (event 1 and event 2) instead of the conditional (event 1 given event 2). Note that if you use the right name/description for the probability you are calculating then your calculations are agreeing with Mosteller.
The same goes for case 2. You are calculating $P(\text{A survives}\ \cap \ \text{A shoots at B and misses})$
Now that we have named the probabilities more accurately, you can ask yourself: How do they help me in answering the question? Isn't it more useful if I know the conditional probabilities instead? Yes it is. This is what will help you decide on the optimal action for A.
So as Monteller says: If A has to pick a target, he has to go for B. If A misses B, then B kills C on the next round, and on the round after that A has a single chance at B. So A survives with probability $0.3$.
If A kills B then we find that the probability of surviving a duel with C (where C goes first) is $\frac{3}{13}$. So we see it's better to miss. And indeed A can choose to deliberately miss, and this is what he should do.
Finally here's another way to find A's survival probability against a duel with C.
Let's define two probabilities, $P_A$ and $P_C$:
$$
P_A \equiv P(\text{A shoots first and A survives at the end}) \\
P_C \equiv P(\text{C shoots first and C survives at the end})
$$
Now note the relationship between the two, that creates a $2\times 2$ system that can be easily solved:
$$
\left.
\begin{array}{}
P_A = 0.3 + 0.7\cdot (1-P_C)\\
P_C = 0.5 + 0.5\cdot (1-P_A)
\end{array}
\right\}
\iff
\begin{array}{}
P_A = \frac{6}{13}\\
P_C = \frac{10}{13}
\end{array}
$$
What we care about in our scenario is person A surviving given that person C goes first, which is equal to $1-P_C = \frac{3}{13}$
Best Answer
Your first calculation finds the probability that the person hits the target $4$ times in a row. That is very different (and much smaller) than the probability that the person hits at least once.
Let's do the problem in another way, much too long, but it will tell us what is going on.
What is the probability of hitting at least once? The event "at least one hit" could happen in several ways: (i) exactly $1$ hit; (ii) exactly $2$ hits; (iii) exactly $3$ hits; (iv) exactly $4$ hits.
(i) The probability of exactly one hit is $\binom{4}{1}(1/4)(3/4)^3$. This is because the hit could happen in any one of $4$ (that is, $\binom{4}{1}$) places. Write H for hit and M for miss. The probability of the pattern HMMM is $(1/4)(3/4)(3/4)(3/4)$. Similarly, the probability of MHMM is $(3/4)(1/4)(3/4)(3/4)$. You will notice this probability is the same as the probability of HMMM. We get the same probability for MMHM and for MMMH, for our total of $\binom{4}{1}(1/4)(3/4)^3$.
(ii) Similarly, the probability of exactly $2$ hits is $\binom{4}{2}(1/4)^2(3/4)^2$.
(iii) The probability of $3$ hits is $\binom{4}{3}(1/4)^3(3/4)$.
(iv) The probability of $4$ hits is $\binom{4}{4}(1/4)^4$. This is the $(1/4)^4$ that you calculated.
Add up. We get the required answer.
However, that approach is a lot of work. It is much easier to find the probability of no hits, which is the probability of getting MMMM. This is $(3/4)^4$. So the probability that the event "at least one hit" doesn't happen is $(3/4)^4$. So the probability that the event "at least one hit" does happen is $1-(3/4)^4$.