To summarize the discussion in the comments:
In any given trial, the answer must be $k\in \{4,5,6\}$. Any other tosses are irrelevant. Let $p_k$ be the probability that the result is $k$.
$p_4=\frac 13$ since you see $4,5$ or $6$ first with equal probability.
$p_6=\frac 12$ since you see $6$ before $4$ with probability $\frac 12$.
$p_5=\frac 16$ since the only sequence which results in $k=5$ is $5,4,6$. That is to say, the only way to get $k=5$ is to see $5$ before either $4$ or $6$ and then to see a $4$ before a $6$.
Thus the answer is $$E=4\times \frac 13+5\times \frac 16 +6\times \frac 12=\frac {31}6=5.1\overline 6$$
I did the maths, and it turns out there actually is a right answer, if both players are playing to maximise their chance of winning (and both players know that). $A$ should reroll on a 7 or lower, and $B$ should reroll on a 9 or lower.
We can calculate $P(X \ge Y)$ by considering separately the four cases where the 12-sided and 20-sided dice are rolled once or twice each. Assuming $m \ge k$, we have
\begin{align}
P(X\ge Y|k,m)&=
\frac{km\sum_{i=1}^{12} i}{12^2\times 20^2}
+\frac{m\sum_{i=k+1}^{12} i}{12\times 20^2}
+ \frac{k\sum_{i=m+1}^{12} (i-m)}{12^2 \times 20}
+ \frac{\sum_{i=m+1}^{12} (i-m)}{12 \times 20} \\
&=\frac{78km}{12^2\times 20^2}
+ \frac{78m-\frac{k(k+1)}{2}m}{12\times 20^2}
+ \frac{k\frac{(12-m)(13-m)}{2}}{12^2\times 20}
+ \frac{\frac{(12-m)(13-m)}{2}}{12\times 20}.
\end{align}
Taking $m<k$ is clearly non-optimal for $B$, but you can handle it by replacing the $\sum_{i=m+1}^{12}(i-m)$ in the last term with $\sum_{i=k+1}^{12}(i-m)$, i.e. subtracting $\frac{\frac{(k-m)(k-m+1)}{2}}{12\times 20}$ from the final result.
Here is a table of the probability (multiplied by $12^2\times 20^2$ for clarity) that $A$ will win, for each $k$ (columns) and $m$ (rows).
I've highlighted the minimum probability in each column in yellow, and the maximum in each row in green. If $A$ chooses $k$ as 1-7 or as 11, then $B$ should choose $m$ as 9, to maximise their probability of winning. But if $A$ chooses $k$ as 8-10, then $B$ should choose $m=10$. Similarly, $A$ should choose $k=7$ if they expect $B$ to choose $m$ between 5 and 7 or between 9 and 10 (actually, if they know $B$ will choose $m=10$, it's an equally good idea for $A$ to choose $k=6$ or $k=7$, but if there's even a tiny chance $B$ might choose $m=9$ then $A$ should probably choose $k=7$). Thus, if both players are playing rationally, they will settle on $A$ playing $k=7$ and $B$ playing $m=9$. In that case, $A$ wins with probability $\frac{12594}{57600}=\frac{2099}{9600}$, and $B$ wins with probability $\frac{7501}{9600}$.
If one of the players chooses something else, it will decrease their odds of winning even if the other player keeps the same strategy. But the other player might be able to improve their odds even further by adjusting to their opponent's mistake. For example, if $A$ chooses $k=8$ while $B$ chooses $m=9$, then $A$'s probability of winning goes down to $\frac{12552}{57600}$, but $B$ can capitalise on $A$'s aggression by choosing $m=10$, and decrease $A$'s odds of winning further to $\frac{12480}{57600}$.
Best Answer
Hint
$$E(X) = \frac{4}{10} + 2 \cdot \frac{6}{10} \cdot \frac{4}{10} + 3 \cdot \left(\frac{6}{10}\right)^2 \cdot \frac{4}{10} +.... + (n+1) \cdot \left(\frac{6}{10}\right)^n \cdot \frac{4}{10} + ... \infty$$