The probability till Mike and Dean get the same results at most 5 rounds:
The final results can be... $${HH}\ or\ {TT}$$
The probability that Mike and Dean get the same results in a round
$$={0.6}\times{0.1}+{(1-0.6)}\times{(1-0.1)}=0.42 $$
The probability that Mike and Dean do not get the same results in a round $$={1-0.42}=0.58$$
Let X be the number of rounds until Mike and Dean get the same results. $X\sim Geo(0.42)$
$$P(X\leq5)=P(X=1)+P(X=2)+P(X=3)+P(X=4)+P(X=5)$$
$$P(X\leq5)=0.42+0.58\times0.42+(0.58)^2\times0.42+(0.58)^3\times0.42+(0.58)^4\times0.42\approx0.9344\ (corr.to\ 4\ d.p.)$$
The binomial distribution PMF isn't really describing the same thing - it looks for the possibility of getting a specific exact number of tails. If that's really what you're looking for (the probability that 3 are tails and the rest are heads), the replacement for nCk is n-k+1, or 3, because there are 3 places where there could be 3 consecutive tails (the beginning, the middle 3, or the last 3). Following this should get you 3/32.
On the other hand, if you're looking for the probability of there just being 3 consecutive tails (so that 5 tails in a row would also count), you're going to have a lot of difficulties with overcounting. The easiest method in this case is really just to count the possibilities (there are 8 - ttttt,tttth,tttht,ttthh,thttt,htttt,httth,hhttt). Divide this by the total number of possibilities (there are 32 ways to flip a coin 5 times) to get the overall probability of 1/4.
If you really want to do the second part in a slightly fancier way, you have to avoid counting sequences like ttttt multiple times. The way to do this is to look for an h followed by 3 t's - if you find httt, there can be no previous sequence of 3 t's, and you avoid overcounting. Therefore, you are looking for ttt??,httt?, and ?httt, which have probabilities of 1/8,1/16,and 1/16 respectively, summing to 1/4.
Note that this method won't work for significantly longer sequences, as it's possible to have 3 consecutive t's, then httt. More advanced methods are required.
TL;DR depending on what you mean, the answer is probably 1/4, but the method you described isn't particularly useful
Best Answer
The probability of getting $THH$ is much higher because the sequence of flips is continuous; that is, the trials are not independent.
If I have a chain of coin flips in which neither sequence has appeared, such as below:
$...HTHTT$
then it would be impossible for the sequence $HHH$ to appear before $THH$ appears.