Here's the full combinatorial argument. Raney's lemma was about half of it, I'd say. The argument shows that $S(n,2) = \binom{3n-1}{n} \frac{3}{3n-1}$ (which is equivalent to the two formulations I give in the question). At the end I'll discuss the generalization to the $r$ case.
Intro.
Consider the sequences with $2n$ occurrences of $-1$ (i.e., $2n$ heads) and $n$ occurrences of $+2$ (i.e., $n$ tails). We want to show that the number of these sequences with all partial sums nonzero is $\binom{3n-1}{n} \frac{3}{3n-1}$. The complete sum and empty sum are clearly $0$, so "partial sum" excludes those two cases. The sequences we want to count can be split into three groups: (1) all partial sums positive, (2) all partial sums negative, (3) some partial sums positive and some negative.
Group 1: The number of these sequences with all partial sums positive is $\binom{3n-1}{n} \frac{1}{3n-1}$.
This is the part that uses Raney's lemma. If all partial sums are positive, the last element in the sequence must be $-1$. Thus we want to count the number of sequences with $2n-1$ occurrences of $-1$ and $n$ occurrences of $+2$ that add to $+1$ and have all partial sums positive. Ignoring the partial sums restriction, there are $\binom{3n-1}{n}$ such sequences. If we partition these $\binom{3n-1}{n}$ sequences into equivalence classes based on cyclic shifts, Raney's lemma says that exactly one sequence in each equivalence class has all partial sums positive. Because there are $3n-1$ elements in each sequence there $3n-1$ sequences with the same set of cyclic shifts, and so there are $3n-1$ sequences in each equivalence class. Thus the number of sequences in Group 1 is $\binom{3n-1}{n} \frac{1}{3n-1}$.
Group 2: The number of these sequences with all partial sums negative is also $\binom{3n-1}{n} \frac{1}{3n-1}$.
To see this, just reverse the sequences counted in Part 1.
Group 3: The number of these sequences with some positive partial sums and some negative partial sums is, yet again, $\binom{3n-1}{n} \frac{1}{3n-1}$.
This one is a little trickier. First, because of the $-1$'s, it is not possible to switch from positive partial sums to negative partial sums. Thus any sequence counted here must have exactly one switch: from negative partial sums to positive partial sums. The switch must occur at some point where the partial sum is $-1$ and the next element is $+2$. Thus we have some sequence $(a_1, \ldots, a_m, +2, a_{m+2}, \ldots, a_n)$ where the sums $a_1, a_1 + a_2, \ldots, a_1 + \ldots + a_m$ are all negative. Consider the sequence $(+2, a_m, \ldots, a_2, a_1, a_{m+2}, \ldots, a_n)$. Since $+2 + a_m + \ldots + a_1 = 1$, and $a_k + a_{k-1} + \ldots + a_1 < 0$ for all $k$, $1 \leq k \leq m$, it must be the case that $+2 + a_m + \cdots + a_{k+1} > 1$ for all $k$, $1 \leq k \leq m-1$. So the sequence $(+2, a_m, \ldots, a_2, a_1, a_{m+2}, \ldots, a_n)$ is in Group 1. To see that this mapping is a bijection, note that any sequence in Group 1 must start with $+2$ and have a first time that a partial sum is equal to $+1$. Thus this transformation is reversible.
Summing up.
Putting Groups 1, 2, and 3 together we see that the total number of sequences we want to count is $\binom{3n-1}{n} \frac{3}{3n-1}$.
Generalization to the $r$ case.
The arguments for Group 1 and Group 2 adapt in a straightforward manner for general $r$; we get $\binom{(r+1)n-1}{n} \frac{1}{(r+1)n-1}$ for both. Sequences in Group 3 can still only switch once - from negative partial sums to positive partial sums. But Group 3 can be broken up into subgroups depending on whether the first partial sum to become positive is $+1, +2, \ldots, r-1$. Using transformations like the one described above for Group 3 in the $r = 2$ case, we can show that there is a bijection between each of these $r-1$ subgroups and Group 1. Thus there are $\binom{(r+1)n-1}{n} \frac{r-1}{(r+1)n-1}$ sequences in Group 3. In total, then, $S(n,r) = \binom{(r+1)n-1}{n} \frac{r+1}{(r+1)n-1}$.
Final comment.
All of this is may be easier to visualize in terms of paths from $(0,0)$ to $((r+1)n,(r+1)n)$ that use right steps of size $1$ and up steps of size $r$ and that do not step on the diagonal except at $(0,0)$ and $((r+1)n,(r+1)n)$. (They can cross the diagonal, though.) I found it easier to discuss the partial sums interpretation, though, given the difficulty of creating such graphics on this forum.
It may be easier to split another way. The probability of heads is $0.5$. So the probability of $3$ heads or $4$ or $5$ is
$\binom{5}{3}(0.5)^5+\binom{5}{4}(0.5)^5+\binom{5}{5}(0.5)^5$.
We need in addition the probability of $2$ heads, and $0$ or $1$ tails, and therefore $3$ or $2$ edges, which is
$\binom{5}{2,0,3}(0.5)^2(0.4)^0(0.1)^3+\binom{5}{2,1,2}(0.5)^2(0.4)^1(0.1)^2$.
We need also the probability of $1$ head, $0$ tails, and $4$ edges, which is $\binom{5}{1,0,4}(0.5)^1(0.4)^0(0.1)^4$.
Remark: In general, suppose that we toss the coin $n$ times. We will find the number of toss patterns that result in $a$ heads, $b$ tails, and $c$ edges, where $a+b+c=n$.
The locations of the heads can be chosen in $a$ ways. For each choice of locations of heads, the locations of the tails can be chosen in $\binom{n-a}{b}$ ways. So the total number of patterns is $\binom{n}{a}\binom{n-a}{b}$. This is equal to $\frac{n!}{a!(n-a)!}\cdot \frac{(n-a)!}{b!(n-a-b)!}$.
Cancel the $(n-a)!$ from numerator and denominator, and note that $n-a-b=c$. Thus the number of patterns is $\frac{n!}{a!b!c!}$. This is precisely $\binom{n}{a,b,c}$.
Best Answer
As pointed out in comments, it seems they were counting number of beginnings of groups of $H$. Out of all $H$'s, we are given 3 of $H$s are preceeded by another $H$, so these 3 cannot be a beginning of a group, hence only the remaining $8-3=5$ of $H$s correspond to a group beginning.
However this seems unnecessarily complicated, as we can count number of these groups "directly", without complementing (subtracting) ... Since the whole sequence starts with $T$ (already recognized in the solution), we know all groups of $H$ must start with $TH$, and by problem statement there are exactly $5$ of these...
Or we can also count the number of ends of these groups directly - there are $4$ cases of $HT$, plus there is one case of $H$ at the end of the whole sequence (these are the only two ways the group of $H$ can end), so $4+1=5$ groups in total.
The tails case is similar...