I am not very sure about the methodology. Please let me know if the logic is faulty anywhere.
We have $3$ slots.$$---$$
The middle one has to be a vowel. There are only two ways in which it can be filled: O, A.
Let us put O in the middle.$$-\rm O-$$
Consider the remaining letters: O, A, $\bf P_1$, $\bf P_2$, R, S, L. Since we will be getting repeated words when we use $\bf P_1$ or $\bf P_2$ once in the word, we consider both these letters as a single letter P. So, the letters now are O, A, P, R, S, L. We have to select any two of them (this can be done in $^6C_2$ ways) and arrange them (this can be done in $2!$ ways). So, total such words formed are $(^6C_2)(2!)=30$. Now there will be one word (P O P) where we will need both the P's. So we add that word. Total words=$31$.
Now, we put A in between.$$-\rm A-$$
Again proceeding as above, we have the letters O, P, R, S, L. Total words formed from them are $(^5C_2)(2!)=20$. We add two words (O A O) and (P A P). Thus, total words formed here=$22$
Hence, in all, total words that can be formed are $22+31=53$.
(P.S: Please edit if anything wrong.)
Here is another way - take all three letter words and deduct those containing just vowels or just consonants. This comes to $$8\cdot 7 \cdot 6-5\cdot 4 \cdot 3 - 3\cdot 2 \cdot 1=336-60-6=270$$
If letters are allowed to be repeated the number is $$8^3-5^3-3^3=512-125-27=360$$
Another way of counting is to count the number of possibilities for each pattern of vowels and consonants.
$VVC: 5\times 4 \times 3=60$
$VCV: 5\times 3 \times 4=60$
$VCC: 5\times 3 \times 2=30$
$CCV: 3\times 2\times 5=30$
$CVC: 3\times 5\times 2=30$
$CVV: 3\times 2\times 5=60$
This gives $270$
I've done this longhand for clarity
Best Answer
The textbook is saying "choose the $5$ out of $8$ positions for consonants (so $3$ for vowels), then choose the $3$ vowels possibly with repetition and the $5$ consonants possibly with repetition"
$^{5}C_3$ would be choosing three distinct vowels in any order rather than the $5^3$ since order matters and repetitions are allowed, and similarly $^{21}C_5$ should be $21^5$ for the same reason.
I am not clear what your $8^8$ was intended to be. If the question had prohibited repeated letters, the answer would have been $8!\,^{5}C_3\, ^{21}C_5$