Let $a,b,c,d,e$ be the number of occurences of $A,B,C,D,E$, respectively.
Consider that among $B,C,D,E$ there cannot be $2$ (or more) letters with $3$ occurences.
Knowing that, let's split cases!
Case $1$: $b=3\implies (c=1)\land (d,e\leq2)$
Let's split cases again of $d$ and $e$:
- Sub-case 1: $d=2,e=2\implies a=4$ (invalid)
- Sub-case 2: $d=2,e=1\implies a=5$
- Sub-case 3: $d=1,e=2\implies a=5$
In sub-case 2, $(a,b,c,d,e)=(5,3,1,2,1)$, so there are $\frac{12!}{5!3!1!2!1!}=332640$ possibilities.
In sub-case 3, $(a,b,c,d,e)=(5,3,1,1,2)$, so there are $\frac{12!}{5!3!1!1!2!}=332640$ possibilities.
Therefore, in this case, there are $2\times332640=665280$ possibilities.
Case $2$: $c=3\implies (b=1)\land (d,e\leq2)$
Case $3$: $d=3\implies (e=1)\land (b,c\leq2)$
Case $4$: $e=3\implies (d=1)\land (b,c\leq2)$
Consider that in each of the three cases, the number of possibilities is the same as in the first case (simply change the variables to see that).
So, across the four cases, there are $$4\times665280=2661120$$ possibilities.
Case $5$: $b,c,d,e\leq 2$
Consider that in each of $\{b,c\}$ and $\{d,e\}$, the members cannot both be $1$. So the possibilities are $(2,2),(2,1),(1,2)$. Consider that the number of $1$s in that set is either $0$ ($1$ possibility) or $1$ ($2$ possibilities).
Therefore, the number of $1$s in $\{b,c,d,e\}$ is one of
- $0$ ($1\times1=1$ possibility),
- $1$ ($2\times1+1\times2=4$ possibilities), or
- $2$ ($2\times2=4$ possibilities).
Let that number of $1$s be $n$. Let's split cases again based on $n$:
In this sub-case, there are $4\times\frac{12!}{5!2!2!2!1!}=1995840$ possibilities.
- Sub-case 3: $n=2$ ($4$ possibilities) $\implies a=6$
In this sub-case, there are $4\times\frac{12!}{6!2!2!1!1!}=665280$ possibilities.
Therefore, in this case, there are $$1995840+665280=2661120$$ possibilities in this case.
Therefore, the number of possibilities across the five cases is
$$2661120+2661120=\fbox{5322240}\text.$$
Best Answer
There are two ways to look at it.
One way to look at it is that you have 8 slots to fill, and you need to choose 3 of those slots to fill with letters. That's the same problem as choosing a team of 3 people from a pool of 8 people, and there's a combinatorial function that answers that question: $\binom83 = \frac{8!}{3!(8-3)!}$
The other way to look at it is that any time you have 8 characters (say $A_1A_2A_3N_1N_2N_3N_4N_5$) to arrange in a sequence, there are $8!$ ways to do so. But since some of the characters in this case are indistinguishable, you have to divide out certain symmetries. The symmetries you have to divide out are all rearrangements of the 3 As ($3!$) and all rearrangements of the 5 Ns ($5!$). So the answer will be $\dfrac{8!}{3!5!}$