Break into two cases:
- There is one suit which has three cards selected, and two suits such that each have only one card selected
- There are two suits which have two cards selected each, and one suit which has only one card selected
For ease of calculation, we treat the cards in our hand as order not mattering, allowing us to use binomial coefficients and not have to break this case down further into aaabc, aaacb, aabac, aabca, etc...
In trying to calculate the number of outcomes in the first case, we continue:
- Pick the single suit used which has three cards of its type appearing in our hand. $4$ choices
- Simultaneously pick which two suits used have one card of its respective type appearing in our hand (we do this simultaneously because we could not otherwise distinguish between which was the "first" selected and the "second" selected). $\binom{3}{2}$ choices
- Now, pick which actual ranks appear for the suit that has three cards selected. $\binom{13}{3}$ choices
- For the remaining two selected suits, pick which rank was used for each individually. (Specifically, choose the rank for the selected suit that appears first alphabetically, and then choose the rank for the remaining selected suit after, that way there is no ambiguity as to which rank corresponds to which suit) $13\times 13$ choices.
Multiplying these out give the number of hands where order doesn't matter which have one suit that has three cards and two suits which each have one card as being $$4\times \binom{3}{2}\times \binom{13}{3}\times 13\times 13$$
Now, if you want order of cards to matter, just multiply the result by $5!$.
The second case is calculated similarly, and I leave it to you to complete yourself.
Edit: More details
When applying multiplication principle, outcomes selected in one step are treated differently than outcomes selected in a later step. As such, in order to avoid overcounting, you need to ask yourself the question of if you were to answer the steps in one order you could have given answers in a different order and still arrived at the same outcome. Similarly, to avoid undercounting you need to ask yourself if every outcome is actually counted.
To illustrate your mistake here, let us consider a smaller problem. We have four people, two men and two women. Let the women be $A,B$ and the men be $X,Y$. We ask the question of "how many ways are there to select four people such that both genders are represented?"
Here... it should be painfully obvious that every way of selecting the people will have both genders represented. If we were to treat the selections as order matters, then this is very simply $4!$ number of arrangements, and if we treat selections as order doesn't matter there is very simply only $1$ outcome.
According to your logic, however... "Pick a representative for a first gender, pick a representative for the remaining gender, then pick the remaining" when counting the number of arrangements where order matters you would have had an answer along the lines of $4\times 2\times 2\times 1$ which is not the full $4!$. The reason why is that you very specifically counted only the cases where the first two people in the line are of a different gender and neglected to consider outcomes like $XYAB$ and $ABXY$, etc...
On the other hand, if we treated order as not being important, according to your logic, "Pick the used genders, pick a guaranteed representative for the males, pick a guaranteed representative for the females, pick the remaining people" you would have arrived at an answer similar to $\binom{2}{2}\times 2\times 2\times \binom{2}{2}=4$ which is certainly not $1$ which we knew ahead of time to be the correct answer. The error here is that by selecting a person to be the guaranteed representative for a gender, we have placed unnecessary importance on them. This would have been the correct answer to the question of picking four people from our four people, such that there are two genders present, where each gender has a designated "leader." Compare the proposed steps with "Pick the used genders, pick a leader for the males, pick a leader for the females, pick the remaining people."
To correctly count... anything which can be confused for one another if they were picked in sequence, instead pick simultaneously. In the first example, looking at a final outcome there is no confusion as to which suit was the suit that has three cards in it, it is obvious which that is. The two suits which each have one card however, we wouldn't have been able to tell which suit was chosen first and which was chosen second had the cards been mixed up. In the same way, the three cards chosen for the suit with three cards in it, if we were to pick them in sequence and then mixed the cards, we won't have the ability to know in what order they were picked any more after the mixing.
The steps I would use for correctly counting the second case, which again is the number of ways in which we can have two suits with two cards each and one suit with one card such that order doesn't matter:
- Pick the two suits to be used for the suits which have two cards each simultaneously
- Of the suits selected in the previous step, we focus on the suit which comes first alphabetically. Pick the two ranks of cards that will appear for that suit simultaneously. (NOTE: here, we take advantage of the fact that after we have picked the suits, there is an unambiguous way to order them and we can affect them each one at a time. We do not multiply by anything in order to decide "which" suit was first alphabetically.)
- For the remaining selected suit, pick the two ranks of cards that will appear for that suit
- Pick the remaining card from a suit different than either of the previous two.
This gives $\binom{4}{2}\binom{13}{2}\binom{13}{2}\times 26$ outcomes for the second case.
The final count where order doesn't matter is then:
$$4\times \binom{3}{2}\times\binom{13}{3}\times 13\times 13 + \binom{4}{2}\binom{13}{2}\binom{13}{2}\times 26$$
Where order does matter, multiply the above result by $5!$
Best Answer
Stars-and-bars counts ways to place identical items into distinct containers. Thus it is inapplicable to counting the ways to select cards, which are distinguishable by rank. It is the wrong way to go.
The alternative method does consider the selection of rank. This is the right thing to do.
$7$ can be partitioned into sums of four terms as $(4+1+1+1)$, $(3+2+1+1)$, and $(2+2+2+1)$. We count ways to select suits for each amount, and ways to select those amounts of ranks in the selected suits.
$$\binom 4 {1,3}\binom {13}4^1\binom{13}1^3+\binom{4}{1,1,2}\binom{13}3^1\binom{13}2^1\binom{13}1^2+\binom 4{3,1}\binom{13}2^3\binom{13}1^1$$