There are indeed $\binom{52}{13}$ different 13-card hands and this will indeed be the size of our sample space and thus our denominator when we finish our calculations.
For the numerator, we need to pause for a moment and understand what the problem is actually asking, since this appears to be where you got stuck.
We are asked to find the probability that in our hand of thirteen cards, there is at least one suit for which we have all three face cards. For example $(A\spadesuit,2\spadesuit,3\spadesuit,\dots,10\spadesuit,J\spadesuit,Q\spadesuit,K\spadesuit)$ has all three of the face cards for spades. Similarly if all those cards happened to be hearts instead it would also count since we would have all of the face cards for hearts. Similarly still, a hand like $(J\spadesuit,Q\spadesuit,K\spadesuit,J\heartsuit,Q\heartsuit,K\heartsuit,J\diamondsuit,Q\diamondsuit,K\diamondsuit,\dots)$ would count since we have all of the face cards from spades (we also happen to have all of the face cards from hearts and diamonds too).
Let $\spadesuit$ represent the event that we have have all of the face cards from spades. Similarly, let $\diamondsuit, \heartsuit, \clubsuit$ represent the event that we have all of the face cards from diamonds, hearts, and clubs respectively.
You are asked to find $Pr(\spadesuit\cup\diamondsuit\cup \clubsuit\cup \heartsuit)$
To do this, let us apply inclusion exclusion. We expand the above as:
$Pr(\spadesuit\cup \diamondsuit\cup\clubsuit\cup\heartsuit) = Pr(\spadesuit)+Pr(\diamondsuit)+\dots-Pr(\spadesuit\cap \diamondsuit)-Pr(\spadesuit\cap \clubsuit)-\dots+Pr(\spadesuit\cap \diamondsuit\cap \clubsuit)+\dots-Pr(\spadesuit\cap\diamondsuit\cap \clubsuit\cap \heartsuit)$
Now, let us calculate each individual term in the expansion.
The calculation you did before is relevant. Indeed, we calculate $Pr(\spadesuit)=\dfrac{\binom{3}{3}\binom{49}{10}}{\binom{52}{13}}$. This is again merely the probability that we have all of the face cards from the spades and is not the final probability that we were tasked with calculating.
We continue and calculate more terms:
For example $Pr(\spadesuit\cap \diamondsuit)=\dfrac{\binom{6}{6}\binom{46}{7}}{\binom{52}{13}}$
We then notice what symmetry there is in the terms and can simplify some. Finally, we write the final expression for our final answer (and get an exact number only if actually requested or required, usually opting to leave the answer in terms of binomial coefficients without additional simplification).
The answer for $A$ seems to have approximation error. Here is how I look at $A$ getting to sum of $10$.
In one draw - gets one of the remaining $3$ cards with face value $5$.
In two draws - $(4,1)$ or $(3,2)$
In three draws - $(1, 1, 3)$ or $(2, 2, 1)$
In four draws - $(1, 1, 1, 2)$
In five draws - $(1, 1, 1, 1, 1)$.
So desired probability $ = \displaystyle \small \frac{3}{34} + 2 \cdot 2! \big(\frac{4}{34}\big)^2 + 2 \cdot \frac{3!}{2!} \big(\frac{4}{34}\big)^3 + \frac{4!}{3!} \big(\frac{4}{34}\big)^4 + \big(\frac{4}{34}\big)^5$
$ = \displaystyle \small \frac{437763}{2839714} \approx 0.154 $
Best Answer
Normal approximation:
Consider a single card draw from the deck:
E(X)=7
Var(X)=14
With the sum of 5 cards (drawn without replacement):
$E(X)=7 \times 5$
$\text{Var}(X)=14\times 5\times \frac{52-5}{52-1}$
(the last term being the 'finite population correction', which applies as much to the variance of the sum as it does to the variance of the mean).
The approach to normality in this situation is reasonably rapid.
This suggests the sum on 5 cards might be be roughly approximated by a normal distribution with mean $35$ and variance $70\times\frac{47}{51}\approx 64.5098$. Using a continuity correction this gives an approximate probability of totalling at least 40 on 5 cards of:
$$1-\Phi(\frac{39.5-35}{\sqrt{70\times\frac{_{47}}{^{51}}}})\approx 0.288$$
Simulation (in R) of ten million five-card draws indicates the probability is
about $0.293$ (with s.e. $\approx 1.4\times 10^{-4}$):
As a check on the earlier calculation of the variance, the standard deviation of of those simulated sums was 8.0316; the previous calculation gives 8.0318.
Edit: Here's a comparison of the empirical cdf of the simulated data with the above normal approximation; they're pretty close:
More extensive simulations are consistent with the other two answers based on complete enumeration: