Here is a slightly indirect way of obtaining the variance:
Let $X_k$ be the number on the $k$th ticket, $k=1,2,\ldots,m$.
So we have a uniform distribution for the $X_k$'s, namely
$$ P(X_k=j)=\begin{cases}\frac{1}{n}&,\text{ if }j=1,2,\cdots,n\\\\\,0&,\text{ otherwise }\end{cases}$$
So,
\begin{align}
\operatorname{Var}(X_k)&=E(X_k^2)-(E(X_k))^2
\\\\&=\frac{n^2-1}{12}=\sigma^2\,,\text{ say }
\end{align}
If the correlation between $X_i$ and $X_j$ $\,(i\ne j)$ be $\rho$, then $$\rho=\dfrac{\text{Cov}(X_i,X_j)}{\sigma^2}$$
You are looking for \begin{align}\operatorname{Var}(X)&=\operatorname{Var}\left(\sum_{k=1}^m X_k\right)\\&=\sum_{k=1}^m \operatorname{Var}(X_k)+2\sum_{i<j}\text{Cov}(X_i,X_j)\\&=m\sigma^2+2\binom{m}{2}\rho\sigma^2
\\&=m\sigma^2(1+(m-1)\rho)\tag{1}\end{align}
Now note that the joint distribution of $(X_i,X_j)\,,i\ne j$ is independent of $m$.
So we see that
\begin{align}
\operatorname{Var}\left(\sum_{k=1}^{\color{red}{n}}X_k\right)&=\operatorname{Var}(\text{constant})=0
\\&\implies\color{red}{n}\sigma^2(1+(\color{red}{n}-1)\rho)=0
\\&\implies\rho=\frac{1}{1-n}
\end{align}
Substituting this value of $\rho$ and the value of $\sigma^2$ in $(1)$, we finally get the variance of $X$ as
$$\operatorname{Var}(X)=\frac{m(n+1)(n-m)}{12}$$
I wrote the probability as a fraction with denominator
$\binom {52} 5$. For the numerator I wrote $\binom {12}3\binom {40}{2}$. My answer was approximately $.0660$.
Yes. $\left.\binom {12}3\binom{40}{2}\middle/\binom{52}{5}\right.$ is the probability for selecting three from the twelve face cards and two from the forty non-face cards, when drawing any five from all fifty-two cards without replacement. That is the probability for the event you sought, as there are indeed those counts for face and non-face cards among a standard deck.
Best Answer
There are three mistakes in your calculation. In the numerator, you're taking the order of the three tickets into account, and in the denominator you aren't. The upper index of the binomial coefficient in the numerator, if you do take order into account, should be $36+3-1$, not $36-1$. And you're counting sums where the same ticket is used twice.
To get it right, you could calculate the number of ordered triples that sum to $36$, allowing equal parts, which is $\binom{36+3-1}{3-1}-3=700$ (subtracting $3$ because there's no ticket numbered $36$), then subtract the number of these with three equal numbers, which is $1$, and the number with exactly two equal numbers, which is $3\cdot17=51$ (the triples of the form $(x,x,36-2x)$ with $x=1\ldots18$ except $x=12$, with $3$ different orders each), and then divide by $6$ for the $6$ different orders to get
$$ \frac{700-1-51}6=108\;. $$
Now you can divide this by the number of ways to select 3 tickets to get the required probability.