Let $X_i$ be the value of the $i^{th}$ ticket drawn. Let $X$ denote the largest value drawn, and notice that $X = \max{ \{ X_1, ..., X_n \}}$. Let's try to find the pmf of $X$. This is actually easiest if we use CDFs.
$F_X(k)=P(X \leq k)=P(\max{ \{ X_1, ..., X_n \}} \leq k) = P((X_1 \leq k) \cap ... \cap (X_n \leq k))$.
The second equality holds because we need each of the $X_i$ to be less or equal to $k$ if we want the maximum to be less or equal to $k$.
Now notice that we are sampling the tickets with replacement, so each draw is independent. This allows us to say
$P((X_1 \leq k) \cap ... \cap (X_n \leq k)) = P(X_1 \leq k)\cdot \cdot \cdot P(X_n \leq k) = P(X_i \leq k)^n$.
Again, since we are sampling with replacement, he last equality holds because each draw has the same distribution. Now we know that each $X_i$ is a discrete uniform distribution on the set $ \{ 1, 2, ... N \}$, so, putting it all together, what we end up with is
$F_X(k) = P(X_i = k)^n = (k/N)^n$.
From this we can easily find $p_X(k)$, and then $E(X) = \sum_{k=1}^{N} kp_X(k)$
Addressing your comment as to whether the balls are drawn one at a time, or together: it doesn't matter.
If you draw the balls one at a time, you would probably think of choosing five numbers from ten, including the number $8$, with order being important. To count the number of favourable draws,
- choose a place for the $8$. . . . . . $5$ ways
- choose a digit greater than $8$ (so that $8$ is the second largest). . . . . . $2$ ways
- choose a place for this digit. . . . . . $4$ ways
- choose three digits below $8$, with order important. . . . . . $7\times6\times5$ ways.
To count the total number of draws:
- choose five digits, order important. . . . . . $10\times9\times8\times7\times6$ ways.
The probability is
$$\frac{5\times2\times4\times7\times6\times5}{10\times9\times8\times7\times6}=\frac{5}{18}\ .$$
If you draw them all together you are probably thinking of order as being irrelevant. The number of favourable draws is $2C(7,3)$, the total number of draws is $C(10,5)$, and the probability is
$$\frac{2C(7,3)}{C(10,5)}=\frac{2\times7\times6\times5}{3\times2\times1}\frac{5\times4\times3\times2\times1}{10\times9\times8\times7\times6}=\frac{5}{18}\ .$$
Best Answer
I think your issue is a language one. I will attempt to rephrase a specific case of the question in a different way so we can really see what the meaning of $n,m,k$ are.
(a)
As there are ten letters and we are choosing three of them, there are $\binom{10}{3}$ different selections that can be made. This will be our denominator.
As for the numerator, if we suppose that our largest letter chosen was an $F$ that means in particular that an $F$ was chosen and there are two more additional letters left to choose and those two additional letters must be smaller than $F$ otherwise $F$ wouldn't have been the largest letter chosen. That is, we look at how many ways there are to choose two letters from $\{A,B,C,D,E\}$ to fill out the remainder of our set of chosen letters. There are $\binom{5}{2}$ ways to do this.
This gives us a probability of $\dfrac{\binom{5}{2}}{\binom{10}{3}}$
It should hopefully be clear why the above example is the exact same as the problem of finding the probability that the largest number chosen was $m$ when $k$ balls are chosen out of $n$ available. In the above example we had $n=10$ balls available, we selected $k=3$ balls, and the largest ball was meant to be $m=6$. The same logic applied above shows that the probability that the maximum is $m$ is $\dfrac{\binom{m-1}{k-1}}{\binom{n}{m}}$
It is worth noting that $\binom{m}{k}-\binom{m-1}{k}=\binom{m-1}{k-1}$ so this answer matches both that of @lulu and @greedoid.
(b)
Again, there are $\binom{10}{3}$ ways to select three letters from our set of ten. This is again our denominator.
Choosing our three letters, we don't care what they are so long as they all come from the set $\{A,B,C,D,E,F\}$. There are six letters in this set and choosing three of them can be done in $\binom{6}{3}$ ways.
This gives a probability of $\dfrac{\binom{6}{3}}{\binom{10}{3}}$
Again, this should be a clear metaphor for the original problem where $n=10,m=6,k=3$. For arbitrary values of $n,m,k$ you will find using the same logic as above that the probability is $\dfrac{\binom{m}{k}}{\binom{n}{k}}$
It is worth pointing out that $\sum\limits_{i=k}^m\binom{i-1}{k-1}=\binom{m}{k}$ via the "hockeystick identity" and so this answer matches both that of @greedoid and that of @lulu.
Yes, $\binom{m}{k}$ represents the number of ways of choosing $k$ objects from $m$ objects. We are in the second problem as I have phrased it choosing $k$ balls from those balls numbered $\{1,2,3,4,\dots,m\}$. There is no requirement for $k$ to be equal to $m$ however. We merely want the largest appearing label to be less than or equal to $m$ (or equal to $m$ in the case of part (a)), we are not requiring that the number of balls selected be equal to $m$. That was why I chose to use letters in my metaphors. We wanted all letters chosen to be appearing at or before $F$ in the alphabet. When we were choosing three letters, the number $k=3$ is largely unrelated to the letter $F$ (the $m=6$'th letter of the alphabet).