a) The first award can go to one of $5$ students, the second to $4$ and the third to $3$, hence your answer of $60$ is correct.
b) For each award, choose a student it goes to. This results in $5\times5\times5=125$ ways. Remove the $5$ ways where all $3$ awards go to the same person, and it results in $120$ ways in total.
Another way by cases: There are $60$ ways the awards go to 3 students, if the awards go to different students. If a student gets 2 awards, there 3 ways to split the awards $2$ to $1$. The group of $2$ can be given to one in $5$ students, and the group of $1$ can be given to one in $4$ students, making a total of $60+3\times4\times5=120$ ways.
a) Pick who the referee is in $11$ ways. Then among the ten people leftover, one will be the youngest. Pick who the other four members of that player's team is in $\binom{9}{4}$ ways. We get a total of $11\binom{9}{4}=1386$ ways.
b) Pick who the referee is in $11$ ways. Simultaneously pick the two captains in $\binom{10}{2}$ ways. Pick who the other four members of the youngest captain's team is in $\binom{8}{4}$ ways. We get a total of $11\binom{10}{2}\binom{8}{4}=34650$ ways.
Alternate approach:
a) Arrange five a's, five b's, and a c in a line in $\frac{11!}{5!5!1!}$ ways. Have the players go to either team $A$, team $B$, or referee position according to the arrangement of the a's, b's, and c's compared to their names. Now, "forget" which team was actually team $A$ and which team was team $B$ by dividing by two. This gives us $\frac{11!}{5!5!1!}\cdot\frac{1}{2}=1386$ ways.
b) Arrange four a's, four b's, one A, one B, and one C in a line in $\frac{11!}{4!4!1!1!1!}$ ways. The a's will correspond to which players are normal members of team $A$, the A will correspond to the player who is captain for team $A$ etc... Finally, "forget" which team was actually team $A$ and which was team $B$ by dividing by two. This gives us $\frac{11!}{4!4!1!1!1!}\cdot\frac{1}{2}=34650$ ways.
Similarly, one could instead just take the answer to part (a) and then pick who from the youngest person's team is captain and pick who the captain is for the other team, making it so there is $5\cdot5\cdot1386=34650$ ways.
Best Answer
For the first problem (any number of awards), line up the $5$ trophies. There are $30$ choices for who receives the first trophy. For each such choice, there are $30$ choices for who receives the second. So there are $30^2$ ways to award the first two trophies. For each such way, there are $30$ ways to award the third trophy, and so on, for a total of $30^5$.
For the second problem, the reasoning is almost the same. There are $30$ ways to award the first trophy. For each such way there are $29$ ways to award the second trophy, and so on for a total of $(30)(29)(28)(27)(26)$.
There are other approaches to the second problem. For example, we can choose the $5$ people who will get trophies in $\binom{30}{5}$ ways. For each such choice, the decision as to who gets what can be done in $(5)(4)(3)(2)(1)=5!$ ways, for a total of $\binom{30}{5}5!$. This expression, when evaluated, yields $(30)(29)(28)(27)(26)$.
Remark: In order to get the right answer, it can be very useful to visualize the awards process in a very concrete way. So for the first problem, think of five medals, gold, silver, bronze, plastic, cardboard. The cardboard medal can be awarded in $30$ ways. For each such way, the plastic medal can be awarded in $30$ ways, and so on.