You can divide your task in steps, count the ways that you can conduct each step and then use the multiplication principle to count the total number of ways that you can conduct your task.
a) You can do it in two steps.
1.Step. Order your books. You can do it in $25!$ ways.
2.Step. Decide on how many books will be placed on each shelf. That is equivalent to writing on piece of paper 10 nonnegative (zeros are here allowed in c they are not!) integers $x_1,x_2,\dots,x_{10}$ so that they add up to 25. For $i=1,2,\ldots,10$, $x_i$ represents the number of books that will be placed on the $i$-th shelf. The total number of ways that you can conduct this step is equal to the number of solutions to the equation $$x_1+x_2+\ldots+x_{10}=25,$$ with $0\leq x_i, i=1,2,\ldots,10$. This is a typical stars and bars problem with $n=10$ and $k=25$. Then (according to the Wikipedia page, Stars and Bars, Theorem two) you can do that in $$\dbinom{n+k-1}{n-1}=\dbinom{10+25-1}{10-1}=\dbinom{34}{9}$$
So in sum the task in (a) can be conducted in $25!\times\dbinom{34}{9}$ ways.
c). Based on a) the answer is straightforward by allowing in Step 2. only positive values for the $x_i, i=1,2,\ldots,10$. The total number of ways that you can conduct this step is equal to the number of solutions to the equation $$x_1+x_2+\ldots+x_{10}=25,$$ with $1\leq x_i, i=1,2,\ldots,10$. Then (according to the Wikipedia page, Stars and Bars, Theorem one) you can do it in $\dbinom{k-1}{n-1}=\dbinom{25-1}{10-1}$ ways.
So in sum the task in (c) can be conducted in $25!\times\dbinom{24}{9}$ ways.
b) 25C25 is equal to 1 so, your approach here is not correct. Here you can think it as follows. Write on each book a number from 1 to 10 (this number indicates the shelf that it will be placed). So you can do it $10\times10\times\ldots\times10=10^{25}$ ways. (With steps as in the previous questions: 1.Step Choose a shelf for book Nr1. You can do it in 10 ways. 2.Step Choose a shelf for book Nr2. You can do it in 10 ways and so on until 25.Step Choose a shelf for book Nr25. You can do it in 10 ways. So by multiplication rule you can do it in $10^{25}$ ways.)
Arrange the non-maths books, $7!$ ways. There are now $8$ gaps between books (including a "gap" at each end of the row). Choose one of these gaps for the first maths book, a different gap for the second and a different gap for the third.
Answer: $7!\times8\times7\times6$.
Best Answer
You are going in the right direction, but don't think in terms of slots. Just organize the books on the floor, then put them on the shelf.
First, order the computer science books. There are $5!$ ways of putting them in. Then put the two art books on both ends; there are $2$ ways of doing it.
Now you have $7$ books, with eight spaces between and to the left and right of them. We just need to put in the three math books. Let's say they are a Calculus I, a Calculus II, and a Calculus III textbooks.
We put in the Calc I book first. There are 8 places where it can go, relative to the other 7 books. Next, put the Calc II book in; there are 9 places where it can go, relative to the other 8 books that have already been placed. Finally, put in the Calc III book.