So for the first question you need to place a rook in each column. The rook in the first column can go in $8$ places, the rook in the second column then has $7$ possible rows etc.
In the second part, you need first to choose the $8$ columns out of the $10$ available, and then place a rook in each of the columns. If once again you move from left to right, counting the number of possibilities, you should be fine.
Rather than talking about "any particular" column or row, you should make a habit of defining the scheme more closely (the first, second rather than "any" etc). This will help to avoid a great deal of confusion and can also highlight any double-counting.
The flaw in the reasoning is because you write "the second rook can be placed in $7$ different rows and you can choose $6$ different columns and so on" which is not actually correct - and should stand out as a red flag because no justification is given to a pretty non-obvious and important fact in your argument. Also, your "and so on" hides what happens to the last rook in your argument, which you would say can be put into $1$ row and $0$ columns - which is clearly wrong!
Suppose we put the coordinates on the grid ranging from $(1,1)$ to $(8,8)$ where the diagonal in question is those points of the form $(n,n)$. If you put the first rook at $(1,2)$, your claim is that there are $42$ valid positions for the second rook - but this is not so! More specifically, you claim that we can fix the first coordinate in $7$ ways and then will have $6$ choices for the second coordinate - but this does not hold. In particular, if we choose the first coordinate for the second rook to be $2$, we find that all positions $(2,x)$ are legal except for $(2,2)$ - which is both attacked by the the first rook and on the main diagonal. Oops - so there are actually $43$ valid positions for the second rook!
Patching this argument turns out to be really hard because the number of valid positions for the next rook will, in general, depend on the placement of the previous rooks - so finding another approach is warranted. (For instance, one can count the number of arrangements of rooks that do include the diagonal and also the total number of arrangements of rooks, then subtract. It's also to get a recurrence relation by considering that each square on the diagonal is attacked by two rooks - which then means you have some sort of relation on the rooks which is useful to counting the number of possible placements)
Best Answer
Your partial work of $~\left[\binom{5}{3}\right]^2 = 100$
is the right starting point.
In effect, you have determined that there are 100 possible ways of choosing the pertinent (3 rows) and (3 columns).
Now, you have to consider how many ways are there of placing 3 rooks in a 3 $\times$ 3 chessboard, so that they don't attack each other.
Let $R_i$ denote the rook placed in row $i$.
Then, there are three choices (i.e. columns) that $R_1$ can be placed in, and then two (remaining) columns for $R_2.$
Final answer:
$$\left[\binom{5}{3}\right]^2 \times 3! = 600.$$