These things can be pretty wild. For instance, for $n \geq 5$ the mapping class group of the $n$-torus is not even finitely generated : it is a split extension of $\text{GL}_n(\mathbb{Z})$ by an infinite rank abelian group. See Theorem 4.1 of
A. E. Hatcher, Concordance spaces, higher simple-homotopy theory, and applications. Proc. Sympos. Pure Math., XXXII, Part 1, pp. 3-21, Amer. Math. Soc., Providence, R.I., 1978.
and Example 4 of
Hsiang, W. C.; Sharpe, R. W.
Parametrized surgery and isotopy.
Pacific J. Math. 67 (1976), no. 2, 401–459.
I also recommend Hatcher's survey about the homotopy type of the diffeomorphism groups of manifolds here; it discusses a lot of things that are known about $\pi_0$.
It should be remarked that the above trouble happens because we are considering manifolds with infinite fundamental groups, which often causes terrible things to happen in high-dimensional topology. If $M$ is a compact simply-connected manifold of dimension at least $5$, then Sullivan proved that the mapping class group of $M$ is an arithmetic group, and hence has every possible finiteness property. This was extended to manifolds with finite fundamental groups by Triantafillou. For all of this, I recommend the survey
Triantafillou, Georgia
The arithmeticity of groups of automorphisms of spaces. Tel Aviv Topology Conference: Rothenberg Festschrift (1998), 283–306,
Contemp. Math., 231, Amer. Math. Soc., Providence, RI, 1999.
I think that the correct theorem should be the following:
If $L$ does not contain any split unknot or any two coaxial components, the image of the map is finite
Following Cameron Gordon, two coaxial components are two components $K_1$ and $K_2$ such that $K_1$ is an essential curve in the boundary $\partial N(K_2)$ of a solid torus regular neighbourhood of $K_2$. We also suppose that no other component of $L$ intersects $N(K_2)$.
Here is one example:
If you have two coaxial components, there is an incompressible annulus in $S^3\setminus L$ which is contained in $N(K_2)$, which winds once along $K_1$ and some $n$ times along $K_2$. The Hopf link corresponds to the case
where $K_2$ is the unknot and $n=0$.
Every time you have an incompressible annulus that connects two different components of the link you can do a Dehn twist along it and act non-trivially on the boundary. It has infinite order.
I think that the proof should go as follows: the only infinite-order automorphisms in a 3-manifold with boundary that arise as a link complement should be twists along discs or annuli. So $S^3\setminus L$ contains an essential disc or annulus. In the second case, since we are in $S^3$, the annulus arises necessarily from two coaxial components.
Best Answer
Let me assume that M is at least 5-dimensional.
Sullivan's proof only uses surgery theory and properties of O(n) that also hold for Top(n), so the answer to your first question is yes.
Regarding your other questions: the homotopy fiber Homeo(M)/Diff(M) of the map BDiff(M) -> BTop(M) is subject of smoothing theory. There is a map from Homeo(M)/Diff(M) to the space of sections of the bundle over the d-manifold M with fiber Top(d)/O(d) which is associated to the frame bundle of M using the action of O(d) on Top(d)/O(d). This map is injective on path components and an isomorphism on all homotopy groups. This can be found for instance in the book of Kirby and Siebenmann.
Up to a question about components, this reduces the study of the homotopy fiber you asked about to understanding the homotopy type of Top(d)/O(d) (and the twist of the bundle), which is hard. The homotopy groups of Top(d)/O(d) are object of study in geometric topology for a long time and can be divided into three ranges:
Later edit: The identification of $\pi_i(Top(d)/O(d))\otimes\mathbb{Q}$ in the range $i\lesssim \frac{4}{3}d$ mentioned above holds in fact for $i\lesssim 2d$ by Corollary 4.2 of Randal-Williams and Corollary C of Krannich. Beyond this range, progress in even dimensions has been made by Kupers--Randal-Williams and in odd dimensions by Krannich--Randal-Williams.