I will give you the argument that is presented in the book I learned from, Physical Gas Dynamics. For reference, it's page 104, Chapter 4 section 6. I present it instead because it does not rely on the Boltzmann distribution for the derivation so perhaps there is some additional insight for you. They established in section 5 that:
$$ \frac{N_j^*}{C_j} = \frac{1}{e^{\alpha + \beta \epsilon_j} \mp 1} $$
for the BE and FD statistics respectively. The other notation: $N_j^*$ is the number of particles in the energy state $j$, $C_j$ is the number of possible levels and the coefficients $\alpha$ and $\beta$ have yet to be determined.
At "sufficiently high" temperatures, $C_j \gg N_j^*$ and the only way for that to be true is if the denominator is large. And when the denominator is large, the exponential is much bigger than 1 so that term may be neglected. So both FD and BE statistics lead to the same expression:
$$ \frac{N_j^*}{C_j} = e^{-\alpha - \beta \epsilon_j} $$
Then they apply this to the number of microstates (after Sterling's approximation):
$$ \ln W = \Sigma_j \left[ \pm C_j \ln \left(1\pm\frac{N_j}{C_j}\right)+N_j\ln\left(\frac{C_j}{N_j}\pm 1\right)\right]$$
where the $\pm$ is for BE and FD respectively. So they plug in the expression for $N_j^*/C_j$ and further give the approximation: $\ln\left(1 \pm x\right) \approxeq \pm x$ for $x \ll 1$ to arrive at the Boltzmann limit:
$$ \ln W = \Sigma_j N_j \left( \ln \frac{C_j}{N_j} + 1 \right)$$
So the entire derivation really just hinges on the notion that at "sufficiently large" temperatures, there are considerably more energy states than particles and so the vast majority of them are empty. Furthermore, both FD and BE statistics have the same result because so many states are empty that the likelihood of two particles attempting to occupy the same level is very low.
This entire derivation relies only on information about quantum statistics and does not require the hand-waving needed to reach the same conclusion from classical mechanics like the derivation from the Boltzmann distribution does (so say the authors of the book immediately following the derivation).
It goes on further to say that $\epsilon_j$ has a wide range of values, including zero, and so the only way for the exponential to be large for all states is to have $\alpha \gg 1$. This is equivalent to (and it's left as an exercise):
$$ \frac{V}{N} \times \frac{\left(2 \pi m k T\right)^{3/2}}{h^3} \gg 1$$
which is violated for very small $m$ with a very large number density $N/V$.
Then it's fairly straight forward to eliminate $\alpha$ and come up with an expression with only $\beta$, which is not possible to solve practicably. So then they use the expression for the maximum number of microstates and perturb the solution about $N_j^*$ and neglect terms second or and higher of $\Delta N_j$.
Based on all of that, it is found that $\beta = \frac{1}{k T}$. But this hinges on the assumption that all possible microstates of a system are a priori equally probable.
Obviously I skipped a bunch of steps and equations in the derivation, but I highly recommend this book (or any other that shows the derivation from the quantum mechanics side rather than from the Maxwell-Boltzmann distribution side).
Unless you are in the low temperature limit, then the F-D distribution should be written as
$$ F(E) = [\exp (E-\mu)/kT +1]^{-1},$$
where $\mu$ is the chemical potential.
I think what you are missing is that the chemical potential is not a constant, it is temperature-dependent. At high temperatures then $\mu < 0$ and $\exp(-\mu/kT) \gg 1$. You could have a look at this answer to another question for an explanation of why that is.
Thus
$$[\exp (E-\mu)/kT +1]^{-1} \simeq [\exp(-\mu/kT)\exp(E/kT)]^{-1} =\exp(\mu/kT)\exp(-E/kT) $$
which is the Boltzmann distribution and is $\ll 1$ for all $E$.
Best Answer
If we neglect the possibility of negative temperature, then OP is right:
The Fermi-Dirac distribution
$$f_{FD}(\epsilon) ~\longrightarrow ~\Theta(\epsilon_F - \epsilon) \qquad \text{for}\qquad T ~\longrightarrow ~0^{+}, $$
where $\Theta$ is the Heaviside step function.