There may be some problems in properly define the derivative for arbitrary unbounded operators. This is because as far as I know there is no suitable definition of topology on the set of unbounded operators.
If we restrict to closed operators (such as self-adjoint operators) acting on a Hilbert space $H$, then it is possible to define a metric. The set of closed operators becomes then a (non-complete) metric space $\mathcal{C}(H)$. Before discussing (briefly) what the metric is, let me remark that $\mathcal{C}(H)$ is not a linear space, for in general it is not possible to sum two closed unbounded operators. The distance between closed operators $T$ and $S$ is defined, roughly speaking, as the gap between the graphs $G(T)$ and $G(S)$. The graph of an operator is a closed linear manifold in $H\times H$ defined by
$$G(T)=\{(\varphi,\psi) \in H\times H \;, \; \varphi\in D(T)\; , \; \psi=T\varphi \}\; .$$
For all the details of the definition, see e.g. Kato's book of 1966 on perturbation theory of linear operators.
In $\mathcal{C}(H)$, we have thus a notion of convergence $T_n\to T$. Convergence in this sense (called by Kato "generalized sense") extends roughly speaking the convergence in norm of bounded operators. If the resolvent set $\varrho(T)$ of $T$ is not empty, the generalized convergence is equivalent to convergence in the norm resolvent sense, i.e. it is equivalent to the convergence in norm of the resolvents (as bounded operators):
$$T_n\to_\mathrm{gen} T \Leftrightarrow (T_n-z)^{-1}\to_{\mathrm{norm}} (T-z)^{-1}\; ,\; \forall z\in \varrho(T)\; .$$
More precisely, there exists an $n^*\in \mathbb{N}$ such that $z\in \varrho(T_n)$ for any $n\geq n^*$, and the convergence of resolvents holds. Of course convergence in the generalized sense is equivalent to convergence in norm if the operators are bounded.
Nevertheless one has still a problem in defining the derivative, since as I remarked before it is not in general possible to sum two closed operators and obtain another closed operators. It is possible to give abstract conditions on (densely defined) $T$ and $S$ for them to densely define a closed operator $T+S$, see this paper. However as you may notice, things are getting messier and messier. Anyways, let $T_0\in \mathcal{C}(H)$ be a fixed densely defined closed operator. We denote by $\mathcal{C}_{T_0}(H)$ the set
$$\mathcal{C}_{T_0}(H)=\{T\in \mathcal{C}(H), T-T_0\in \mathcal{C}(H)\}\; .$$
Remark that $\mathcal{C}_{T_0}(H)$ may as well be empty. Nevertheless, let now $\alpha:\mathbb{R}\to \mathcal{C}_{T_0}(H)$ for some $T_0$, and $\alpha(x)=T_0$. Then the derivative $\alpha'(x)$ can be defined in the usual way since $h^{-1}\Bigl(\alpha(x+h)-\alpha(x)\Bigr)$ is a closed operator:
$$\alpha'(x)=\lim_{h\to 0}h^{-1}\Bigl(\alpha(x+h)-\alpha(x)\Bigr)\; ;$$
where the limit is intended in the generalized sense (provided it exists). However, we are still not assured that the derivative makes sense in another point $x'\neq x$, if $\alpha(x')\neq T_0$!
As a matter of fact, I actually never saw this construction applied in any concrete physical or mathematical problem, and maybe it is never used.
As a final remark, the derivative of functions with values in the continuous (bounded) linear operators are used very often. In this case, the derivative can be intended in any topology of the bounded operators, such as e.g. the norm topology (that would be equivalent to the construction above and the OP already noted); but also in the strong topology, or in the weak one. As a matter of fact, derivatives may sometimes exist in the strong or weak sense, but not in the norm sense.
Best Answer
I'm not sure what you are after, but, normally, for finite, as for infinite-dimensional operators, one uses some version of Sylvester's formula, and one investigates convergence properties afterwards.
If, by "its own eigenbasis", you mean the eigenbasis of momentum, $$ |\Psi\rangle= \int\!\!dp~~|p\rangle \langle p| \Psi\rangle, $$ then you evidently have $$\sqrt{I+b\hat p^2}|\Psi\rangle= \int\!\!dp~~|p\rangle ~\sqrt{1+b p^2} ~\langle p| \Psi\rangle,$$ as you appear to anticipate. (In sadistic undergraduate QM, in momentum space, this presents as $f(p)\Psi(p)$.)
You may be familiar with the operator based on the Casimir operator of angular momentum, $$ \sqrt{\hat L^2+1/4} -1/2 , $$ whose eigenvalues are $\ell$ when acting on irreducible spin multiplets.