In most quantum mechanics text books, the resolution of the identity or completeness relation is stated in the following (or similar) form
$$ \mathbb I_\mathcal H = \sum\limits_n |\lambda_n\rangle \langle \lambda_n| + \int |\lambda\rangle\langle \lambda| \,\mathrm d\lambda\quad , \tag{1} $$
where $|\lambda_n\rangle$ and $|\lambda\rangle$ denote the (generalized) eigenvectors of some observable $O$ (here a non-degenerate case is assumed). I wonder why or under which conditions we can do this:
In general, the spectral theorem states (at least one formulation of it) that for a given self-adjoint operator, there exists a (unique) spectral family $\{E_\lambda\}_{\lambda\in \mathbb R}$ such that
$$ O =\int\limits_\mathbb R \lambda\,\mathrm dE_\lambda \quad . \tag{2} $$
For the identity operator, we thus obtain
$$\mathbb I_\mathcal H = \int \limits_\mathbb R \mathrm dE_\lambda \quad .\tag{3} $$
I think that in order to arrive at $(1)$, we have to be able to write, at least formally, in the continuous part of the spectrum: $$\mathrm dE_\lambda = \frac{\mathrm dE_\lambda}{\mathrm d\lambda} \mathrm d \lambda = |\lambda\rangle \langle \lambda| \, \mathrm d \lambda \quad , \tag{4}$$
which suggests to me that $E_\lambda$ must be differentiable and hence continuous. Am I right to think that this, probably among other things, means that in the range of the integration in $(1)$, there cannot be an eigenvalue embedded in the continuum$?^\dagger$ I think I could make sense of $(1)$ if the spectrum is split like e.g. in the case of the hydrogen atom: For all energies below $0$, we have bound states and hence eigenvalues and eigenvectors and then the continuum part follows, not 'interrupted' by bound states again, such that we could integrate from $0$ to $\infty$.
But what happens if there is an eigenvalue embedded in the continuum? Are the corresponding projectors already in the sum in $(1)$? Then how can one appropriately split the integration domain?
My concern is due to the fact that at an eigenvalue $\lambda_k$, $E_{\lambda_k}$ is not left continuous (although right continuous by definition), so I guess there could arise some problems.
Summing up: When and why can we explicitly split the discrete (more precisely point (?)) and continuous part of the spectrum in the resolution of the identity in $(1)$? In the range of the integration, is it assumed that there are no eigenvalues embedded in the continuum? If this holds true, how would the resolution of the identity look like in the case of embedded eigenvalues?
My knowledge in functional analysis as well as measure theory is very limited, hence my naive question. I know that many formulations are not precise and mathematical rigorous.
$^{\dagger}$ In chapter $2$, page $48$ of Quantum Mechanics I. Galindo and Pascual. Springer it is noted that the in order to write $(1)$, it is assumed that the corresponding operator has no continuous singular spectrum, but I don't know enough of the math such that I could make sense of this (and it isn't even defined in the book).
In all other text books (for physicists) of quantum mechanics I've looked into, if the spectral theorem was elaborated at all, then only the cases of a purely continuous and purely discrete spectrum are discussed, c.f. chapter $1$ of Quantum mechanics: A modern development. Ballentine. World Scientific or section $4.5$ in Quantum Theory: Concepts and Methods. Peres. Kluwer Academic. The examples presented there make sense to me, but no spectral family for an operator with mixed spectrum is constructed, which seems to be part of my problem in understanding this question.
Best Answer
The problem with the raised issue is that it is mainly formulated in the non-rigorous (but very effective) jargon of physicists. However, it can be re-formulated into a more mathematically precise version and the answer is straightforward.
First of all, if $A : D(A) \to H$ is selfadjoint, its spectum $\sigma(A)$ is the disjoint union of the continuous part of the spectrum $\sigma_c(A)$ and the point spectrum $\sigma_p(A)$. The latter is the set of eigenvectors.
Generally speaking, if $P: {\cal B}(\mathbb R) \to \mathfrak{B}(H)$ is a projection-valued measure defined on the Borel sets of the real line and $A,B$ are Borel set with $A \cap B =\emptyset$, we have $P_AP_B=0$ and $$P_{A \cup B}= P_A + P_B\:.$$
Specializing to the case of $A= \sigma_p(A)$ and $B= \sigma_c(A)$ and $P=P^{(A)}$, the spectral measure of $A$, we conclude that: $$P_{\sigma(A)}= P_{\sigma_p(A)} + P_{\sigma_c(A)}\:.$$ This identity propagates immediately to the spectral calculus giving rise to $$A = \int_{\sigma_p(A)} \lambda dP^{(A)}(\lambda) + \int_{\sigma_c(A)} \lambda dP^{(A)}(\lambda)$$ where ($s-$ denoting the convergence in the strong topology) $$\int_{\sigma_p(A)} \lambda dP^{(A)}(\lambda) = s-\sum_{u\in N} \lambda_u |u\rangle \langle u|$$ and $N$ is any Hilbert basis of the closed subspace $P^{(A)}_{\sigma_p(A)}(H)$ (viewed as a Hilbert space in its own right) made of eigenvectors $u$ with eigenvalues $\lambda_u$ (we may have $\lambda_u=\lambda_{u'}$ here). It does not matter if $\lambda \in \sigma_p(A)$ is embedded in $\sigma_c(A)$, the above formulas are however valid. Everything is true when considering every Borel-measurable function $f$ of $A$: $$f(A) = \int_{\sigma_p(A)} f(\lambda) dP^{(A)}(\lambda) + \int_{\sigma_c(A)} f(\lambda) dP^{(A)}(\lambda)$$ where $$\int_{\sigma_p(A)} f(\lambda) dP^{(A)}(\lambda) = s-\sum_{u\in N} f(\lambda_u) |u\rangle \langle u|\:,$$ with the special case $$I = \int_{\sigma_p(A)} dP^{(A)}(\lambda) + \int_{\sigma_c(A)} dP^{(A)}(\lambda)$$ where $$\int_{\sigma_p(A)}dP^{(A)}(\lambda) = s-\sum_{u\in N} |u\rangle \langle u|$$
A mathematical remark helps understand why embedded eigenvalues do not create problems. The following facts are valid for $\lambda \in \mathbb{R}$.
(a) $\lambda \in \sigma_p(A)$ if and only if $P^{(A)}_{\{\lambda\}} \neq 0$,
(b) $\lambda \in \sigma_c(A)$ if and only if $P^{(A)}_{\{\lambda\}}= 0$, but $P^{(A)}_{(\lambda-\delta, \lambda+\delta)}\neq 0$ for $\delta>0$
You see that the spectral measure $P^{(A)}$ cannot see the single points of $\sigma_c(A)$. And, if it sees a point, that point stays in $\sigma_p(A)$.
Furthermore, independently of the fact that some eigenvalues may be embedded in the continuous spectrum, $P^{(A)}_{\sigma_c(A)}(H)$ and $P^{(A)}_{\sigma_p(A)}(H)$ are orthogonal subspaces of $H$.
ADDENDUM
A projector-valued measure (PVM) is a map $P : \Sigma(X) \to \mathfrak{B}(H)$ associating the sets $E$ of the $\sigma$-algebra $\Sigma(X)$ over $X$ to orthogonal projectors $P_E: H \to H$ on the Hilbert space $H$ such that
(1) $P(X)=I$,
(2) $P_EP_F = P_{E\cap F}$ if $E,F \in \Sigma(X)$,
(3) $P_{\cup_{j\in N} E_j}x = \sum_{n\in N}P_{E_n}x$ for every $x\in H$ and any family $\{E_n\}_{n\in N} \subset \Sigma(X)$ where $N \subset \mathbb{N}$ and $E_n \cap E_m = \emptyset$ if $n\neq m$. (If $N$ is infinite, the sum refers to the topology of $H$.)
Given a PVM $P: \Sigma(X) \to \mathfrak{B}(H)$ and a pair of vectors $x,y\in H$, the map $$\mu^{(P)}_{xy} : \Sigma(X) \ni E \mapsto \langle x| P_Ey\rangle \in \mathbb{C}$$ turns out to be a complex measure (with finite total variation). If $f:X \to \mathbb{C}$ is measurable, there exists an operator denoted by $$\int_X f dP : \Delta_f \to H$$ which is the unique operator such that $$\left\langle x \left| \int_X f dP y \right. \right\rangle = \int_X f(\lambda) d\mu^{(P)}_{xy}(\lambda)\quad x \in H\:, y \in \Delta_f\tag{1}$$ where the domain $\Delta_f$ is the dense linear subspace of $H$ $$\Delta_f := \left\{ y \in H \left| \int_X |f(\lambda)|^2 d\mu^{(P)}_{yy}(\lambda) < +\infty\right.\right\}\:.$$ With this definition, which is valid for every choice of $X$ and the $\sigma$-algebra $\Sigma(X)$, the spectral theorem for selfadjoint operators reads
THEOREM If $A : D(A) \to H$ is a selfadjoint operator with domain $D(A) \subset H$, there exists a unique PVM $P^{(A)} : {\cal B}(\mathbb{R}) \to \mathfrak{B}(H)$, where ${\cal B}(\mathbb{R})$ is the Borel $\sigma$-algebra on $\mathbb{R}$, such that $$\int_{\mathbb R} \lambda dP^{(A)}(\lambda) = A\:.$$ More precisely, the support of $P^{(A)}$ (i.e., the complement of the largest open set $O \subset \mathbb{R}$ with $P_O=0$) is exactly $\sigma(A)$.
Remarks
(1) With this result it is natural to define $$f(A) := \int_{\mathbb R} f(\lambda) dP^{(A)}(\lambda)=: \int_{\sigma(A)} f(\lambda) dP^{(A)}(\lambda)\:,$$ for every complex-valued Borel-measurable map defined on $\mathbb{R}$ (or more weakly on $\sigma(A)$).
(2)The above spectral theorem is still valid if replacing $A$ for a closed normal operator (in particular unitary operators). In that case $X= \mathbb{C}$. In the selfadjoint case $X = \mathbb{R}$ and, in this special case, the definition of a PVM can be given in terms of Stiltjes integrals, but this option is very limitative and for that reason it is rarely used nowadays.
(3) The notation $P(d\lambda)$ in place of $dP(\lambda)$ is a bit misleading (though the relevant object is the operator in the left hand side of Eq.(1) and not $dP$) since it refers to an apparently given measure $d\lambda$. But there is no such special choice in general. The standard measures arise once you give the vectors $x$ and $y$ and not before. In general there is no relation between $P^{(A)}$ and the Lebesgue measure $d\lambda$ on $\mathbb{R}$. Even if some forced (and sometime useful) relation can be found when comparing, e.g. $\langle x| P^{(A)}_E y \rangle$ and the Lebesgue measure $\int_E d\lambda$ and taking advantage of Lebesgue's theorem of decomposition of measures. This use leads to a further decomposition of the spectrum into absolutely continuous, singular continuous, and pure point parts. This decomposition, though useful in several contexts, is different from the original one into continuous, point, and residual which does not exploit the Lebesgue measure in any sense and that I used in my answer. Unfortunately the sloppy language of physicists (I stress that I am a physicist!) does not permit to always appreciate some subtleties of these different decompositions and leads to misunderstandings and sometimes unmotivated issues. For instance the notions of discrete spectrum and point spectrum are different though they are often used as synonyms in the physical literature; the same happens for absolutely continuous spectrum and continuous spectrum.