Let's consider an infinite dimensional space $X$ and a linear operator $T$. The resolvent operator of $T$ is $R_\lambda (T) = (T-\lambda I)^{-1}$. A regular value $\lambda$ of $T$ is a complex number such that:
(R1) $R_\lambda(T)$ exists
(R2) $R_\lambda(T)$ is bounded
(R3) $R_\lambda(T)$ defined on a set which is dense in $X$
The resolvent set $\rho(T)$ consists of all regular values $\lambda$ of $T$. The complement $\sigma(T)=C-\rho(T)$ is the spectrum of $T$ and we may distinguish parts of the spectrum:
point spectrum (eigenvalues) $\sigma_p(T)$: (R1) isn't satisfied
continuous spectrum $\sigma_c(T)$: (R2) isn't satisfied, but (R1) and (R3) are satisfied
residual spectrum $\sigma_r(T)$: (R3) isn't satisfied, (R1) is satisfied, (R2) – doesn't matter
Please, help me to clarify a couple of points:
Question 1: The point spectrum consists of eigenvalues and exists in finite dimensional case. So its meaning seems to be the same as in a finite dimensional case (scaling of eigenvectors that roughly represent orientation of the distortion by $T$). What is the meaning of the continuous and the residual spectrum?
Question 2: Why do we care about dense in the definitions? I have found a related question but didn't get the exact answer from it.
Best Answer
Question 1: If $\lambda$ is in the continuous spectrum, then $R_{\lambda}(T)$ is unbounded. That means there exists a sequence $\{ x_n \}$ such that $\|x_n\|=1$ and $\|R_{\lambda}(T)x_n\|\rightarrow\infty$. So, $y_n=\frac{1}{\|R_{\lambda}(T)x_n\|}R_{\lambda}(T)x_n$ is a sequence of unit vectors in $X$ such that $(T-\lambda I)y_n\rightarrow 0$ as $n\rightarrow\infty$. It is typical to refer to this sequence $\{ y_n \}$ of unit vectors as an "approximate eigenvector."
The reason one refers to this as "continuous spectrum" Historically had nothing to do with continuity; such spectrum was found to fill a continuum, rather than being discrete. For example, the multiplication operator $(Mf)(x)=xf(x)$ defined on $L^2[0,1]$ has no eigenvalues, but $[0,1]$ is in the continuous spectrum of $M$. You can see this because the step function $f_{\lambda,\delta}=\frac{1}{\sqrt{2\delta}}\chi_{[\lambda-\delta,\lambda+\delta]}$ is a unit vector for which $$ Mf_{\lambda,\delta} \approx \lambda f_{\lambda,\delta}, $$ in the sense that $$ \|(M-\lambda I)f_{\lambda,\delta}\|^2=\int_{\lambda-\delta}^{\lambda+\delta}(x-\lambda)^2f_{\lambda,\delta}^2dx \le 2\delta^2\|f_{\lambda,\delta}\|^2. $$ So $\|(M-\lambda I)f_{\lambda,\delta}\|\rightarrow 0$ as $\delta\rightarrow 0$ while $\|f_{\lambda,\delta}\|=1$.
The same type of spectrum occurs for the differentiation operator $\frac{1}{i}\frac{d}{dx}$ on $L^2(\mathbb{R})$ where you would like to identify $e^{isx}$ as an eigenvector with eigenvalue $s$, but it's not in the space. However, $s$ is in the continuous spectrum, a.k.a. approximate point spectrum. This is one of the earliest cases that motivated these definitions, where you could think of $$ F(x)=\int_{-\infty}^{\infty}c(s)e^{isx}ds $$ as a "continuous" (i.e. integral) sum of approximate eigenfunctions of the differentiation operator, with a coefficient function $c(s)$ defined on the continuum. And, when considered in $L^2$, the operator $\frac{1}{i}\frac{d}{dx}$ has continuous spectrum $\mathbb{R}$.
Question 2: For your second question, you don't care about a dense domain for $R_{\lambda}(T)$ if it is bounded because you can then extended it to the whole space. And, if $T$ is closable, then the extension of $R_{\lambda}(T)$ and the closure of $T$ remain inverses of each other. However, if $\lambda$ is an approximate eigenvalue and $T$ is closed, then $R_{\lambda}(T)$ must be unbounded, and you're basically stuck with a dense domain for $R_{\lambda}(T)$ because of the closed graph theorem.