The classical operator
$$
L = -\frac{d^{2}}{dx^{2}}+V,\;\;\; a \le x \le b,
$$
is different if $V$ is very singular. If $V \in L^{1}[a,b]$, then things are nice because there are 2 linearly-independet classical solutions of $Lf = \lambda f$ for every $\lambda$. That is, such solutions are continuous on $[a,b]$, their first derivatives are continuous on $[a,b]$, and their first derivatives are absolutely continuous with $-f''+Vf=\lambda f$. In that case, everything is nice. Whether you impose separated endpoint conditions
$$
Af(a)+Bf'(a) =0, \;\;\; Cf(b)+Df'(b)=0,
$$
or periodic conditions
$$
f(a)=f(b),\;\;\; f'(a)=f'(b),
$$
there is a discrete set of eigenvalues $\lambda_{1} < \lambda_{2} < \lambda_{3} < \cdots$ for which eigenfunctions exist satisfying these conditions, and there is a complete orthonormal system of eigenfunctions in $L^{2}[a,b]$. The eigenfunction expansions have the same pointwise convergence properties as the trigonometric series where $V=0$.
If $V$ is not integrable on $[a,b]$, then things change, and the situation is different on the half-line, too. The typical way to deal with a strong singularity of $V$ is to put the singularity at $b$, and to assume that $V$ is integrable on $[a,b-\epsilon)$ for all $\epsilon > 0$. If the potential is well-behaved in the finite plane, but unbounded at $\infty$, then you might consider the problem on $[a,b=\infty)$, for example.
The interesting part of the analysis in the singular case of $[a,b)$ is that there always exists one classical eigenfunction for eigenvalue $\lambda \notin\mathbb{R}$ that is in $L^{2}$ near the singular endpoint. But it can still happen that both eigenfunctions for $\lambda\notin\mathbb{R}$ are in $L^{2}[a,b)$. (These are classified as limit point or limit circle cases, and have to with how these theorems were originally proved by Weyl ~1905 using purely classical methods.) In the limit point case, one need only impose a single classical condition at $a$, and assume that all functions in the domain of the operator are in $L^{2}[a,b)$ in order to obtain a selfadjoint operator. In the limit circle case where both classical eigenfunctions for $\lambda \notin\mathbb{R}$ are in $L^{2}[a,b)$, then an additional condition must be imposed near $b$; such a condition often has to do with asymptotic behavior of functions near $b$, and is rarely obvious. In any event, after imposing the required conditions, the operator with domain of functions satisfying these conditions becomes selfadjoint.
The existence of one eigenfunction in $L^{2}$ allows one to construct a resolvent $(\lambda I-L)^{-1}$ for $\lambda\notin\mathbb{R}$ using the classical Green function solution. Then the spectral measure associated with $A$ is constructed as a generalized residue on $(\alpha,\beta)$:
$$
E(\alpha,\beta)f = \lim_{\epsilon\downarrow 0}\frac{1}{2\pi i}
\int_{a}^{b}((u-i\epsilon)I-L)^{-1}f-((u+i\epsilon)I-L)^{-1}f\,du.
$$
The singularities of the resolvent $(\lambda I - L)^{-1}$ for the regular problem are poles of order 1 and the residues are the projections onto the eigenspaces. In the singular cases, continuous singularities (such as branch cuts) can arise, and you can end up with integral expressions instead of discrete ones, or in addition to discrete ones.
Eigenfunction Expansions and Functional Calculus: On the interval $[a,b)$ ($b$ finite or infinite) with regular endpoint at $a$, you end up with
$$
f(x) = \int_{\sigma}\left(\int_{a}^{b}f(t)\phi_{\lambda}(t)\,dt\right)\phi_{\lambda}(x)\,d\mu(\lambda)
$$
In reasonable applications, the measure $\mu$ has two components: a continuous component and a discrete one. The singular continuous component is realistically absent. So, you can write this as
$$
f(x) = \sum_{n}w_{n}\int_{a}^{b}f(t)\,\phi_{\lambda_{n}}(t)\,dt\,\phi_{\lambda_{n}}(x)+\int_{-\infty}^{\infty}\left(\int_{a}^{b}f(t)\phi_{\lambda}(t)\,dt\right)\phi_{\lambda}(x)\omega(\lambda)\,d\lambda
$$
where $\omega$ is a non-negative density function that can be $0$ over a large part of $\mathbb{R}$, and where $w_{n}$ are normalization weights. This is the general form of the expansion on $[a,b)$ when the singular continuous spectrum is absent (which is the case in everything you'll probably ever see.) Here $\phi_{\lambda}$ are the classical eigenfunctions for each real $\lambda$, and are chosen to satisfy a common condition at $a$. Furthermore, this expansion turns the operator $L$ into multiplication by $\lambda$, which is really just a statement of a form of diagonalization by the generalized eigenfunction expansion. Using the last form, the diagonalization is
$$
Lf = \sum_{n}w_{n}\int_{a}^{b}f(t)\phi_{\lambda_{n}}(t)\,dt\,\lambda_{n}\phi_{\lambda_{n}}(x)
+ \int_{-\infty}^{\infty}\left(\int_{a}^{b}f(t)\phi_{\lambda}(t)\,dt\right)\lambda \phi_{\lambda}(x)\,\omega(\lambda)\,d\lambda
$$
And you get Parseval's equality:
$$
\int_{a}^{b}|f(t)|^{2}\,dt = \sum_{n}w_{n}\left|\int_{a}^{b}f(t)\phi_{\lambda_{n}}(t)\,dt\right|^{2}
+\int_{a}^{b}\left|\int_{a}^{b}f(t)\phi_{\lambda}(t)\,dt\right|^{2}\omega(\lambda)\,d\lambda.
$$
Note: If you form a sum and integral over any part of the spectrum, and another over a disjoint part, then the two functions are orthogonal in $L^{2}[a,b)$.
Continuous Spectrum: The continuous spectrum exists wherever $\omega(\lambda)$ is positive, and you can see the reason for the original use of the term continuous spectrum. You have an integral sum of eigenfunctions over a continuous range of eigenvalues. Later, the definition evolved in order to study this is a more abstract setting. The continuous spectrum in Quantum corresponds to unbound states. The discrete spectrum $\{ \lambda_{n} \}$ corresponds to actual bound states with perfectly well-defined energies. In these cases $\phi_{\lambda}$ is not in $L^{2}$ for $\lambda$ in the continuous spectrum, even though integrals over the smallest intervals with respect to $\omega(\lambda)d\lambda$ are in $L^{2}$. Wave packets that you encounter in traditional trigonometric analysis are there in the most general context of continuous spectrum. You don't have pure states in the continuous spectrum, but you have approximate pure states--as close as you want.
Self-adjointness is used to show that $\langle Ax,x\rangle$ is always real. Recall that $\langle u,v\rangle = \overline{\langle v,u\rangle} $ and use the definition of self-adjointness:
$$
\langle Ax,x\rangle=\langle x,Ax\rangle = \overline{\langle Ax,x\rangle}
$$
The boundedness of $A$ implies that the set $\{\langle Ax,x\rangle:\|x\|=1\}$ is bounded. So, it's a bounded subset of $\mathbb R$. Now $m$ and $M$ make sense and the proof goes as before.
By the way, there is a gap in is bounded below and is hence invertible: for example, the shift operator $(x_1,x_2,\dots)\mapsto (0,x_1,x_2,\dots)$ is bounded from below but is not invertible. This is another place where self-adjointness will be invoked: re-read the proof to see what goes on there.
If you don't need bounds for the spectrum, you could simplify the proof by dropping $M$ and $m$. Since the modulus of a complex number is at least the modulus of its imaginary part,
$$
\|(A- \lambda I)x\| \ge |\langle (A- \lambda I)x, x \rangle| =|\langle Ax, x \rangle - \lambda| \ge |\operatorname{Im} \lambda|
$$
Best Answer
Since $K$ is compact, $0\in\sigma(K)$ (because $K$ is not invertible). Because $K$ is compact, any nonzero element of its spectrum has to be an eigenvalue.
Now suppose that $Ku=\lambda u$ for some $\lambda\ne0$. This, with your specific $k$, looks like $$ \lambda u(x)=\int_0^x y\,u(y)\,dy-x\,\int_1^x u(y)\,dy. $$ Since $u$ is integrable (otherwise $Ku$ makes no sense), the right-hand-side is continuous; so $u$ is continuous. But then the right-hand-side is differentiable, so $u$ is differentiable. Note also that $u(0)=0$, again because the right-hand-side is $0$ at $x=0$.
Now, if we differentiate, $$ \lambda u'(x)=x\,u(x)-\int_1^xu(y)\,dy-x\,u(x)=-\int_1^xu(y)\,dy. $$ Reasoning as before, we deduce that $u'(1)=0$, and that $u'$ is differentiable. Taking derivatives again, $$ \lambda u''(x)=-u(x). $$ The case $\lambda=0$ gives $u=0$, so $0$ is not an eigenvalue (it belongs to the spectrum, though). When $\lambda\ne0$, this is an easy second order boundary value problem: $$ u''+\frac1\lambda\,u=0,\ \ u(0)=0, \ u'(1)=0. $$ The general solution is, if we write $r=1/\sqrt\lambda$, $$ u(x)=\alpha\cos rx+\beta\sin rx. $$ The initial conditions force $\alpha=0$, $\cos r=0$. So $r=\frac{2k+1}2\pi$, $k\in\mathbb Z$, that is $$ \frac1{\sqrt\lambda}=\frac{2k+1}2\pi, $$ so the eigenvalues are given by $$ \lambda_n=\frac4{(2n+1)^2\pi^2}, \ n\in\mathbb N\cup\{0\}, $$ with corresponding eigenfunctions $$ u_n(x)=\sin\frac{(2n+1)\pi}2\,x $$