$\newcommand{\id}{I}$
As it was mentioned in the comments, the domain where you defined the operator is not correct - If you take $C^1$-functions with derivatives in $L^2$ the domain will be "too small" in the sense that $A-\lambda I$ does not have closed range and therefore cannot be surjective for any $\lambda$ (The range would be the continuous functions in $L^2$, its easy to show that this is not closed under the $L^2$-norm.). This means that $A-\lambda I$ cannot be invertible for any $\lambda$, hence $\sigma(A)=\mathbb{C}$. In this case, the range of $A-\lambda I$ are the continuous $L^2$-functions which are known to be dense in $L^2$, hence $\sigma(A)=\sigma_c(A)$ since you already showed there cannot be any eigenfunctions.
Now considering the "correct" domain for this operator: Naturally, this will be given by the Sobolev space $H^1(\mathbb{R})$ Since we are in one dimension, this space can be characterized as a subspace of the absolutely continuous $L^2$-functions on $\mathbb{R}$, since for these functions you may define a derivative almost everywhere. The subspace you consider as a domain are then just the functions such that this a.e. derivative ("weak derivative") again is $L^2$-integrable.
I will now shift your operator by $-i$, since this operator is a bit easier to handle:
You may check or look up, that the operator $\hat{p}=-i\frac{d}{dx}=-iA$ is self-adjoint with this domain, i.e. $\hat{p}=\hat{p}^\dagger$ where the domain of $\hat{p}^\dagger$ is canonically defined. On a side note, this operator corresponds to the momentum in quantum mechanics.
You may also check, that for densely defined linear operators, we have
$$
R(A)^\perp=\operatorname{Kern}(A^\dagger).
$$Using this relation, you can in particular prove, that
- self-adjoint operators have a spectrum which is fully contained in the real line.
- self-adjoint operators have empty residual spectrum.
You can also look up this stuff in any basic textbook on functional analysis which contains some operator theory, e.g. Rudin or the first volume of Reed-Simon. I will now prove that $\sigma (\hat{p})=\mathbb{R}$, which implies that $\sigma(A)=i\mathbb{R}$.
Let $\lambda\in\mathbb{R}$, we show that $\hat{p}-\lambda I$ is not invertible. Consider a nonzero smooth function $f\in C_0^{\infty}(\mathbb{R})\subset D(\hat{p})$, i.e. $f$ has compact support, and define $g_k(x)=\frac{1}{\sqrt{k}}e^{i\lambda x}f(k^{-1}x)$.
Clearly, $g_k\in D(\hat{p})$ and you may check that $\|f\|=\|g_k\|$. Furthermore, you may calculate $\|(\hat{p}-\lambda\id) g_k\|=\frac{1}{k}\|f'\|$. Assume that $\hat{p}-\lambda\id$ has a bounded inverse, then there would hold
$$
\|f\|=\|g_k\|
=\|(\hat{p}-\lambda\id)^{-1}(\hat{p}-\lambda\id)g_k\|
$$ $$
\leq\|(\hat{p}-\lambda\id)^{-1}\| \|(\hat{p}-\lambda\id)g_k\|
= k^{-1}\|(\hat{p}-\lambda\id)^{-1}\| \|f'\|
$$
for any(!) $k\in\mathbb{N}$, which would imply $f$ being constant zero. Since $f$ was chosen not to be constant zero, this is a contradiction, and hence $\mathbb{R}\supset\sigma(\hat{p})$. Since $\hat{p}$ is self-adjoint, it cannot be any more, hence the claim follows.
Since the residual spectrum is empty and you already showed that the point spectrum is empty, this implies that the continuous spectrum of $A=\frac{d}{dx}$ is $i\mathbb{R}$ and all other spectra are empty.
You may of course ask yourself "how the f does one come up with such a function?", the reasoning is the following: For $\hat{p}$, $e^{i\lambda x}$ is an eigenfunction to $\lambda$, but it does not lie in $L^2$. However, it is not that bad of a function in the sense, that you can approximate it sufficiently well with functions from the domain of $\hat{p}$ - this is exactly what $g_k$ does, the function $f$ serves as a cutoff function. Diving further into the theory, you will find that $g_k$ is what we call a "Weyl sequence" for $\hat{p}$ at $\lambda$.
Best Answer
Hint: First make the substitution $u=\frac 1 t$ in $\int |x(t)|^{2}dt$. Write $\lambda$ as $\sin y$. Use the fact that $\sin u -\sin y=2\sin (\frac {u-y} 2) \cos (\frac {u+y} 2)$. In an interval around $y$ the function $\frac 1 {\sin (\frac {u-y} 2)}$ behaves like $\frac 2 {u-y}$ and the square of this is not integrable around $y$.