I'm reading Sakurai's Quantum Mechanics. One of the problem in the book asks to use the relation
$$
\langle{x}|p\rangle=\frac{1}{\sqrt{2\pi\hbar}}e^{\frac{ipx}{\hbar}}
$$
to evaluate $\langle{x}|[X,P]|\alpha\rangle=\langle{x}|XP|\alpha\rangle-\langle{x}|PX|\alpha\rangle$ in terms of $\psi_{\alpha}(x)=\langle{x}|\alpha\rangle$ without using the fact that in the $x$ representation, $P$ acts like $-i\hbar\frac{d}{dx}$.
I'm not sure how to proceed with this. Here is my attempt:
The eigenvalue equations for the position operator $X$ and the momentum operator $P$ are, respectively
$$
X|x'\rangle=x'|x'\rangle \text{ and } P|p'\rangle=p'|p'\rangle
$$
So, for example, let's evaluate $\langle{x}|PX|\alpha\rangle$:
$$
\begin{equation}
\begin{split}
\langle{x}|PX|\alpha\rangle & = \langle{x}|PX|\int_{-\infty}^{\infty}|x'\rangle\langle{x'}|\alpha\rangle dx' \\
& = \langle{x}|P|\int_{-\infty}^{\infty}x'|x'\rangle\psi_{\alpha}(x') dx' \\
& = \langle{x}|P|\int_{-\infty}^{\infty}x'\left(\int_{-\infty}^{\infty}|p'\rangle\langle{p'}|x'\rangle dp'\right)\psi_{\alpha}(x') dx' \\
& = \langle{x}|P|\int_{-\infty}^{\infty}x'\left(\int_{-\infty}^{\infty}|p'\rangle\frac{1}{\sqrt{2\pi\hbar}}e^{-\frac{ip'x'}{\hbar}} dp'\right)\psi_{\alpha}(x') dx' \\
& = \langle{x}|\int_{-\infty}^{\infty}x'\left(\int_{-\infty}^{\infty}p'|p'\rangle\frac{1}{\sqrt{2\pi\hbar}}e^{-\frac{ip'x'}{\hbar}} dp'\right)\psi_{\alpha}(x') dx' \\
& = \int_{-\infty}^{\infty}x'\left(\int_{-\infty}^{\infty}p'\langle{x}|p'\rangle\frac{1}{\sqrt{2\pi\hbar}}e^{-\frac{ip'x'}{\hbar}} dp'\right)\psi_{\alpha}(x') dx' \\
& = \int_{-\infty}^{\infty}x'\left(\int_{-\infty}^{\infty}p'\frac{1}{\sqrt{2\pi\hbar}}e^{\frac{ip'x}{\hbar}}\frac{1}{\sqrt{2\pi\hbar}}e^{-\frac{ip'x'}{\hbar}} dp'\right)\psi_{\alpha}(x') dx' \\
& = \frac{1}{{2\pi\hbar}}\int_{-\infty}^{\infty}x'\left(\int_{-\infty}^{\infty}p'e^{\frac{ip'(x-x')}{\hbar}} dp'\right)\psi_{\alpha}(x') dx' \\
\end{split}
\end{equation}
$$
but then I got stuck because the middle integral is not convergent. I sensed that I did something wrong as well.
Best Answer
Two main points are....
When $[X,P]=XP-PX$ is an well-defined operator in a Hilbert space, $H=L^2([a,b])$, space of square-integrable functions in $[a,b]$, the domain of definition of $[X,P]$ is a set of functions $|\alpha\rangle$ satisfying
$|\alpha\rangle$ is in the domain of operator $X$
$|\alpha\rangle$ is in the domain of operator $P$
$P|\alpha\rangle$ is in the domain of operator $X$
$X|\alpha\rangle$ is in the domain of operator $P$
However, the domain of definition of $XP$ is a set of functions $|\alpha\rangle$ satisfying
$|\alpha\rangle$ is in the domain of operator $P$
$X|\alpha\rangle$ is in the domain of operator $P$
In the similar manner you can expect the form of the domain of $PX$.
So if you want to assert that $\langle{x}|[X,P]|\alpha\rangle = \langle{x}|XP|\alpha\rangle-\langle{x}|PX|\alpha\rangle$, you should have additional condition, $|\alpha\rangle$ is a function in the domain of $[X,P]$. Try to prove $\langle{x}|[X,P]|\alpha\rangle = \langle{x}|XP|\alpha\rangle-\langle{x}|PX|\alpha\rangle$ using $|\alpha\rangle=|p\rangle$ and Hermitianity of $X$ and $P$. You may find a contradiction.
From the last segment, $\left(\int_{-\infty}^{\infty}p'e^{\frac{ip'(x-x')}{\hbar}} dp'\right) \\$ is the form of Fourier transform of $p'$ and can be described by (Dirac) functional-derivative, $\delta'(x-x')$.
$\int_{-\infty}^{\infty}p'e^{\frac{ip'(x-x')}{\hbar}} dp'=\int_{-\infty}^{\infty}-i\hbar \frac{d}{dx}e^{\frac{ip'(x-x')}{\hbar}}dp'=-i2\pi\hbar^2 \frac{d}{dx} \delta(x-x')$.