In the context of mathematical quantum mechanics, a well known no-go theorem known as Hellinger-Töplitz tells us that an unbounded, symmetric operator cannot be defined everywhere on the Hilbert space $\mathcal H$. Thus, for many operators of interest in quantum mechanics, we must resort to restricting the domain on which they may act. Along with the Hamiltonian, the momentum operator
$$ p:=-i\frac{\rm d}{\mathrm{d}x}$$
is the most prominent example of such an operator. Now, one of the most important questions is clearly: What is the correct domain of definition for the momentum operator?
In order to answer this question, it is of crucial importance to consider the question of self-adjointness: Only self-adjoint operators are admissible as observables of the theory, and there are certain vital mathematical results, such as the spectral theorem and Stone's theorem, which make it clear that any 'good' momentum operator must certainly be self-adjoint: $ p^\star= p$. So, the original question is refined slightly: What is the correct domain $\mathcal D( p)\subset \mathcal H$ that allows us to construct a self-adjoint momentum operator?
Recently, one of my lecturers treated a concrete incarnation of this (general and possibly vague) question: Consider a Hilbert space $\mathcal H=L^2(0,1)$ and two 'candidate momenta', $ p_0$ and $ p_\alpha$, with domains
$$\mathcal D(p_0)=\{\psi\in \mathcal H \,|\, \psi\in \text{AC}(0,1),\psi'\in\mathcal H, \psi(0)=0=\psi(1)\} $$
$$ \mathcal D(p_\alpha)=\{\psi\in \mathcal H \,|\, \psi\in \text{AC}(0,1),\psi'\in\mathcal H, \psi(0)=\alpha\psi(1)\},\hspace{.5cm}|\alpha|=1 $$
Are these operators self-adjoint? As it turns out, it can be shown that $p_0$ is symmetric but not self-adjoint. However, $p_\alpha$ is self-adjoint. It seems that the domain of $p_0$ is 'too small'.
Now, my question is: What does the conclusion that $p_0\neq p_0^\star$ imply for the canonical freshman QM example of the infinite potential well? As far as I'm aware, it is conventional to take exactly the boundary conditions that we assume to define the domain of $p_0$ when solving the Schrödinger equation in this case. Can we conclude that the notion of momentum cannot be rigorously defined in this elementary example? Or, at the very least, do we have to admit some $\psi$ that obey non-physical boundary conditions?
For those interested, this is a related question, also touching upon domain issues of momentum operators.
Best Answer
Although we can define the momentum as a self-adjoint operator in $L^2[0,1]$ as you proposed, I think it's rather artificial to think about it as having relation to momentum in the case of $L^2(\Bbb{R})$. Realize that the operator $p_1$ with domain $D(p_1)=\{\psi\in\mathcal{H}^1[0,1]\,|\,\psi(1)=\psi(0)\}$, is related to spatial translations via the unitary group $U(t)=\exp(-itp_1)$, whose action is $$(U(t)\psi)(x)=\psi[x-t\pmod{1}],$$ so it's about a particle in a torus, not in an infinite square-well. Different values of $\alpha$, just give different phases to the wavefunction when it reaches the border and goes to the other side.
So, in my opinion, these operators are not actually related to the momentum as usually conceived. The idea of an infinite square-well does not allow spatial translations, so there's no self adjoint operator associated to a unitary translation group in this case. This happens for example in the case of a particle in the postive real line $\Bbb{R}_+$. In this case, the space $L^2[0,\infty)$ allows only translations to the right, not to the left, so you can not have a self-adjoint operator associated to a unitary group of translations. In this case, the operator $p=-i\dfrac{d}{dx}$ has no self-adjoint extensions, for any initial domain, although it is symmetric. For a particle in a box, we can think in the same way. There's no operator associated to spatial translations, because there is no spatial translations allowed.
It's also important to note that the hamiltonian $H$ in this case is given by the Friedrich extension of $$p_0^2=-\frac{d^2}{dx^2}\\ \mathcal{D}(p_0^2)=\{\psi\in\mathcal{H}^2[0,1]\,|\,\psi(0)=\psi'(0)=0=\psi'(1)=\psi(1)\}$$ $H$ cannot be the square of any $p_\alpha$, since the domains do not match.
Edit: As pointed out by @jjcale, one way to take the momentum in this case should be $p=\sqrt{H}$, but clearly, the action of $p$ can't be a derivative, because it has the same eigenfunctions of $H$, which are of the form $\psi_k(x)=\sin \pi kx$. This ilustrates the fact that it's not related to spatial translations as stated above.
Edit 2: There's is a proof that the Friedrich extension is the one with Dirichilet boundary conditions in Simon's Vol. II, section X.3.
The domains defined by the spectral theorem are indeed $\{\psi: p_\alpha\psi\in\mathcal{D}(p_\alpha)\}$. To see this, realize that in this case, since the spectrum is purely point, by the spectral theorem, we have $$p_\alpha=\sum_{n\in \Bbb{Z}}\lambda_{\alpha,n}P_n,$$ where $\lambda_{\alpha,n}$ are the eigenvalues associated to the normalized eigenvectors $\psi_{\alpha,n}$, and $P_n=\psi_n\langle\psi_n,\cdot\rangle$ are the projections in each eigenspace. The domain $\mathcal{D}(p_\alpha)$ is then given by the vectors $\xi$, such that $$\sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^2\|P_n\xi\|^2=\sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^2|\langle\psi_n,\xi\rangle|^2<+\infty$$ Also, $\xi\in\mathcal{D}(p_\alpha^2)$ iff $$\sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^4\|P_n\xi\|^2=\sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^4|\langle\psi_n,\xi\rangle|^2<+\infty$$ But then, $p_\alpha\xi$ is such $$\sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^2\|P_np_\alpha\xi\|^2=\sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^2|\langle\psi_n,p_\alpha\xi\rangle|^2= \sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^2|\langle p_\alpha\psi_n,\xi\rangle|^2= \sum_{n\in \Bbb{Z}}|\lambda_{\alpha,n}|^4|\langle\psi_n,\xi\rangle|^2<+\infty$$ So, $\mathcal{D}(p_\alpha^2)= \{\psi: p_\alpha\psi\in\mathcal{D}(p_\alpha)\}$.