A potential of the form
$$
V(x) = \left\{
\begin{split}
0&\quad \operatorname{in}\ [-a, a] \\
+\infty&\quad \operatorname{elsewhere}
\end{split}
\qquad a>0,
\right.
$$
compels us to solve
$$
-{\hbar}^2\frac{{\mathrm{d}}^2}{{\mathrm{d}}x^2} \psi(x) = E\psi(x)
$$
on $\mathscr{H}\equiv L^2([-a, a])$, and then extend it (uniquely) on $L^2(\mathbb{R})$
by setting
$$
\bar\psi(x) = \left\{
\begin{split}
\psi(x)&\quad \operatorname{in}\ [-a, a] \\
0&\quad \operatorname{elsewhere}
\end{split}
\right.
$$
Now, $P^2=-\hbar^2\frac{{\mathrm{d}}^2}{{\mathrm{d}}x^2}$ is a linear unbounded operator.
Necessary and sufficient conditions for it to be self-adjoint are
- $\operatorname{dom}(P^2)$ dense
- $\langle \phi | P^2 \psi\rangle = \langle P^2\phi| \psi \rangle \quad \forall\, \phi, \psi \in \operatorname{dom}(P^2)$.
- $\operatorname{ker}(P^2\pm i\mathbb{I})\equiv\{0\}$
First and foremost, the TISE written above makes sense only for those $\psi(x)\in L^2([-a,a])$ that have $\frac{{\mathrm{d}}^2}{{\mathrm{d}}x^2}\psi(x)\in L^2([-a,a])$ too. If we call such set $A$, for sure $\operatorname{dom}(P^2)\subseteq A$.
This already restricts the number of candidates for point 1.
For what concerns point 2., integration by parts leads to
$$
\langle \phi | P^2 \psi\rangle = -{\hbar}^2\int_{-a}^a {\mathrm{d}}x\,\phi^*\,\frac{{\mathrm{d}}^2\psi}{{\mathrm{d}}x^2} = -\hbar^2 \left[\phi^*\frac{{\mathrm{d}}\psi}{{\mathrm{d}}x}\right]_{-a}^{a}+\hbar^2\left[\left(\frac{{\mathrm{d}}\phi}{{\mathrm{d}}x}\right)^*\psi\right]_{-a}^{+a}+\langle P^2\phi|\psi\rangle
$$
At this point, there are two possible choices.
Either $\psi(\pm a)=0 \ $ or $\ \displaystyle \frac{{\mathrm{d}}\psi}{{\mathrm{d}}x}\bigg\vert_{\pm a}=0$
In this case, the good choice is to take $\psi(\pm a)=0$, probably because it makes the extension $\bar\psi$ continuous (why is it needed, by the way? Couldn't a wave function be discontinuous?) There might be another reason, but I'm really not sure about that.
If we consider $P = -i\hbar\nabla$ instead,
$$
\langle \phi | P \psi\rangle = -i{\hbar}\int_{-a}^a {\mathrm{d}}x\,\phi^*\,\frac{{\mathrm{d}}\psi}{{\mathrm{d}}x} = -i\hbar \left[\phi^*\psi\right]_{-a}^{a}+\langle P\phi|\psi\rangle
$$
here we are forced to put $\psi(\pm a)=0$ if we want $P$ symmetric.
Anyway, I don't think that this boundary condition alone would make $\operatorname{dom}(P)=\operatorname{dom}(P^2)$, does it?
Best Answer
There are some issues with your analysis.
First of all, your 3 conditions are equivalent to selfadjointness (with a correction concerning 3 written below) only if the considered operator is closed.
Your $P^2=-d^2/dx^2$ defined on twice differentiable functions, even imposing boundary conditions, is not closed. So the statement is not correct.
However, for a densely defined symmetric operator $A$, the three conditions you wrote, with the last one replaced by $$ker(A^*\pm iI)=\{0\}$$ are equivalent the the fact that $A$ is essentially selfadjoint. This means that $A$ admits only one selfadjoint extension (this is the closure of $A$ itself which, in turn, coincides with the double adjoint of $A$).
The operator $P^2$ has an adjoint $(P^2)^*$ which depends on the domain of $P^2$ and which is not a differential operator as it is defined on spaces of (also) non-differentiable functions.
The correct formulation of condition 3 is difficult to check directly and there are other more indirect methods that guarantee selfadjointness of $P^2$.
A powerful method arises fron Nelson's criterion. As a matter of fact, if $P^2$ admits an orthonormal basis of eigenvectors in its dense domain where it is symmetric, then it is essentially selfadjoint.
There are three important domains of $C^2$ functions which are $L^2([-a,a])$ with their second derivatives, concerning three different boundary conditions. In those three cases $P^2$ turns out to be essentially selfadjoint.
Periodic boundary conditions. Here there is a Hilbert basis of eigenvectors
made of imaginary exponential functions.
Dirichlet boundary conditions (the wavefunction vanishes at the boundary). Here there is a Hilbert basis of eigenvectors made of sinus functions.
Neumann boudary conditions (the first derivative of the wavefunction vanishes at the boundary). Here there is a Hilbert basis of eigenvectors made of cosinus functions.
In those three cases $P^2$ is essentially selfadjoint and the unique selfadjoint extension, the physical observable, is an operator defined on a space of, in general, non differentiable functions (though continuous in the one dinensional case).
The three different selfadjoint operators obtained in that way restrict to $-d^2/dx^2$ on the $C^2$ functions in common in the three domains above.
It remains the question on which are the physically correct boundary conditions in the case of the infinite well.
The answer relies upon physical considerations. The infinite well is the limit case of a very steep well defined on the whole real line. It is clear that the probability density must vanish for $x\geq a$ and $x\leq -a$. Hence the physically correct boundary conditions for the infinite well are the Dirichlet ones.