In Quantum Mechanics, if one deals with wave functions, the Hilbert space in question is $L^2(\mathbb{R}^n)$ for a particle in $n$-dimensions, and the position operator corresponding to the $i$-th coordinate is $X^i$ defined by
$$(X^i \psi)(a) = x^i(a) \psi(a), \quad \forall a\in \mathbb{R}^n.$$
Now, if we generalize things slightly, we have a state space $\mathcal{H}$ which is a Hilbert space whose state vectors $|\psi\rangle \in \mathcal{H}$ describes the possible states of the system. In that case, the position operator is usually defined by
$$\langle x | X |\psi \rangle = x \langle x |\psi \rangle,$$
but here comes a problem, this definition depends on the particular basis $\{|x\rangle : x \in \mathbb{R}\}$. But how those kets are defined? Well, they are defined exactly as the eigenvectors of the position operator.
So, to define the position operator, we define it in a certain basis. But to define the vectors of that basis, we define then as the eigenvectors of the position operator.
This is circular. After all, the definition of the operator is being done in terms of vectors which are defined in terms of the operator.
Also, in this picture I cannot use the definition in terms of wave functions. And the reason is exactly because the wavefunction is then defined as the function $\psi(x)= \langle x| \psi\rangle$, which again depends on the kets $|x\rangle$.
There should probably be a correctly way to define the position operator and the position basis without being circular.
In that case, how does one correctly and rigorously define the positon operator and the position basis? I believe it all comes down to defining the operator, because once we prove it is hermitian we get a basis which from the postulates is naturally the position basis.
Best Answer
In QM the position operator and the Hilbert space of a particle are defined contextually: The Hilbert space is $L^2(\mathbb R^3, d^3x)$ and the operator position along $x_k$ is defined, in that space, as $(X_k\psi)(x):= x_k \psi(x)$ with the obvious domain.
You can adopt a more abstract viewpoint if you simultaneously define the momentum and the position operator satisfying CCR, exploiting the so-called Stone-von Neumann theorem. If the representation of the CCR is irreducible (and some technical requirements hold) then the Hilbert space is unitarily isomorphic to $L^2(\mathbb R^3, d^3x)$ and the position and momentum operators become the standard ones under the mentioned isomorphism.
Another more sophisticated approach relies upon the notion of (Mackey's) system of imprimitivity, where the momentum operator is the generator of spatial translations of the three position momenta.