Suppose I have a set of functions $g=g(x)$, that form a basis in the Hilbert Space. I can define states $|g\rangle$ associated with this basis.
Suppose I have some integral in the $x$ basis. I can normally use change of variables or $u$ substitution to convert this integral such that $g=g(x)$ becomes the dependent variable. However, instead of using $u$ substitution, I want to see this transformation directly in the Dirac notation.
For example, say we have an operator $\hat{A}$, such that $\hat{A}|x\rangle=A(x)|x\rangle$, in the $x$ basis. Moreover, we have $\hat{1}=\int dx |x\rangle\langle x|$.
We also know that $\langle x|x'\rangle=\delta(x-x')$.
Using this, we can easily convert from Dirac notation to an integral, when we want to find some expectation value. For example :
$$\langle\hat{A}\rangle = \frac{\langle\psi|\hat{A}|\psi\rangle}{\langle\psi|\psi\rangle}= \frac{\langle\psi|\hat{1}\hat{A}\hat{1}|\psi\rangle}{\langle\psi|\hat{1}\psi\rangle} = \frac{\int dx'\int dx\langle\psi|x\rangle\langle x|\hat{A}|x'\rangle\langle x'|\psi\rangle}{\int dx\langle\psi|x\rangle\langle x|\psi\rangle} = \frac{\int dx'\int dx \psi^*(x)A(x')\delta(x-x')\psi(x')}{\int dx \psi^*(x)\psi(x)}= \frac{\int dx \psi^*(x)A(x)\psi(x)}{\int dx\psi^*(x)\psi(x)}$$
This is how we calculate the expectation value as an integral.
However, let us now try to repeat the procedure for the $|g\rangle$ basis.
The first problem is $\hat{1} \ne \int dg |g\rangle\langle g|$ as these are not normalized. This is mentioned in this answer. So, I would rather write something like $\hat{1} = \int dg \space h(g) |g\rangle\langle g|$. Here $h(g)$ can be thought of some sort of weight or measure.
However, the second problem is I don't know what is $\langle g|g'\rangle = \space ?$
However, I do know that $\hat{A}|g\rangle = F(g)|g\rangle$. It is represented by a different function here in this new basis.
$$\langle\hat{A}\rangle = \frac{\langle\psi|\hat{A}|\psi\rangle}{\langle\psi|\psi\rangle}= \frac{\langle\psi|\hat{1}\hat{A}\hat{1}|\psi\rangle}{\langle\psi|\hat{1}\psi\rangle} = \frac{\int dg'\int dg \space h(g) h(g')\langle\psi|g\rangle\langle g|\hat{A}|g'\rangle\langle g'|\psi\rangle}{\int dg\space h(g)\langle\psi|g\rangle\langle g|\psi\rangle} = \frac{\int dg'\int dg\space h(g)h(g') \psi^*(g)F(g')\langle g|g'\rangle\psi(g')}{\int dg \space h(g)\psi^*(g)\psi(g)}=\space ?$$
I cannot proceed further as I don't know what $\langle g|g'\rangle$ is.
My intuition is, we remember $\hat{1} = \int dg\space h(g)|g\rangle\langle g|$. Hence we can normalize $|g\rangle$, by setting $|g\rangle \rightarrow \frac{|g\rangle}{\sqrt{h(g)}}$.
In that case, I may set $\langle g|g'\rangle = \frac{1}{h(g)}\delta(g-g')$. However, I don't know if this is correct in any way or not. I would also want to know how to prove this to be true, in case this is correct.
Moreover, I also notice that the function $h(g)$ which acts as some sort of a measure or weight factor in the resolution of identity on the new basis, is nothing more than the jacobian, that we would obtain from the $u$ substitution method. Hence, we can say, since $g=g(x)$, the jacobian is just $h(g)=\frac{1}{g'(x)}$.
In the numerator, there are two $h(g)$ terms, and as they represent the jacobian, I need one of them to cancel out. So I am inclined to believe that $\langle g|g'\rangle = \frac{1}{h(g)}\delta(g-g')$ is correct, as this would do just that.
However, I'm not sure if this is mathematically rigorous or even correct. If not, can someone show me the proper way of doing this? This question is a follow-up from the comments to the answer of this question.
How can I represent $u$ substitution directly using Dirac notation? I believe I am almost doing it correct, but then again, I might be horribly wrong and would appreciate any help.
I know that the final answer in the $g$ basis, should be equal to the following :
$$\space ?\space = \frac{\int dg \space h(g) \psi^*(g)F(g)\psi(g)}{\int dg \space h(g)\psi^*(g)\psi(g)}$$
Here $F(g)$ is the representation of operator $\hat{A}$ in the $g$ basis, and $h(g)$ is nothing but the Jacobian, that arises out of change of variables.
However, I want to know how to get to this form directly using Dirac notation, and not $u$ substitution, even thought the latter is much simpler.
Best Answer
There is an identity for delta functions of functions that basically answers your question here! Note: $$ \delta(f(x)) = \sum_i\frac{1}{|f'(x_i)|}\delta(x-x_i)\,, $$ where $x_i$ are the zeros of the function $f$. Consider $\delta(g(x)-g(x_0))$. Then the argument will be zero exactly when $g(x)=g(x_0)$. Assuming that $g$ is invertible, this only happens when $x=x_0$, and hence $$ \delta(g(x)-g(x_0)) = \frac{1}{|g'(x_0)|}\delta(x-x_0)\,. $$ We'll use this in the answer below, which does what you've done but in the Dirac notation as much as possible.
We start by just understanding how to transform the state from being represented in the $x$-representation to being represented in the $u$-representation.
Suppose we have a quantum state $\lvert \psi\rangle$, representable in the position eigenbasis as \begin{align*} \lvert\psi\rangle &= \int_{-\infty}^{\infty}dx\,\lvert x\rangle\langle x | \psi \rangle = \int_{-\infty}^{\infty}dx\,\psi(x)\lvert x\rangle\,. \end{align*} Consider an operator $\hat{u}=g(\hat{x})$, where $u=g(x)$ is (for simplicity) an invertible, increasing function on $\mathbb{R}$. The eigenbasis of $\hat{u}$ is the same as the eigenbasis for $\hat{x}$, since $$ \hat{u}\lvert x\rangle = g(\hat{x})\lvert x\rangle = g(x) \lvert x\rangle = u \lvert x\rangle\,. $$ However, despite the fact that the eigenbases are identical, we don't want to label the vector with $u$ directly, i.e., $\lvert x\rangle \neq \lvert u\rangle$; there is a difference in overall factor that comes in due to a Jacobian.
Let's take the pure Dirac expressions for this vector and effect the "re-labeling" $x=g^{-1}(u)$, i.e, \begin{align*} \lvert\psi\rangle &= \int_{-\infty}^{\infty}\frac{du}{g'(g^{-1}(u))}\lvert g^{-1}(u)\rangle\langle g^{-1}(u) | \psi \rangle\,. \end{align*} Note that, as numbers, $$ \langle g^{-1}(u) | \psi \rangle = \langle x | \psi \rangle\Longrightarrow \psi(g^{-1}(u)) = \psi(x)\, $$ where $\psi(x)$ is the particular functional form that represents $\lvert\psi\rangle$ in the position basis. Let's figure out how to move into the "u" representation.
The $u$ states really are the same as the $x$ states, because $\lvert g^{-1}(u)\rangle=\lvert x\rangle$, but suppose we were measuring $\hat{u}$ instead of $\hat{x}$. We want to have a "wave function" of a sort that is a function of $u$. To do this, let's make the following definition: $$ \lvert u\rangle = \frac{1}{\sqrt{g'(g^{-1}(u))}}\lvert g^{-1}(u)\rangle =\frac{1}{\sqrt{g'(x)}}\lvert x\rangle\,. $$ This state, although normalized in a different way, still represents the same state $\lvert x\rangle = \lvert g^{-1}(u)\rangle$, where $g(x)=u$. Under this definition, we have \begin{align*} \lvert\psi\rangle = \int_{-\infty}^{\infty}{du}\,\lvert u\rangle\langle u | \psi \rangle\,. \end{align*} Note that this implies that the $\lvert u\rangle$ states are normalized in the usual way, i.e., $$ \langle u\lvert u'\rangle=\delta(u-u')\,. $$ To verify that this is the case, we use a standard feature of the Dirac delta distribution: \begin{align} \delta(u-u') &= \delta(g(x)-g(x')) = \frac{1}{g'(x')}\delta(x-x') = \frac{1}{\sqrt{g'(x')}}\frac{1}{\sqrt{g'(x)}}\delta(x-x')\\ &= \frac{1}{\sqrt{g'(g^{-1}(u'))}}\frac{1}{\sqrt{g'(g^{-1}(u))}}\langle x \lvert x'\rangle\\ = &\frac{1}{\sqrt{g'(g^{-1}(u'))}}\frac{1}{\sqrt{g'(g^{-1}(u))}} \langle g^{-1}(u')\lvert g^{-1}(u)\rangle\\ &\left(\frac{1}{\sqrt{g'(g^{-1}(u'))}}\langle g^{-1}(u')\rvert\right) \left(\frac{1}{\sqrt{g'(g^{-1}(u))}}\lvert g^{-1}(u)\rangle\right)\\ &=\langle u' | u \rangle\,. \end{align}
Finally, let's consider expectation values. Let's take an operator $\hat{A}$ whose representation in the $x$-basis is $A(x)$ and whose matrix elements are $$ \langle x' \lvert \hat{A} \rvert x\rangle = A(x) \delta(x-x')\,. $$ Then, \begin{align*} \langle \psi \lvert \hat{A} \rvert \psi\rangle &= \left(\int_{-\infty}^{\infty}{dx'}\,\langle \psi | x \rangle\langle x\lvert\right) \hat{A} \left(\int_{-\infty}^{\infty}{dx}\,\lvert x\rangle\langle x | \psi \rangle\right) =\int_{-\infty}^{\infty}dx\,\langle \psi | x \rangle A(x) \langle x | \psi \rangle\, \end{align*} Performing the substitution $u=g(x)$, this becomes \begin{align*} \langle \psi \lvert \hat{A} \rvert \psi\rangle &= \int_{-\infty}^{\infty}\frac{du}{g'(g^{-1}(u))}\,\langle \psi | g^{-1}(u) \rangle A(g^{-1}(u)) \langle g^{-1}(u) | \psi \rangle\,, \end{align*} and after absorbing the square roots of the Jacobian into the states, this becomes \begin{align*} \langle \psi \lvert \hat{A} \rvert \psi\rangle &= \int_{-\infty}^{\infty}{du}\,\,\langle \psi | u \rangle A(g^{-1}(u)) \langle u | \psi \rangle = \int_{-\infty}^{\infty}{du}\,\,\langle \psi | u \rangle \tilde{A}(u) \langle u | \psi \rangle \,, \end{align*} By defining the $u$-space wave function as $\tilde{\psi}(u)=\langle u | \psi\rangle$, everything works out nicely.
Note that the primary difference between the OP and this calculation is that (1) I made sure to work with a normalized ket $\lvert \psi\rangle$ throughout, never representing it as a normalized or unnormalized wave function, and (2), made sure to carefully distinguish $\lvert u \rangle$ and $\lvert g^{-1}(u) \rangle$ where $u=g(x)$. This is a necessary step if we want to the $u$-representation to mirror the $x$-representation in all particulars.
The last thing we need to check is that the definition of $\tilde{A}(u)$ is consistent. To do this, start with the matrix elements in the $x$-basis and make some transformations, i.e., \begin{align*} \langle x' \lvert \hat{A} \rvert x\rangle = A(x) \delta(x-x') &= A(g^{-1}(u)) \delta (u-u')g'(g^{-1}(u))\, \end{align*} On the other hand, \begin{align*} \langle x' \lvert \hat{A} \rvert x\rangle &= \left(\sqrt{g'(g^{-1}(u'))}\langle u' |\right)\hat{A} \left(\sqrt{g'(g^{-1}(u))}\lvert u\rangle\right) \\ &= g'(g^{-1}(u'))\langle u' \lvert\hat{A} \rvert u\rangle\,, \end{align*} and by setting both sides equal to each other, we can see that \begin{align*} \tilde{A}(u)\delta(u-u') = A(g^{-1}(u))\delta(u-u')=\langle u' \lvert\hat{A} \rvert u\rangle\, \end{align*} exactly as it should.