Suppose I have a function $u=g(x)$, and I want to create a basis $|u\rangle$ out of this. If I know the form of some wavefunction $\psi$ in a known basis, say $|x\rangle$, how can I find the form of the wavefunction in this new basis ?
For example, suppose $u=x^2$.
We have $\psi(x)=\langle x|\psi\rangle=\sin(x)$, for example.
Basic substitution tells us that in the basis $u=x^2$, this wavefunction should have the form $\phi(u) = \langle u|\psi\rangle = \sin(\sqrt{u})$ instead.
Remember, in this problem, my wave function didn't evolve or change. It is the same, just represented in two different basis. In basic substitution, we have $u=g(x)$, and $\psi(x)=\psi(g^{-1}(u))=\phi(u)$
How do I realize this using Dirac Notation?
For example, $\langle u|\psi\rangle = \int dx \langle u|x\rangle\langle x|\psi\rangle = \int dx\langle u|x\rangle\sin(x)$.
However, I don't see how shall I get the $\phi(u)=\sin(\sqrt{u})$ form from here.
The transformation between the position and momentum basis is rather simple as it is related by a Fourier transform. However, it is difficult to visualize this simple transformation from a basis $x$ to a basis $g(x)$, using the Bra Ket notation.
Any help would be highly appreciated.
Best Answer
This is a long post. Beware! But I think it answers the OP's concerns. Later on, I'll try to post a tldr.
Preliminaries: promoting $u$ to an operator
I don't think of the transformation $u=g(x)$ as actually a change of basis, for the following reason.
One way to interpret the relationship $u=g(x)$ is as a relationship between the eigenvalues of two operators. That is, there is an operatorial relationship, given by $$ \hat{u} = g(\hat{x})\,, $$ and the eigenbasis of $\hat{u}$ is exactly the same as the eigenbasis for $\hat{x}$, because $$ \hat{u}\lvert x\rangle=g(\hat{x})\lvert x\rangle=g(x)\lvert x\rangle = u\lvert x\rangle\,. $$ In this case, the transformation from one basis to another is not really a transformation at all, it is really a re-labeling (which is basically the same as a re-ordering of the basis, which is a trivial kind of basis change). For clarity, let's consider the operator $\hat{u}=\hat{x}^3$. Then, we have $$ \hat{u}\lvert x\rangle = \hat{x}^3\lvert x\rangle = x^3\lvert x\rangle = u\lvert x\rangle\,. $$ The basis is singly degenerate because the function $x^3$ is injective, and so we can label these vectors with the eigenvalues of as \begin{align} \lvert u\rangle = \lvert x^3\rangle\,, \end{align} where $u=x^3$, uniquely. Alternatively, we can write this as \begin{align} \lvert x\rangle &= \lvert \sqrt[3]{u}\rangle\,. \end{align}
The ambiguity comes in because we are dealing with a continuous basis, so let's consider two separate cases.
$\lvert x\rangle$ is a discrete basis
For the moment, let's pretend that the basis is actually discrete, and let $x$ take on integer values between $-\infty$ and $\infty$. In that case, we can expand an arbitrary state $\lvert \psi\rangle$ in this basis. Starting with the $x$ basis, we have \begin{align} \lvert \psi \rangle &= \sum_{x=-\infty}^{\infty} \lvert x\rangle\langle x| \psi\rangle = \sum_{x=-\infty}^{\infty} \lvert \sqrt[3]{u}\rangle\langle \sqrt[3]{u}| \psi\rangle = \sum_{u} \lvert \sqrt[3]{u}\rangle\langle \sqrt[3]{u}| \psi\rangle\,. \end{align} Note that in these sums, $x$ takes on integer values, whereas $u$ takes on only cubed-integer values. Despite the re-labeling, nothing has really changed, because $\langle \sqrt[3]{u}| \psi\rangle = \langle x|\psi\rangle$. The expansion coefficients haven't changed; instead, we've just rearranged them. We could also choose to re-label the state $\langle \sqrt[3]{u}| \psi\rangle\to\langle u| \psi\rangle$, so that we are actually labeling the state with the eigenvalue of the operator $\hat{u}$, but to make the connection to the continuous case, I'll leave it as is.
$\lvert x\rangle$ is a continuous basis
Now, let's do the infinite case. Starting with the $x$ basis, we make the substitution $u=x^3$, yielding \begin{align} \lvert x \rangle &= \int_{-\infty}^{\infty}dx\, \lvert x\rangle\langle x| \psi\rangle = \int_{-\infty}^{\infty}\left(\frac{1}{3u^{2/3}}\,du\right) \lvert \sqrt[3]{u}\rangle\langle \sqrt[3]{u}| \psi\rangle\,. \end{align} Comparing to the above, we can see that $\langle \sqrt[3]{u}| \psi\rangle$ and $\langle x| \psi\rangle$ are equal to each other as numbers (ignore the interpretation of them as functions of $x$ or $u$ for the moment). We've done nothing but re-parameterize these expressions, so they are still equal.
We can see that the big difference is in the integration measure $dx\to\frac{1}{3u^{2/3}}\,du$. Part of the question is about where is the "proper" place for the factor of $\frac{1}{3u^{2/3}}$. There are a few ways to go about interpreting what we have. The first is to recognize that we have again just written the identity operator in a different way, i.e., $$ \hat{I} = \int_{-\infty}^{\infty}\left(\frac{1}{3u^{2/3}}\,du\right) \lvert \sqrt[3]{u}\rangle\langle \sqrt[3]{u}|\,, $$ since it clearly hits $\lvert \psi\rangle$ and makes $\lvert \psi\rangle$. We don't have to worry about the meaning of wave functions at all. Instead, we just recognize that this is another way to resolve the identity, and so it's just another way to re-write our state. Thus, $\frac{1}{3u^{2/3}}$ stays with the integration measure. The problem with this is that we also want to interpret what we have in a physical context, and we know that the interpretations here all about probabilities. Thus, we look at the second way to make sense of this.
A brief important digression on the discrete case
In the discrete case, there's no problem of interpretation, because there's no change in the "integration measure" in the discrete case, and so $$ |\langle \sqrt[3]{u}| \psi\rangle|^2= |\langle x|\psi\rangle|^2 $$ is just the probability that if we get the result $u=x^3$ upon measuring the observable $\hat{u}$, which is the same probability as getting $x$ if we measure the operator $\hat{x}$.
A second brief important digression on terminology
The symbol $\psi$ is reserved for the name of the state represented by the normalized ket $\vert\psi\rangle$. Every time I write a ket, it will mean a normalized ket in the abstract Hilbert space. This ensures that things remain as consistent as they can between the discrete and continuous cases. It also means that wherever $\frac{1}{3}\sqrt[3]{u}$ shows up, it will never show up attached to a lone ket.
Probability interpretation of the transformed continuous distribution
This is essentially answered in the linked answer by Cosmos Zachos, but I want to approach it in a way that keeps the $\frac{1}{3u^{2/3}}$ away from the kets. I don't think it belongs there, because kets "should" be normalized (but that might be a matter of taste). In any case, let's suppose our measurement of $\hat{u}$ corresponds to getting a value of $u$ in some interval $[a,b]$. How should we represent that in our current language? The idea is to construct the projection operator as a "piece" of the identify operator, i.e., define $$ \hat{P}_{u,[a,b]} = \int_{a}^{b}\left(\frac{1}{3u^{2/3}}\,du\right) \lvert \sqrt[3]{u}\rangle\langle \sqrt[3]{u}\lvert\,, $$ in which case the probability of getting a result in $[a,b]$ given that the state is $\lvert\psi\rangle$ is \begin{align} \langle \psi \lvert \hat{P}_{u,[a,b]} \lvert \psi\rangle &= \int_{a}^{b}\left(\frac{1}{3u^{2/3}}\,du\right) \langle \psi\lvert \sqrt[3]{u}\rangle\langle \sqrt[3]{u}\lvert \psi\rangle\\ &= \int_{a}^{b}\left(\frac{1}{3u^{2/3}}\,du\right) \left\lvert\langle \sqrt[3]{u}\lvert \psi\rangle\right\rvert^2\,. \end{align} Now here's the crux of the matter! Do we keep $\frac{1}{3u^{2/3}}$ with the integration measure, or do we attach it to the "wave function". Well, if we want a probability density function, we are forced to say that this probability density function is defined as \begin{align} p(u) = \frac{1}{3u^{2/3}}\left\lvert\langle \sqrt[3]{u}\lvert \psi\rangle\right\rvert^2\,. \end{align} This means that if want to define a wave function associated with the $u$ variable, we are forced to define it in the following way, as $$ \tilde{\psi}(u) = \frac{1}{\sqrt{3}\sqrt[3]{u}}\langle \sqrt[3]{u}\lvert \psi\rangle =\frac{1}{\sqrt{3}\sqrt[3]{u}}\psi\left(\sqrt[3]{u}\right)\, $$ where $$ \psi(x) = \langle x \lvert \psi\rangle\,. $$ Crucially for our understanding, here, we know that, as numbers, $ \psi(x)$, $\langle x \lvert \psi\rangle$, $\langle \sqrt[3]{u} \lvert \psi\rangle$, and $\psi\left(\sqrt[3]{u}\right)$ are all equal, exactly because $u=x^3$. In addition, I am using the symbol $\psi$ for both the wave functions $\psi(x)$ and $\tilde{\psi}(u)$, because they represent the same state $\lvert\psi\rangle$. However, from the identification above, they are clearly not the same function, in the sense that they have different functional forms, which is why one is called $\psi$ and the other $\tilde{\psi}$. (A similar thing is done for the position-space and momentum-space wave functions, which I also often denote as $\psi(x)$ and $\tilde{\psi}(p)$.)
Alternatively, we can decide to keep the $\frac{1}{3u^{2/3}}$ with the integration measure, and just say that $\tilde{\psi}(u) = \psi(\sqrt[3](u))$ is our wave function. However, the square of the wave function is no longer a probability density function, because we have a different integration measure. But that's fine: we just remember that we have to add in a factor of $\frac{1}{3u^{2/3}}$ when calculating probabilities (and, inner products).
Final note
What we've done here has some precedent. When we write a hydrogenic wave function $\psi_{nlm}(r,\theta,\phi)$, the square of this wave function is not a probability density function for $r$, $\theta$, and $\phi$! That's because there is an integration measure $r^2dr\sin\theta d\theta d\phi$, and so the actual probability density function is $$ p(r,\theta,\phi) = r^2\sin\theta |\psi_{nlm}(r,\theta,\phi)|^2\,. $$ In this case, everyone is fine with the way things are. We leave the extra functions attached to the integration measure, and the square of the wave function is no longer a probability density function, but that's fine! We know how to calculate things, and so we're good.