As you suggest, $u=g(x)$, taken invertible, and
$$
[\hat u, \hat x]=[\hat A, \hat x]=[\hat u, \hat A]=0,\\
\hat u \equiv g(\hat x), ~~~\leadsto |u\rangle \propto |x\rangle ~~~\leadsto \hat u |x\rangle = g(\hat x)|x\rangle = g(x)|x\rangle .
$$
Likewise,
$$
\hat A |x\rangle= f(x) |x\rangle= f(g^{-1}(\hat u)) |x\rangle \leadsto \\
\hat A |u\rangle= f(g^{-1}( u)) |u\rangle .
$$
In your sibling question you seem to appreciate that $\psi(x)$ does not "evolve" unitarily to $\psi (u)=\psi(g(x))\equiv \tilde \psi(x)$, since
$|\psi\rangle$ and $|\tilde \psi\rangle$ have different normalizations--see example below.
Let's illustrate some of this with the trivial scaling case g(x)=ax suggested in the sibling question, so g'=a, so du=a dx.
$$
{\mathbb I}=\int \!\!dx~~ |x\rangle \langle x|= \int \!\!du~~ \frac{1}{a}|x\rangle \langle x| \\
=\int \!\!du~~ |u\rangle \langle u| ,
$$
That is, $|u\rangle=\frac{1}{\sqrt{a}}|x\rangle$.
You then have,
$$\psi(u)=\langle u|\psi\rangle =\frac{1}{\sqrt{a}}\langle x|\psi\rangle= \psi (x)/\sqrt{a} =\langle x|\tilde \psi\rangle=\tilde \psi(x)=\psi(ax),\\
|\tilde \psi\rangle= |\psi\rangle / \sqrt{a} ~~~\implies ~~~
\langle \tilde \psi | \tilde \psi \rangle = \langle \psi | \psi \rangle/a,
$$
as remarked at the beginning. For this trivial case (only), the two expectation values you are computing turn out to be equal, but this is not a general feature, of course!
You may take it from here,
$$|\tilde \psi\rangle= \frac{1}{\sqrt{g'(\hat x)}}|\psi\rangle, ~~ |u\rangle= {1\over \sqrt{g'}}|x\rangle\leadsto \\ \frac{\langle\psi|\hat{A}|\psi\rangle}{\langle\psi|\psi\rangle}
= \int\!\! du |\psi(u)|^2 f(g^{-1}(u)) , ~~\mathbf{ but}\\
\frac{\langle\tilde\psi|\hat{A}|\tilde\psi\rangle}{\langle\tilde \psi|\tilde \psi\rangle}
= {\int\!\! {du\over g'(g^{-1}(u))} |\psi(u)|^2 f(g^{-1}(u)) \over \int\!\! {du\over g'(g^{-1}(u))} |\psi(u)|^2 }~~,
$$
as stressed! Check their identity for simple scaling, as per above remark.
There is an identity for delta functions of functions that basically answers your question here! Note:
$$
\delta(f(x)) = \sum_i\frac{1}{|f'(x_i)|}\delta(x-x_i)\,,
$$
where $x_i$ are the zeros of the function $f$. Consider $\delta(g(x)-g(x_0))$. Then the argument will be zero exactly when $g(x)=g(x_0)$. Assuming that $g$ is invertible, this only happens when $x=x_0$, and hence
$$
\delta(g(x)-g(x_0)) = \frac{1}{|g'(x_0)|}\delta(x-x_0)\,.
$$
We'll use this in the answer below, which does what you've done but in the Dirac notation as much as possible.
We start by just understanding how to transform the state from being represented in the $x$-representation to being represented in the $u$-representation.
Suppose we have a quantum state $\lvert \psi\rangle$, representable in the position eigenbasis as
\begin{align*}
\lvert\psi\rangle &= \int_{-\infty}^{\infty}dx\,\lvert x\rangle\langle x | \psi \rangle
= \int_{-\infty}^{\infty}dx\,\psi(x)\lvert x\rangle\,.
\end{align*}
Consider an operator $\hat{u}=g(\hat{x})$, where $u=g(x)$ is (for simplicity) an invertible, increasing function on $\mathbb{R}$. The eigenbasis of $\hat{u}$ is the same as the eigenbasis for $\hat{x}$, since
$$
\hat{u}\lvert x\rangle = g(\hat{x})\lvert x\rangle = g(x) \lvert x\rangle = u \lvert x\rangle\,.
$$
However, despite the fact that the eigenbases are identical, we don't want to label the vector with $u$ directly, i.e., $\lvert x\rangle \neq \lvert u\rangle$; there is a difference in overall factor that comes in due to a Jacobian.
Let's take the pure Dirac expressions for this vector and effect the "re-labeling" $x=g^{-1}(u)$, i.e,
\begin{align*}
\lvert\psi\rangle &= \int_{-\infty}^{\infty}\frac{du}{g'(g^{-1}(u))}\lvert g^{-1}(u)\rangle\langle g^{-1}(u) | \psi \rangle\,.
\end{align*}
Note that, as numbers,
$$
\langle g^{-1}(u) | \psi \rangle = \langle x | \psi \rangle\Longrightarrow \psi(g^{-1}(u)) = \psi(x)\,
$$
where $\psi(x)$ is the particular functional form that represents $\lvert\psi\rangle$ in the position basis. Let's figure out how to move into the "u" representation.
The $u$ states really are the same as the $x$ states, because $\lvert g^{-1}(u)\rangle=\lvert x\rangle$, but suppose we were measuring $\hat{u}$ instead of $\hat{x}$. We want to have a "wave function" of a sort that is a function of $u$. To do this, let's make the following definition:
$$
\lvert u\rangle = \frac{1}{\sqrt{g'(g^{-1}(u))}}\lvert g^{-1}(u)\rangle
=\frac{1}{\sqrt{g'(x)}}\lvert x\rangle\,.
$$
This state, although normalized in a different way, still represents the same state $\lvert x\rangle = \lvert g^{-1}(u)\rangle$, where $g(x)=u$.
Under this definition, we have
\begin{align*}
\lvert\psi\rangle = \int_{-\infty}^{\infty}{du}\,\lvert u\rangle\langle u | \psi \rangle\,.
\end{align*}
Note that this implies that the $\lvert u\rangle$ states are normalized in the usual way, i.e.,
$$
\langle u\lvert u'\rangle=\delta(u-u')\,.
$$
To verify that this is the case, we use a standard feature of the Dirac delta distribution:
\begin{align}
\delta(u-u') &= \delta(g(x)-g(x'))
=
\frac{1}{g'(x')}\delta(x-x')
=
\frac{1}{\sqrt{g'(x')}}\frac{1}{\sqrt{g'(x)}}\delta(x-x')\\
&=
\frac{1}{\sqrt{g'(g^{-1}(u'))}}\frac{1}{\sqrt{g'(g^{-1}(u))}}\langle x \lvert x'\rangle\\
=
&\frac{1}{\sqrt{g'(g^{-1}(u'))}}\frac{1}{\sqrt{g'(g^{-1}(u))}}
\langle g^{-1}(u')\lvert g^{-1}(u)\rangle\\
&\left(\frac{1}{\sqrt{g'(g^{-1}(u'))}}\langle g^{-1}(u')\rvert\right)
\left(\frac{1}{\sqrt{g'(g^{-1}(u))}}\lvert g^{-1}(u)\rangle\right)\\
&=\langle u' | u \rangle\,.
\end{align}
Finally, let's consider expectation values. Let's take an operator $\hat{A}$ whose representation in the $x$-basis is $A(x)$ and whose matrix elements are
$$
\langle x' \lvert \hat{A} \rvert x\rangle = A(x) \delta(x-x')\,.
$$
Then,
\begin{align*}
\langle \psi \lvert \hat{A} \rvert \psi\rangle
&=
\left(\int_{-\infty}^{\infty}{dx'}\,\langle \psi | x \rangle\langle x\lvert\right)
\hat{A}
\left(\int_{-\infty}^{\infty}{dx}\,\lvert x\rangle\langle x | \psi \rangle\right)
=\int_{-\infty}^{\infty}dx\,\langle \psi | x \rangle A(x) \langle x | \psi \rangle\,
\end{align*}
Performing the substitution $u=g(x)$, this becomes
\begin{align*}
\langle \psi \lvert \hat{A} \rvert \psi\rangle
&=
\int_{-\infty}^{\infty}\frac{du}{g'(g^{-1}(u))}\,\langle \psi | g^{-1}(u) \rangle
A(g^{-1}(u))
\langle g^{-1}(u) | \psi \rangle\,,
\end{align*}
and after absorbing the square roots of the Jacobian into the states, this becomes
\begin{align*}
\langle \psi \lvert \hat{A} \rvert \psi\rangle
&=
\int_{-\infty}^{\infty}{du}\,\,\langle \psi | u \rangle
A(g^{-1}(u))
\langle u | \psi \rangle
=
\int_{-\infty}^{\infty}{du}\,\,\langle \psi | u \rangle
\tilde{A}(u)
\langle u | \psi \rangle
\,,
\end{align*}
By defining the $u$-space wave function as $\tilde{\psi}(u)=\langle u | \psi\rangle$, everything works out nicely.
Note that the primary difference between the OP and this calculation is that (1) I made sure to work with a normalized ket $\lvert \psi\rangle$ throughout, never representing it as a normalized or unnormalized wave function, and (2), made sure to carefully distinguish $\lvert u \rangle$ and $\lvert g^{-1}(u) \rangle$ where $u=g(x)$. This is a necessary step if we want to the $u$-representation to mirror the $x$-representation in all particulars.
The last thing we need to check is that the definition of $\tilde{A}(u)$ is consistent. To do this, start with the matrix elements in the $x$-basis and make some transformations, i.e.,
\begin{align*}
\langle x' \lvert \hat{A} \rvert x\rangle = A(x) \delta(x-x')
&=
A(g^{-1}(u)) \delta (u-u')g'(g^{-1}(u))\,
\end{align*}
On the other hand,
\begin{align*}
\langle x' \lvert \hat{A} \rvert x\rangle &=
\left(\sqrt{g'(g^{-1}(u'))}\langle u' |\right)\hat{A} \left(\sqrt{g'(g^{-1}(u))}\lvert u\rangle\right)
\\
&=
g'(g^{-1}(u'))\langle u' \lvert\hat{A} \rvert u\rangle\,,
\end{align*}
and by setting both sides equal to each other, we can see that
\begin{align*}
\tilde{A}(u)\delta(u-u') = A(g^{-1}(u))\delta(u-u')=\langle u' \lvert\hat{A} \rvert u\rangle\,
\end{align*}
exactly as it should.
Best Answer
the question is a question of regularization and normalization of the wave-functions and basis vectors, which for continuous sets is not immediatly clear.
Let's try to see what is needed for $1 = \int du |u\rangle \langle u|$ to happen $$ \int du |u\rangle \langle u| = \int dx dx' du |x\rangle \langle x | u \rangle \langle u | x' \rangle \langle x' | = \int dx dx' du u(x)u^*(x') |x\rangle \langle x'| \stackrel{?}{=} \int dx |x\rangle \langle x|$$ so we want $$\int du u(x) u^*(x') = \delta(x-x')$$ which already sets a normalization and orthogonality constraint on $u(x)$, not all $u$ will automatically maintain that!
Some such changes of basis will indeed only result in a constant factor, for example his is what happens if we will move to a momentum-basis like set of functions $u(x) = \exp(i u x)$ and then $\int du u(x) u^*(x') = 2\pi \delta(x-x')$. So we will just have to divide by $2\pi$ for every time we insert the "identity". However some such changes might be more problematic, so it really depends on your choice of $u(x)$.