I'll use $(\cdot, \cdot)$ to denote the dot product on $\mathbb{R}^n$. In coordinates, for $u, v \in \mathbb{R}^n$, $n = \dim(M)$,
$$\langle u, v \rangle_{x} = (u, G(x)v)$$
and
$$\langle u, v \rangle'_{x} = (u, G'(x)v).$$
Here $G_{ij}(x) = \langle e_i, e_j \rangle_x$ is the matrix representation of the metric tensor. Thus $\langle u, v \rangle'_{x} = \langle u, P_x v \rangle_x$ for all $u,v$ requires
$$(u, G'(x)v) = (u, G(x)P(x)v).$$
Thus $P(x) = G(x)^{-1}G'(x)$. Since both $G$ and $G'$ are smooth, $P$ is smooth.
For each $x \in M$, $P_x \in L(T_xM)$, and $P_x$ varies smoothly with $x$. Hence $P$ is a smooth map from $M$ to $\bigsqcup_{x \in M}L(T_xM) = \bigcup_{x \in M}(\{x\} \times L(T_xM))$
You ask about the smoothness of the map $f(x) = \|P_x\|_{x}$. In coordinates, if $A$ is a constant $n \times n$ matrix,
\begin{align}
\|A\|_{x}^2 &= \sup_{(u, G(x)u) = 1}\|A u\|_{x}^2 \\
&= \sup_{(u, G(x)u) = 1}(Au,G(x)Au).
\end{align}
Now let $v = G(x)^{1/2}u$ to get
\begin{align}
&= \sup_{(u, G(x)u) = 1}(Au,G(x)Au) \\
&= \sup_{(v, v) = 1}(AG(x)^{-1/2}v, G(x)AG(x)^{-1/2}v) \\
&= \sup_{(v, v) = 1}(G(x)^{1/2}AG(x)^{-1/2}v, G(x)^{1/2}AG(x)^{-1/2}v) \\
&= \|G(x)^{1/2}AG(x)^{-1/2}\|^2.
\end{align}
Hence
$$f(x) = \|G(x)^{1/2}P(x)G(x)^{-1/2}\|.$$
Now this brings us to analysis of smoothness of the usual operator norm on $L(\mathbb{R}^n)$. Using $A(t) = \text{diag}(1 + t, 1 - t)$, we have $\|A(t)\| = \max(1 + t, 1 - t) = 1 + |t|$, which shows that the operator norm is only continuous, even when restricted to positive definite matrices. Thus our function $f$ is continuous, but generally not differentiable.
Best Answer
On every tangent space you can indeed use the Riesz representation theorem to construct a one-form fulfilling the condition at that particular point. Using this, you can define a section of the cotangent bundle in a pointwise manner. What remains to be shown is that this section is actually smooth. For this you only have to understand the linear algebraic situation: the explicit formula for the value at a given point $p \in M$ only contains the values of the Riemannian metric and the given vector field $Y$ at $p$. Convince yourself of this!