This is a process that can feel very arbitrary, but using geometric principles, you should be able to develop an intuition about these problems.
Imagine the coordinate functions $v, u, w$ as scalar fields on the 3d space, assigning their respective coordinates to a given position. For all these coordinates, there are associated gradients: $\nabla v$ for $v$, and so on. These tell us the direction of greatest increase for each coordinate.
What we do then is use these gradient vectors as a basis for our space: a set of vectors $g^v, g^u, g^w$ such that $g^v = \nabla v$ and so on. The contravariant metric tensor just measures the dot products of these vectors, so we can have an idea of how to measure lengths with them.
For instance, take $u = y - a(x)$ as you gave us. Taking the gradient of
$u$, we get
$$g^u = \nabla u = (g^x \partial_x + g^y \partial_y + g^z \partial_z) u(x,y,z) = g^y -a'(x) g^x$$
where $g^x, g^y, g^z$ are a Cartesian basis (thus, they are orthonormal), so it's easy to take the dot product:
$$g^{uu} = g^u \cdot g^u = [g^y - a'(x) g^x] \cdot [g^y - a'(x) g^x] = 1 + [a'(x)]^2$$
as you found. So if we have two vectors expressed using this basis of gradients, we can find the overall dot product using the contravariant metric, rather than having to go back and figure out the relationships between those gradients all over again.
Let me give a simplified overview of the background related to the question. The entire story can be found in the celebrated paper M. Atiyah, R. Bott, V.K. Patodi, "On the heat equation and the index theorem", which is the canonical reference for the subject.
The Riemannian curvature $R_{abcd}$ and the Ricci tensor $R_{ab}$ mentioned in the question are constructed out of the metric in the sense that in a coordinate patch $(U, x^i)$ they are given by a universal formula involving the partial derivatives of the components of the metric.
Explicit expressions can be found, e.g. here.
Moreover, the formulas turn out to be polynomial in the partial derivatives of $g_{a b}$ of all orders $\ge 0$ and the inverse metric $g^{a b}$.
A formula like that gives components of a tensor if they transform under change of coordinates in the tensorial way.
These observations give rise to the following notion.
Let $(M,g)$ be a Riemannian manifold of dimension $n = \dim M$.
Definition 1. A metric invariant (also known as a Riemannian invariant, or an invariant of the Riemannian structure) is a section $P(g)$ of a tensor bundle over $M$, such that for any diffeomorphism $\phi \colon M \to M$ the naturality property holds:
$$
P(\phi^* g) = g^* P(g)
$$
and in any coordinate patch $(U,x^i)$ (that also gives the coordinate trivialization of the tensor bundle where $P(g)$ has values), the components of $P(g)$ are given by a universal polynomial expression in the list of formal variables $\{ g^{i j}, \partial_{k_1} \dots \partial_{k_s} g_{i j} \}$, $s \ge 0$.
Remark 1. If P has values in $\mathbb{R}$, we have a scalar valued invariant.
Remark 2. In a similar fashion we can give a definition of a metric invariant differential operator.
Examples. The metric tensor $g_{a b}$ and its inverse $g^{a b}$, the Riemann curvature tensor $R_{a b}{}^{c}{}_{d}$, the Ricci tensor $R_{a b}$ are tensor valued metric invariants. The scalar curvature $R = g^{a b} R_{a b}$ is a scalar valued tensor invariant. More scalar valued examples are mentioned in another question of the OP. The Levi-Civita connection $\nabla^g$ of the metric $g$ is a metric invariant differential operator. See e.g. Jack Lee, Riemannian Manifolds. An Introduction to Curvature.
As @Jack Lee has pointed in the comments, using the Levi-Civita connection and the Riemannian curvature one can construct many tensor-valued metric invariants, and taking the complete contractions we obtain lots of scalar valued metric invariants. This can be formalized as follows.
Definition 2. A curvature invariant is a linear combination of partial contractions of the iterated covariant derivatives (with respect to the Levi-Civita connection $\nabla$ associated to the metric $g$) of the Riemannian curvature.
More examples can be found here.
The question in the consideration can now be reformulated as follows.
Can all the metric invariants be obtained as curvature invariants?
The answer is known to be positive. This is a consequence of the First Fundamental Theorem of the classical invariant theory. The key geometric tool that is used to reduce this problem to a problem of the representation theory of the orthogonal group is the normal (or geodesic) coordinates. See the details in the aforementioned paper.
This also holds for the metric invariant differential operators.
Best Answer
1), 2): yes. On Manifolds you can introduce a metric, which makes the manifold a Riemmannian manifold (assuming the metric is positive definite). A metric, by definition, is a (0,2) tensor field which defines a scalar product on the tangent bundle, i.e. the metric in a point $p$ is a scalar product on the tangent space at $p$. This, in local coordinates, has a representation which is usually denoted $(g_{ij})$ and corresponds to what you called the metric tensor.
If the metric is positiv definite, this matrix representation is invertible and as you wrote, the inverse is usually denoted $(g^{ij})$. The raising and lowering of indices (making contravariant tensors covariant and vice versa) works the same way you wrote it down. This is nothing but the fact that on a Euclidean vector space $E$ there is a natural isomorphism between the vector space and it's dual, induced by the metric (i.e. if $v$ is a vector, $w\mapsto \langle v, w \rangle$ is a linear map on $E$ and each linear map arises that way).
As for 3), most books on Riemannian geometry should do the job. Which one suits you best depends on you. A very comprehensive description of these things is to be found in Spivaks treatise 'A comprehensive introduction to Differential Geometry' ;-)