Suppose first that $K$ is a finite extension of some $\mathbb Q_p$, with abs.
Galois gp. $G_K$.
A $p$-adic rep. of $G_K$ coming from geometry satisfies some basic conditions: it is pot. semi-stable, and the associated Weil--Deligne rep'n satisfies the Weil conjectures.
There are no other obvious conditions, and my memory (from a talk I saw many years ago, but perhaps it's written somewhere as well) is that Fontaine conjectured that these necessary conditions should be sufficient for an irred. $p$-adic rep. of $G_K$ to come from geometry. In the case when the rep'n looks like it should actually come from an abelian variety, I believe you can use Honda--Tate theory (and maybe some further related tools) to prove the conjecture, which gives some confidence in
the general case.
In the global case, one again has the obvious necessary conditions: finitely many ramified primes, pot. semi-stable locally at primes above $p$, and the Weil conjectures.
Again, there were no other obvious necessary conditions, and so, building on one's confidence in the local conjecture, it is natural to guess that they are also sufficient in the global context.
When Fontaine and Mazur were discussing this (the early 90's, I guess) Mazur pointed out that the condition on the Weil conjectures was not preserved under deformations, and so was an obstruction to ever proving such results. Mazur knew how to compute the expected dimensions of unrestricted local and global deformation rings, and Fontaine knew (at least in some case, such as the Fontaine--Laffaille case) how to compute the dimensions of local pot. semi-stable deformations rings.
If you imagine that the image of the global def. ring in the local rings meets the
pot. semi-stable locus transversely, then you find that the def. space of global reps. that are pot. semi-stable (of some given type and HT weights) is finite,
which fits with e.g. the Langlands reciprocity conjecture. (There are only finitely many Hecke eigenforms of fixed weight and level.)
These sorts of computations (I think there might be one in the original FM article) suggest that the conjecture might be true even without assuming the Galois rep.
satisfies the Weil conjectures, and give more confidence that it is true. To my mind, this deformation theoretic intuition gives pretty non-trivial motivation.
I'm not quite sure exactly how this origin story interacts with Wiles's proof.
I'm pretty sure that F and M made their conj. before Wiles's argument appeared,
and that deformation theory ideas were part of their motivation yoga. On the other
hand, clearly the whole conjecture became a lot more credible after Wiles's results.
As you note, much more is known about the conjecture now than was known when F and M first made their conjecture. This obviously adds to our confidence in it.
This is really a local question: you may as well assume $X$ is defined over a local field $L$, it doesn't matter whether it comes from a number field.
The point is that if $S = \operatorname{Spec} O_L$ and $\pi: \mathfrak{X} \to S$ is the (smooth proper) structure map, then for any locally constant torsion sheaf $\mathcal{F}$ on $\mathfrak{X}$ whose order is invertible on $S$, there is a locally constant sheaf $R^i\pi_\star\mathcal{F}$ on $S$ whose fibre at any geometric point $\bar{x}$ is $H^i(\mathfrak{X}_{\bar{x}}, \mathcal{F}_{\bar{x}})$; see Corollary VI.4.2 of Milne's "Lectures on Etale Cohomology" for the details. Since $S$ is connected, we get an isomorphism between the fibres at the closed point of $S$ and the generic point. That is,
$$H^i(X, \mathcal{F}) = H^i(X_0, \mathcal{F}_0)$$
where $X_0$ is the special fibre of $\mathfrak{X}$. This isomorphism is Galois-equivariant; but the inertia subgroup acts trivially on $X_0$. Hence $H^i(X, \mathcal{F})$ is unramified.
Best Answer
Ideas from $p$-adic Hodge Theory allow one to be more precise about which cohomology groups one expects to find the corresponding Galois representation. For a finite Galois representation, the representation will necessarily be de Rham with all Hodge--Tate weights zero. So one expects the Galois representation to occur inside $H^0$ of some smooth proper $X$. But $H^0(X,\mathbf{Q}_p)$ is nothing but the free group on the (geometric) components of $X$. Moreover, all of these are defined over a finite extension of $\mathbf{Q}$ and the Galois action on the cohomology group is just comes from the permutation representation on the components.
A very easy example to consider is the scheme $X: f(x) = 0$ for a separable polynomial $f(x) \in \mathbf{Q}[x]$ of degree $d$. The set $X(\mathbf{Q})$ is just the roots of $f(x)$, and the action of the Galois group $\mathrm{Gal}(\overline{\mathbf{Q}})/\mathbf{Q})$ on $X$ factors through the action of $G = \mathrm{Gal}(K/\mathbf{Q})$ where $K$ is the splitting field of $K$, and the representation is just the one arising from the natural permutation representation of $G$ on the roots. For example, if you start with a Galois extension $K/\mathbf{Q}$ of degree $|G|$, and you let $\theta \in K$ be a primitive element and $f(x)$ the minimal polynomial, then the corresponding representation of $G$ on $H^0(X/\overline{\mathbf{Q}},\mathbf{Q}_p) \simeq \mathbf{Q}^{|G|}_p$ is just the regular representation of $G$. Any finite representation $V$ of $G$is a summand of some number of copies of the regular representation, so any finite Galois representation $V$ of $G$ will occur inside the cohomology of $\coprod X$ for some number of copies of this $X$.
A small point: this realizes $V$ as inside some cohomology but not as the entire cohomology. You have to allow this. For example, $V$ could be the non-trivial $1$-dimensional representation of the Galois group of a quadratic extension. This can't be all of $H^0$ because $H^0$ always contains a $G$-invariant vector corresponding to the sum of all the components. But of course the Fontaine-Mazur conjecture only requires that $V$ is a subquotient rather than the entire cohomology.