For your first question, $S_X$ is defined as the group of bijections from $X$ to $X$. The group operation is function composition. If $X$ is finite, say with $n$ elements, then the groups $S_X$ and $S_n$ are obviously (noncanonically) isomorphic.
There is always at least one group homomorphism from $G$ to $S_X$, namely the one which sends everything in $G$ to the identity map in $S_X$. This is the "stupid group action" which doesn't do anything ($g.x = x$ for all $g \in G$ and $x \in X$).
There are typically many homomorphisms from $G$ to $S_X$. Therefore, there are typically many group actions of $G$ on $X$.
"Once we specify a group and a set, how do we find such homomorphisms?"
Good question. In many settings, the group action comes up naturally. It's not like people are taking random sets $X$ and $G$ and asking, "I wonder how many different actions of $G$ I can find on $X$." I mean maybe some people are doing this, but usually the group action is a convenient way to describe some existing phenomenon they are trying to study.
Example: Let $X = \{ z \in \mathbb C: \operatorname{Im}(z) > 0\}$ be the upper half plane, and let $G = \operatorname{SL}_2(\mathbb Z)$ be the group of integer matrices with determinant $\pm 1$. There is a natural group action of $G$ on $X$ by
$$\begin{pmatrix} a & b \\ c & d \end{pmatrix}.z = \frac{az+b}{cz+d}.$$
This came up (I believe) when people were working out problems like describing all holomorphic bijections from the Riemann sphere to itself, and it turns out that they all look like the above (where the matrix can be anything in $\operatorname{GL}_2(\mathbb R)$).
One example where people are taking certain groups $G$ and certain sets $X$, and asking what are the ways $G$ can act on $X$, is when $X$ is a vector space. The bijections from $X$ to itself coming from the elements of $G$ are required to also be linear transformations on $X$. Such group actions are called representations. Representation theory includes the problem of describing all representations of a given group on a given vector space, and this turns out to be an extremely difficult problem in general. Finding such group actions is no simple matter.
Best Answer
I do not know what kind of answer you are looking for but I can share my understanding.
Basically, homogeneous spaces are analytical versions of cosets.
If you have a group $G$ and a normal subgroup $N\leq G$, then you can also define the quotient space $G/N$ as the set of all cosets $\{gN : g\in G\}$. This space satisfies the following properties:
$G/N$ is a group.
There is a transitive action of $G$ on the space $G/N$ by $g.(hN) = ghN$.
The converse is also true. Suppose that $X$ is any group and there exists a surjective homomrphism $\varphi:G\rightarrow X$ (this defines an action of $G$ on $X$ by $g.x = \varphi(g)\cdot x$.). Then, there is an isomorphism $X\cong G/N$ where $N = \ker \varphi$.
If $N$ is no longer a normal subgroup, then $G/N$ need not be a group. In this case the following are equivalent: There exists a transitive action of $G$ on $X$ if and only if there is a bijection $X\cong G/N$ where $N=\{g\in G : gx_0=x_0\}$ is the stabilizer of some $x_0\in X$.
Usually, when we work with homogeneous spaces we also assume some topology on the groups. $G$ usually a Lie group and the lattice $N$ is often a discrete co-compact subgroup. Then the space $X=G/N$ is a compact manifold. Conversely, if a compact space $X$ admits a transitive (continuous) action of a Lie group, then it is isomorphic to the manifold $G/N$ where $N$ is the stabilizer of some element $x_0\in N$.
Point is, homogeneous spaces are cosets with analytic properties. The main reason they are so useful is because of the interplay between the algebraic and analytic aspects of this space. That is, on one hand it is a coset space while on the other hand it is a differentiable manifold.