Roughly speaking, two actions of a group $G$ on set $X$ are isomorphic if you can relabel elements of $X$ such that one action will turn into the other. You can find a rigorous definition here:
http://en.wikipedia.org/wiki/Group_action#Morphisms_and_isomorphisms_between_G-sets
As for isomorphism classes, in your case an isomorphism class of actions of $G$ on $X$ is simply an equivalence class with respect to action isomorphism, i.e. a set of all actions that are isomorphic to a given one.
To show that $S_4 \cong V_4 \rtimes S_3$, first note that $V_4$ is isomorphic to the double transpositions in $S_4$, and this $V_4$ is normal in $S_4$.
Consider an isomorphic copy of $S_3$ in $S_4$ in the usual way.
Note that their intersections are trivial.
Denote the two subgroups as $H$ and $K$, then $HK$ is a subgroup of $S_4$ of size $\frac{|H||K|}{|H \cap K|} = 4 \cdot 6 = |S_4|$, so $HK$ is equal to $S_4$, meaning $S_4$ is a semidirect product of $V_4$ and $S_3$.
To show that $S_4 \cong V_4 \rtimes_\phi S_3$ for some isomorphism of $S_3 \to \text{Aut}(V_4)$ (rather than, more generally, some homomorphism), note that the kernel of $\phi$ must be a normal subgroup of $S_3$.
We make use of Jyrki's comment.
The only nontrivial normal subgroups of $S_3$ are $C_3$ and all of $S_3$, which includes $C_3$.
Note that $V_4$ is abelian.
If $C_3$ is in the kernel of $\phi$, then
$$\{ (h, k) \in V_4 \rtimes_\phi S_3 | h \in V_4, k \in C_3 \}$$
is an abelian subgroup of order 12.
However, $V_4$ has no abelian subgroup of order 12.
Therefore the kernel of $\phi$ must be trivial, i.e. it is an isomorphism of $S_3 \to \text{Aut}(V_4)$.
To show that $S_4$ is a semidirect product of $V_4$ and $S_3$ for any isomorphism of $S_3 \to \text{Aut}(V_4)$, see the answer to this question.
Best Answer
As I said, you need to find all subgroups of each group in question up to conjugacy. See this thread for reference. In my answer here, $\text{Id}$ denotes the trivial group, whereas $C_r$ denotes the cyclic group of order $r$ (well, I do not like to use $\mathbb{Z}_r$ to talk about the cyclic group). I also assume that $D_{n}$ is the dihedral group of order $2n$ (some people use this notation to mean the dihedral group of order $n$, which means something only when $n$ is even). As usual, $A_n$ is the alternating subgroup of the symmetric group $S_n$.
Subgroups of $S_4$ up to conjugacy are $\text{Id}$, two kinds of $C_2$, $C_3$, two kinds of $C_2\times C_2$, $C_4$, $S_3$, $A_4$, and $S_4$. Thus, the corresponding transitive $S_4$-sets are $S_4/\text{Id}\cong S_4$, $S_4/\langle (1\;2)\rangle$, $S_4/\langle (1\;2)(3\;4)\rangle$, $S_4/\langle(1\;2\;3)\rangle$, $S_4/\langle (1\;2),(3\;4)\rangle$, $S_4/\langle (1\;2)(3\;4),(1\;3)(2\;4)\rangle \cong S_3$, $S_4/\langle (1\;2\;3\;4)\rangle$, $S_4/\langle (1\;2),(1\;2\;3)\rangle=S_4/S_3$, $S_4/A_4\cong C_2$, and $S_4/S_4\cong \text{Id}$.
For $D_{10}$, the subgroups up to conjugacy are $\text{Id}$, three kinds of $C_2$, $D_2\cong C_2\times C_2$, $C_5$, $C_{10}$, two kinds of $D_{5}$, and $D_{10}$. Write $D_{10}:=\big\langle a,b\,\big|\,a^5=1\,,\,\,b^2=1\,,\text{ and }ba=a^{-1}b\big\rangle$. Thus, the corresponding transitive $D_{10}$-sets are $D_{10}/\text{Id}\cong D_{10}$, $D_{10}/\langle b\rangle$, $D_{10}/\langle ab\rangle$, $D_{10}/\langle a^5\rangle \cong D_5$, $D_{10}/\langle a^5,b\rangle$, $D_{10}/\langle a^2\rangle\cong D_2\cong C_2\times C_2$, $D_{10}/\langle a\rangle\cong D_1\cong C_2$, $D_{10}/\langle a^2,b\rangle$, $D_{10}/\langle a^2,ab\rangle$, and $D_{10}/D_{10}\cong \text{Id}$.
For $C_{10}:=\big\langle c\,\big|\,c^{10}=1\big\rangle$, it has only four subgroups: $\text{Id}$, $\langle c^5\rangle\cong C_2$, $\langle c^2\rangle\cong C_5$, and $C_{10}$. Thus, the corresponding transitive $C_{10}$-sets are simply $C_{10}/\text{Id}\cong C_{10}$, $C_{10}/\langle c^5\rangle\cong C_5$, $C_{10}/\langle c^2\rangle\cong C_2$, and $C_{10}/C_{10}\cong\text{Id}$.
Subgroups of $S_3\times C_4$ up to conjugacy are of the form $H\times N$, where $H$ is a subgroup of $S_3$ and $N$ is a subgroup of $C_4$, a diagonal version of $C_2$, and a diagonal version of $S_3$. The list of $H$ (again, up to conjugacy) is: $\text{Id}$, $\langle(1\;2)\rangle$, $\langle (1\;2\;3)\rangle\cong A_3\cong C_3$, and $S_3$. The list of $N$ is simply $\text{Id}$, $\langle g^2\rangle\cong C_2$, and $C_4$, where $C_4:=\big\langle g\,\big|\,g^4=1\big\rangle$, so that $C_4/\text{Id}\equiv C_4$, $C_4/\langle g^2\rangle\cong C_2$, and $C_4/C_4\cong \text{Id}$. Diagonal versions of $C_2$ and $S_3$ are simply the subgroups (up to conjugacy) $$\tilde{C}_2:=\Big\langle \big((1\;2),g^2\big)\Big\rangle\cong C_2\text{ and }\tilde{S}_3:=\Big\langle \big((1\;2),g^2\big),\big((1\;2\;3),1\big)\Big\rangle\cong S_3\,.$$ Hence, if $G:=S_3\times C_4$, then the transitive $G$-sets up to isomorphism are $S_3\times C_4=G$, $\big(S_3/\langle (1\;2)\rangle\big)\times C_4$, $\big(S_3/A_3\big)\times C_4\cong C_2\times C_4$, $(S_3/S_3)\times C_4\cong \text{Id}\times C_4\cong C_4$, $S_3\times C_2$, $\big(S_3/\langle (1\;2)\rangle\big)\times C_2$, $\big(S_3/A_3\big)\times C_2\cong C_2\times C_2$, $(S_3/S_3)\times C_2\cong \text{Id}\times C_2\cong C_2$, $S_3\times \text{Id}\cong S_3$, $\big(S_3/\langle (1\;2)\rangle\big)\times \text{Id}\cong \big(S_3/\langle (1\;2)\rangle\big)$, $\big(S_3/A_3\big)\times \text{Id}\cong C_2\times \text{Id}\cong C_2$, $(S_3/S_3)\times \text{Id}\cong \text{Id}\times \text{Id}\cong \text{Id}$, $G/\tilde{C}_2$, and $G/\tilde{S}_3\cong C_2$.