Actually we have
$$|X^r| = |X^{r^2}| = k$$
and so the number of orbits is $$|X/G| = \frac{k^3 + 3k^2 + 2k}{6}$$
which gives the right answer ($(8+12+4)/6=24/6=4$) for $k = 2$.
There seems to be a problem with your reasoning, as reflected in your language use ("$X^s$ fixes..."). It is not $X^g$ which fixes elements of $X$, but rather $g$; and $X^g$ is the set of all coloured triangles fixed by $g$. Incidentally, this is very different from $Gx$, the orbit of the element $x$; and from $G_x$, the set of group elements which fix (or "get stuck on") $x$.
We can count $|X^g|$ by realizing that being fixed by $g$ is a severe restriction on the form of the colouring.
For example, to count $|X^r|$, we ask ourselves: if $x = r.x$ i.e. if a triangle looks the same after a rotation, how must it be coloured? All the edges have to be the same! So they can be all one red, or all one blue, etc. Thus, $|X^r| = k$.
Another example: to count $|X^s|$, we consider (for concreteness sake) $s$ to be a flip through an altitude. If $s.x = x$ then the two sides meeting at the vertex (of the altitude) have to have the same colour; the remaining side can be any colour. This gives the exponent 2 on $k$.
Let me just observe that while rigorous notation is a must for any
serious mathematician it can sometimes block the path of the beginning
reader. What is happening here is very simple. We need the cycle index
of the permutations from $S_2\times S_3$ acting on the slots of the
matrix. While the present case can be computed by inspection I will
explain the general method.
We require the cycle index $Z(M_{2,3})$ of the pairs $(\sigma,\tau)$
of row and column permutations acting simultaneously on the slots of
the matrix ($6$ of them, and with $12$ permutations total). Start
from the two cycle indices for the row and column permutations
$$Z(S_2) = \frac{1}{2} a_1^2 + \frac{1}{2} a_2$$
and
$$Z(S_3) =\frac{1}{6} b_1^3 + \frac{1}{2} b_1 b_2 + \frac{1}{3} b_3.$$
The method here is as follows. We draw a diagram of the cycle types of
$\sigma$ and $\tau,$ perhaps one beneath the other. Now for the cycle
index of the cartesian product of $S_2$ and $S_3$ we must factor the
combined action of the two permutations on the slots into cycles. We
represent row-column pairs i.e. slots by marking say the row and the
column and connecting them with a edge, not to be confused with the
directed edges of the two cycles. This edge travels in parallel along
the two cycles it is on and returns to its initial position after
$\mathrm{lcm}(l_1, l_2)$ steps, where $l_{1,2}$ are the lengths of the
cycles. As the pair of cycles contributes $l_1\times l_2$ slot
identifiers we get for the contribition to the combined cycle index
the term
$$a_{\mathrm{lcm}(l_1, l_2)}^{l_1 l_2/\mathrm{lcm}(l_1, l_2)}.$$
We now do the computation. There are thus six possible combinations of
cycle types that combine to form $Z(M_{2,3})$. We process these in
turn, leaving the most difficult part for last.
First, combining $a_1^2$ and $b_1^3.$ This fixes all six slots for a
contribution of
$$\frac{1}{12} a_1^6.$$
Second, combining $a_1^2$ and $b_3.$ This partitions the pairs into
three-cycles for a contribution of
$$\frac{1}{6} a_3^2.$$
Third, combining $a_2$ and $b_1^3.$ Here we have everything on
two-cycles to get
$$\frac{1}{12} a_2^3.$$
Fourth, combining $a_2$ and $b_3$. This will produce a six-cycle to get
$$\frac{1}{6} a_6.$$
Now for the tricky part. Fifth, combining $a_1^2$ and $b_1 b_2.$ There
are two pairs that are fixed by these, and two pairs on two-cycles
and we obtain
$$\frac{1}{4} a_1^2 a_2^2.$$
Finally, sixth, combining $a_2$ and $b_1 b_2.$ We have everything on
two-cycles and obtain
$$\frac{1}{4} a_2^3.$$
Adding everything we now have the cycle index
$$Z(M_{2,3})
= \frac{1}{12} a_1^6
+ \frac{1}{3} a_2^3
+ \frac{1}{6} a_3^2
+ \frac{1}{4} a_1^2 a_2^2
+ \frac{1}{6} a_6.$$
In order to apply Burnside and ask about colorings with at most $N$
colors we have that the assignment of the colors must be constant on
each cycle and we obtain
$$\frac{1}{12} N^6
+ \frac{1}{3} N^3
+ \frac{1}{6} N^2
+ \frac{1}{4} N^4
+ \frac{1}{6} N.$$
This yields the sequence
$$M_n = 1, 13, 92, 430, 1505, 4291, 10528, 23052, \ldots$$
which is OEIS A027670. In particular we
find there are $92$ colorings using at most three distinct colors. We
could apply PET at this point since we have the cycle index. E.g. for
three colors we obtain
$$1/12\, \left( R+G+B \right) ^{6}
\\ +1/4\, \left( R+G+B \right) ^{2} \left( {B}^{2}+{G}^{2}+{R
}^{2} \right) ^{2}+1/6\, \left( {B}^{3}+{G}^{3}+{R}^{3} \right) ^{2}
\\ +1/3\, \left( {B}^{2}+
{G}^{2}+{R}^{2} \right) ^{3}+1/6\,{B}^{6}
+1/6\,{G}^{6}+1/6\,{R}^{6}$$
which expands to
$${B}^{6}+{B}^{5}G+{B}^{5}R+3\,{B}^{4}{G}^{2}+3\,{B}^{4}GR
\\ +3\,{B}^{4}{R}^{2}+3\,{B}^{3}{G}^{3}+6\,{B}^{3}{G}^{2}R
\\ +6\,{B}^{3}G{R}^{2}+3\,{B}^{3}{R}^{3}+3\,{B}^{2}{G}^{4}
\\ +6\,{B}^{2}{G}^{3}R+11\,{B}^{2}{G}^{2}{R}^{2}+6\,{B}^{2}G{R}^{3}
\\ +3\,{B}^{2}{R}^{4}+B{G}^{5}+3\,B{G}^{4}R+6\,B{G}^{3}{R}^{2}
\\ +6\,B{G}^{2}{R}^{3}+3\,BG{R}^{4}+B{R}^{5}+{G}^{6}+{G}^{5}R
\\ +3\,{G}^{4}{R}^{2}+3\,{G}^{3}{R}^{3}+3\,{G}^{2}{R}^{4}
\\+G{R}^{5}+{R}^{6}.$$
The reader might want to verify some of these with pen and paper.
We may also apply inclusion-exclusion to obtain the count of colorings
of the matrix with exactly $N$ colors. The nodes of the poset
correspond to sets $P\subseteq [N]$ of colors which include colorings
that use some subset of these colors, with the top node representing
at most $N$ colors, which is the only $P$ that includes colorings with
exactly $N$ colors. Colorings with exactly $p \lt N$ colors where
$p\ge 1$ are included at all nodes that are supersets of the $p$
colors. We thus obtain for the total weight
$$\sum_{q=p}^N {N-p\choose q-p} (-1)^{N-q}
= (-1)^{N-p} \sum_{q=0}^{N-p} {N-p\choose q} (-1)^q = 0$$
since $N-p\ge 1.$ Colorings with less than $N$ colors have total
weight of zero in the poset. We thus obtain
$$\sum_{q=1}^N {N\choose q} (-1)^{N-q} M_q.$$
We get a finite sequence since with six slots it is not possible to
have a coloring with more than six distinct colors:
$$1, 11, 56, 136, 150, 60$$
The last term represents colorings with exactly six colors. This means
all slots in the matrix are distinct. Therefore all orbits have the
same size, the number of permutations in the group, which is twelve,
and indeed $6!/12 = 60.$ The term for two colors indicates that the
two monochrome colorings have been excluded.
This MSE link
has the Maple code for the general case.
Addendum. Here is what we mean when we say in the introduction
that the cycle index can be computed by inspection. This refers to the
isomorphism between $M_{2,3} = S_2\times S_3$ and $D_6$, the dihedral
group (reflections and rotations of regular polygons) acting on six
slots in this case. The cycle indices $Z(D_p)$ are tabulated and have
simple closed forms, consult
e.g. Wikipedia. Label the
vertices of a hexagon in clockwise order with the labels $(0,0),
(1,1), (0,2), (1,0), (0,1)$ and $(1,2).$ Then it is not difficult to
see that the rotations of the hexagon are in a bijection with the
pairs of cycles from $C_2 \times C_3$ embedded in $M_{2,3}.$ E.g. the
rotation that takes the top vertex to its clockwise neighbor
corresponds to the two cycles $(0,1)$ and $(0,1,2)$. The reflections
in an axis passing through opposite vertices preserve parity
(permutation $(0)(1)$ from $S_2$) and fix one element from $0,1,2$
while permuting the other two in two-cycles. The reflections in an
axis passing through opposite edges flip parity (permutation $(0,1)$
from $S_2$) and fix one of three elements from $0,1,2$ while
exchanging the other two. In this way we have bjectively accounted for
all permutations and the proof of the isomorphism is complete.
Best Answer
What we have here is an instance of Power Group Enumeration as described by Harary and Palmer, Graphical Enumeration. The algorithm is documented at the following MSE-link I. We require the cycle index $Z(Q_n)$ of the action on the edges of the permutations of the two parts of the graph, possibly combined with a horizontal reflection. This is the slot permutation group. We distribute edges of one of $k$ colors into these slots, and the group acting on them is the symmetric group with cycle index $Z(S_k)$. The cycle index $Z(Q_n)$ was computed at the following MSE-link II. We have e.g.
$$Z(Q_3) = {\frac {{a_{{1}}}^{9}}{72}} +1/6\,{a_{{1}}}^{3}{a_{{2}}}^{3} +1/8\,a_{{1}}{a_{{2}}}^{4}+1/4\,a_{{1}}{a_{{4}}}^{2} +1/9\,{a_{{3}}}^{3}+1/3\,a_{{3}}a_{{6}}.$$
and
$$Z(Q_4) = {\frac {{a_{{1}}}^{16}}{1152}} +{\frac {{a_{{1}}}^{8}{a_{{2}}}^{4}}{96}} +{\frac {5\,{a_{{1}}}^{4}{a_{{2}}}^{6}}{96}} +{\frac {{a_{{1}}}^{4}{a_{{3}}}^{4}}{72}} +{\frac {17\,{a_{{2}}}^{8}}{384}} \\ +1/12\,{a_{{1}}}^{2}a_{{2}}{a_{{3}}}^{2}a_{{6}} +1/8\,{a_{{1}}}^{2}a_{{2}}{a_{{4}}}^{3} +1/18\,a_{{1}}{a_{{3}}}^{5} +1/6\,a_{{1}}a_{{3}}{a_{{6}}}^{2} \\ +1/24\,{a_{{2}}}^{2}{a_{{6}}}^{2} +{\frac {19\,{a_{{4}}}^{4}}{96}} +1/12\,a_{{4}}a_{{12}}+1/8\,{a_{{8}}}^{2}.$$
With these ingredients we are ready to run the PGE algorihm. We get for two swappable types of edges the sequence
$$1, 4, 13, 104, 1507, 64203, 8426875, 3671999389, 5366787092478, \\ 26433809041087192, 441089058039611200394, 25113998661290096278734134, \ldots$$
and for three types
$$1, 6, 84, 7946, 5413511, 25231086540, 800871112032930, \\ 177544715836044855636, 281653040526999655665449719, \\ 3266495639384107667257990172349726, \\ 282129919925994006382238965837655927175534, \\ 184379837924757642947198903200667422197524750679153, \ldots $$
The Maple code for this is quite compact and shown below.
with(combinat); pet_cycleind_symm := proc(n) local l; option remember; if n=0 then return 1; fi; expand(1/n*add(a[l]*pet_cycleind_symm(n-l), l=1..n)); end; pet_cycleind_knn := proc(n) option remember; local cindA, cindB, sind, t1, t2, term, res, cmb, len, l1, l2, cycs, uidx, vidx, u, v, inst1; if n=1 then sind := [a[1]]; else sind := pet_cycleind_symm(n); fi; cindA := 0; for t1 in sind do for t2 in sind do res := 1; for u in indets(t1) do l1 := op(1, u); for v in indets(t2) do l2 := op(1, v); len := lcm(l1, l2); res := res * a[len]^(degree(t1, u)*degree(t2, v) *l1*l2/len); od; od; cindA := cindA + lcoeff(t1)*lcoeff(t2)*res; od; od; cindB := 0; for term in sind do res := 1; # edges on different cycles of different sizes for cmb in choose(indets(term), 2) do u := op(1, cmb); v := op(2, cmb); l1 := 2*op(1, u); l2 := 2*op(1, v); res := res * a[lcm(l1, l2)]^((l1*l2/2/lcm(l1, l2))* degree(term, u)*degree(term, v)); od; # edges on different cycles of the same size for u in indets(term) do l1 := 2*op(1, u); inst1 := degree(term, u); # a[l1]^(1/2*inst1*(inst1-1)*l1*l1/2/l1) res := res * a[l1]^(1/2*inst1*(inst1-1)*l1/2); od; # edges on identical cycles of some size for u in indets(term) do l1 := 2*op(1, u); inst1 := degree(term, u); if type(l1/2, even) then # a[l1]^((l1/2)^2/l1); res := res * (a[l1]^(l1/4))^inst1; else # a[l1/2]^(l1/2/(l1/2))*a[l1]^(((l1/2)^2-l1/2)/l1) res := res * (a[l1/2]*a[l1]^(l1/4-1/2))^inst1; fi; od; cindB := cindB + lcoeff(term)*res; od; (cindA+cindB)/2; end; knn_swap_edge_cols := proc(n,k) option remember; local idx_slots, idx_cols, res, term_a, term_b, v_a, v_b, inst_a, inst_b, len_a, len_b, p, q; if n = 1 then idx_slots := [a[1]]; else idx_slots := pet_cycleind_knn(n); fi; if k = 1 then idx_cols := [a[1]]; else idx_cols := pet_cycleind_symm(k); fi; res := 0; for term_a in idx_slots do for term_b in idx_cols do p := 1; for v_a in indets(term_a) do len_a := op(1, v_a); inst_a := degree(term_a, v_a); q := 0; for v_b in indets(term_b) do len_b := op(1, v_b); inst_b := degree(term_b, v_b); if len_a mod len_b = 0 then q := q + len_b*inst_b; fi; od; p := p*q^inst_a; od; res := res + lcoeff(term_a)*lcoeff(term_b)*p; od; od; res; end;