We can almost factor the problem into the separate problems of determining the number $n_s$ of colourings on the square faces distinguishable under swapping and the number $n_r$ of colourings on the rectangular faces distinguishable under rotations about the prism axis. The only interaction between these two subproblems arises because if the square faces have the same colour, the colourings of the rectangular faces will behave differently depending on whether they are chiral, i.e. distinguishable from their mirror image. The product $n_sn_r$ counts a chiral colouring of the rectangles and its mirror image twice, even though they can be transformed into each other by swapping the square faces if those have the same colour, so we have to count the chiral and achiral colourings of the rectangles separately and compensate for the double-counting.
For $(4)$ identical colours on the rectangles, there is $1$ achiral pattern with $6$ colour choices.
For $(3,1)$ identical colours on the rectangles, there is $1$ achiral pattern with $6\cdot5$ colour choices.
For $(2,2)$ identical colours on the rectangles, there are $2$ achiral patterns with $\binom62$ colour choices each.
For $(2,1,1)$ identical colours on the rectangles, there is $1$ achiral pattern with $6\binom52$ colour choices and $1$ chiral pattern with $6\cdot5\cdot4$ colour choices.
For $(1,1,1,1)$ identical colours on the rectangles, there is $1$ chiral pattern with $6\cdot5\cdot4\cdot3/4$ colour choices.
Each of the achiral colourings of the rectangles can be combined with $\binom62+6$ colourings of the square faces, whereas for the chiral colourings of the rectangles the $6$ colourings of the square faces with identical colours are double-counted, so to compensate we should count only $\binom62+3$ colourings of the square faces per chiral colouring of the rectangles.
Putting it all together, we have $6+6\cdot5+2\binom62+6\binom52=126$ achiral colourings of the rectangles with $\binom62+6=21$ colourings of the square faces and $6\cdot5\cdot4+6\cdot5\cdot4\cdot3/4=210$ chiral colourings of the rectangles with $\binom62+3=18$ colourings of the square faces, for a total of $126\cdot21+210\cdot18=6426$ distinguishable colourings of the prism.
Now comes the weird part. The answer given in the book leads to $6246$, with just two digits swapped. But if you correct the error you spotted, you only get $6381$. The reason is yet another error -- the number for "swap ends, as above, rotate $90°$ or $270°$" that Gerry corrected from $6^4$ to $6^2$ should in fact be $6^3$, since this operation swaps two pairs of adjacent rectangles into each other. That makes the total come out right to $6426$.
P.S.: Oddly enough, your typo made the number in your original post come out right, since the overall result of the three errors was just that the numbers in the last two lines were swapped :-). What's also odd is that the "Instructor's Solutions Manual" for the 7th edition that I found online contains the incorrect solution you quote, whereas the Spanish edition that you can also find online, which seems to be from 1988 and thus older than the 7th English edition, doesn't give a detailed solution, but gives the correct number $6426$ in the solutions section.
Best Answer
Since we are painting the edges of squares, we assume that these four colorings are all considered the same, and should not be counted separately:
(It should be clear that painting triangular wedges is the same as painting edges, since there is a natural bijection between wedges and edges.)
It's not completely clear in the question, but we should probably also assume that the squares can be flipped over, so that these two colorings should also be considered the same:
That is the crucial question that decides whether to use $C_4$ or $D_4$: Are those two different colorings, or are they the same coloring? If they are the same coloring, then we say we are allowed to flip over the square, and flipping becomes part of the group we consider in applying the theorem. If they are different colorings, then reflections are not part of the group.
To apply the Cauchy-Frobenius-Burnside-Redfield-Pólya lemma, we begin by observing that our squares have the symmetry group $D_4$, which includes the four reflections. Then we count the number of colorings that are fixed points of a coloring under action by each element of $D_4$. Let $x$ be such an element, and suppose that $x$'s action on the square partitions the set of edges into $o(x)$ orbits. Then for a coloring to be fixed by $x$, all the edges in each orbit must be the same color. If there are $N$ different colors, then $N^{o(x)}$ are left fixed by the action of $x$.
The 8 elements of $D_4$ can be classified as follows:
The number of colorings with $N$ colors is the sum of the terms for each of these, divided by 8, the order of the symmetry group. Adding up the contributions from these we get $$\chi_{D_4}(N) = \frac18(N^4+2N^3+3N^2+2N).$$
This formula gives $\chi_{D_4}(0)=0, \chi_{D_4}(1)=1$ as we would hope. Hand enumeration of the $N=2$ case quickly gives 6 different colorings (four red edges; three red edges; two opposite red edges; two adjacent red edges; one red edge; no red edges) which agrees with the formula.
If reflections are not permitted, we delete the corresponding terms ($2N^3 + 2N^2$) from the enumeration, and divide by 4 instead of 8, as the symmetry group now has 4 elements instead of 8, obtaining instead $$\chi_{C_4}(N) = \frac14(N^4+N^2+2N).$$
This happens to have the same value as $\chi_{D_4}$ for $N<3$; this is because every coloring of the square with fewer than 3 colors has a reflection symmetry. The simplest coloring that has no reflection symmetry requires 3 colors:
You asked “I am to include rotations?” The answer is probably yes, or the question would not have specified the edges of a square, which has a natural rotational symmetry. But suppose you wanted to consider
to be different colorings. Then rotations of a coloring are not allowed, and the group you consider should be one that omits the rotation elements. In the extreme case, we can consider every coloring different, and then the group is the trivial group, and the same analysis says to omit all the terms except the $N^4$ contributed by the identity element, and we get $$\chi_{C_1}(N) = N^4$$ which is indeed the correct number of colorings.