Here I am going to assume that we are not looking for distinct configurations, that is, we count a rotation or reflection of the pentagon to be a distinct possibility.
You both have incorrect terms. Consider having only $3$ colors, ie $q = 3$. In this condition there must be a vertex which lies between two points which have the same color. There are $3 \times 2 = 6$ ways to color these three points, leaving $2$ ways to color the remaining two points for a total of $12$ possibilities. But $3^2 \times 2^2 \times 1 + 0 + (0) = 36$, which is too many.
Hint: If you are coloring a pentagon, there are three possible cases:
The pentagon can be colored with $k = 5$ distinct colors chosen from the $q$ possibilities.
The pentagon can be colored with $k = 4$ distinct colors and one repeat.
The pentagon can be colored with $k = 3$ distinct colors where two colors appear twice. (We also have that one color could appear three times, but then it would be impossible to color a pentagon without the same color being adjacent.)
It is easy to see that it is impossible to color the vertices of a pentagon with only 2 colors such that the same color is not adjacent anywhere.
In your question, your friend is accounting for the $q(q-1)(q-2)(q-3)(q-4)$ ways to color a pentagon using $5$ distinct colors. To find the final solution, we need to count how many ways we can color a pentagon in each of the given cases and then we sum all the possibilities together.
Now, for each of the three cases, $q$ choose $k$ to fix the k elements you are working with, then consider how many ways you can color a pentagon using those $k$ colors.
Any more assistance will be moving towards an outright solution so I believe I should stop here.
After doing this operation, the $13$ counters on white tiles will each move to one of the $12$ black tiles. By Pigeonhole, a black tile will be used at least twice. So there are no ways to do this procedure.
Best Answer
I tried to count the number of matrices obtained from a $2 \times n$ original matrix such that all the elements in the original matrix are placed in an adjacent position horizontally or vertically in the resulting matrix. I computed this number for $1 \le n \le 8$ and found it equal to the squared Fibonacci numbers $F_{n+1}^2$. This seems related to the number of domino tilings for a $2 \times n$ rectangle, which is $F_{n+1}$.
In the general case of a $m \times n$ rectangle with $T(m,n)$ domino tilings (see OEIS A099390), is the number of $m \times n$ matrices with the above described property $T(m,n)^2$?
The answer is yes. Consider a black and white checkerboard on the original matrix and two arbitrary domino tilings of it: each square is covered by one domino of the first tiling and one domino of the second tiling. Now move the black square of each domino in the first tiling to the white square of the same domino, and the white square of each domino in the second tiling to the black square of the same domino: each square will have an adjacent destination and no two different squares can go to the same destination.
So the count you are looking for is:
$$g(m,n) = T(m,n)^2=\prod_{j=1}^{\lceil\frac{m}{2}\rceil} \prod_{k=1}^{\lceil\frac{n}{2}\rceil} \left ( 4\cos^2 \frac{\pi j}{m + 1} + 4\cos^2 \frac{\pi k}{n + 1} \right )^2$$
which is valid also when $m$ and/or $n$ are odd.