In this case, colorings that differ by the action of an element of $D_4$ are considered the same. We also can use any combination of the 4 colors.
I am using Burnside's Lemma for this.
First, the set of possible colorings $X$ has $4^4$ elements since we can paint any edge any color.
The identity fixes every element of $X \therefore \mathrm{Fix}(e)=4^4$
$j_x$ the flip along the x axis, let us choose any colors for the two vertical edges, and the color of the horizontal edges has to match, therefore that is a total of $\mathrm{Fix}(j_x) = 4^3$ possible combinations of color.
Without loss of generality, we obtain the same for $j_y$, the flip along the y axis.
We have two more flips along the diagonals of the square. In this case, the two edges of one side of the diagonal have to match their respective edges of the other side, giving a total of $\mathrm{Fix}(j_{\pm\pi/4}) = 4^2$ possible combinations for each flip.
The rotation by $\pi/4$ needs every color to be the same in every edge, therefore we get $\mathrm{Fix}(r_{\pi/4}) = 4$ possible combinations. This also counts for $\mathrm{Fix}(r_{-\pi/4}) = 4$
Finally, for the rotation by $\pi/2$, the top and bottom edges swap places, and the left and right edges swap places as well, giving $\mathrm{Fix}(r_{\pi/2}) = 4^2$ since the top-bottom pair can have 4 possible colors and the left-right pair can have 4 possible colors.
Using Burnside's Lemma we get
$$\mathrm{\# Colorings} = \frac{1}{|D_4|} \sum_{g\in G} \mathrm{Fix}(g) = \frac{4^4+2\cdot4^3+3\cdot4^2+2\cdot4}{8} = 55$$
Best Answer
$(4):\binom{4}{1}=4$
$(3,1):\binom{4}{2}\binom{2}{1}=12$
$(2,2):\binom{4}{2}\cdot2=12$
Since there are two cases: $OOXX$ and $OXOX$
$(2,1,1):\binom{4}{3}\binom{3}{1}\cdot2=24$
Since there are two casee: $\color{blue}{OO}X\square$ and $\color{blue}{O}X\color{blue}{O}\square.$ Notice that $OO\square X$ can be achieved by a diagonal flip.
$$(1,1,1,1):\frac{4!}{4}\cdot\frac{1}{2}=3,$$
no matter it's a "main diagonal flip" or "counter diagonal flip" the result is that $ABCD$, which assumed I've already considered those clockwise rotations, becomes "counter clockwise" $DCBA$, so divide it by $2$.
Total = $4+12+12+24+3=52+3=55$
So you're correct! And I think group theory must be fun because I have to draw some squares and thinking about the repetitions but you have a complete formula for more colors, anyway it's cool!