You are correct, you want to use Burnside's lemma:
$$\mbox{# of Orbits}=\frac{1}{|G|}\sum_{g\in G}|fix(g)|.$$
You have an action of $G=D_{2m}$ on the set $X$ of $n$-colored necklaces with $m$ beads, and two such necklaces are the same if they are in the same orbit under this action. Therefore, counting the number of orbits counts the number of necklaces.
Let's set up some notation. We can regard a necklace as a regular $n$-gon with vertices numbered $\{1,\ldots,m\}$. Then, $G$ act on the vertices in the natural way. This corresponds to an embedding $G\hookrightarrow S_m$. Identifying an element $g\in G$ with its image in $S_m$, let $\psi(g)$ denote the number of disjoint cycles in the cycle decomposition of $g$.
Let $C$ be a set of $n$ colors (i.e. $|C|=n$). Then $G$ acts on $X=C^m$ by $$g.(c_1,\ldots,c_m)=(c_{g.1},\ldots,c_{g.m}).$$
Lemma: For the action of $G$ on $X$, we have $|fix(g)|=n^{\psi(g)}$.
proof: Since, for all $k$, $g^k.(c_1,\ldots,c_m)=(c_{g^k.1},\ldots,c_{g^k.m})$ we see that $(c_1,\ldots,c_m)$ is in $fix(g)$ if, and only if $c_i=c_{g^k.i}$ for all $i$ and $k$. This just says that $(c_1,\ldots,c_m)\in fix(g)$ if, and only if the coloring is constant on the cycles of $g$. There are $n$ choices to color each cycle, leading to $n^{\psi(g)}$ total colorings that are fixed by $g$.
Example: Suppose $m=4$ and $n=3$. The element of $D_8$ and their cycles types are chosen as follows:
$$
\begin{array}{llll}
1=(1)(2)(3)(4)&r=(1234)
&r^2=(13)(24)&r^3=(4321)\\
j=(24)(1)(3)&rj=(12)(34)&r^2j=(13)(2)(4)&r^3j=(14)(23)
\end{array}
$$
The corresponding orders of $fix(g)$ are given by
$$
\begin{array}{llll}
n^4&n&n^2&n\\
n^3&n^2&n^3&n^2
\end{array}
$$
Therefore, the number of necklaces with $n$ colors is
$$\frac{1}{8}(n^4+2n^3+3n^2+2n).$$
Substituting $n=3$ yields $21$ distinct necklaces.
I would compute the answer as
$$2(3+3)=12$$
Here's the idea: First place the Red, Green, and Yellow beads on the circle; there are $2$ ways to do this. Then place the two Blue beads either together between two beads already on the circle, or apart; in either case there are $3$ ways to place the Blue beads.
If you consider mirror images as identical, then there is only $1$ way to place the Red, Green, and Yellow beads in the first step, which gives the answer $3+3=6$.
Best Answer
The Polya Enumeration Theorem is a natural tool for problems like this and verifies that the answer is $19$.
The group of permutations is $D_8$, the dihedral group on $8$ vertices. The cycle index of $D_8$ is $$Z = \frac{1}{16} (x_1^8 + 4 x_1^2 x_2^3 + 5 x_2^4 + 2 x_4^2 + 4 x_8)$$ The figure inventory for three colors blue, green, and yellow is $b+g+y$. On "substitution" of the figure inventory into the cycle index, we have $$Z = \frac{1}{16}[(b+g+y)^8 + 4(b+g+y)^2 (b^2+g^2+y^2)^3 + 5(b^2+g^2+y^2)^4 + 2(b^4+g^4+y^4)^2 + 4(b^8+g^8+y^8)]$$
The coefficient of $b^4 g^1 y^3$ when $Z$ is expanded is $19$, so the number of distinct colorings with $4$ blue, $1$ green, and $3$ yellow beads is $19$.