$S$ = {necklaces of length 14 with 8 blue, 3 green and 3 brown beads}.
Clearly,$|S| = \frac {14!} {8!3!3!} $.
The group of symmetries, $G = D_{14}$. Clearly, $|G| = 28$. And as you have identified G is acting on S.
And by Burnside lemma, required answer is, $$ \#orbits = \frac 1 {|G|}*\sum_{\sigma \in G}fix(\sigma)$$
1) $\sigma = identity $
Then it fixes any element in $S$. Thus, $fix(\sigma) = |S| = \frac {14!} {8!3!3!}$
2) Rotations, $\sigma$ = rotation by $\frac {360} {14} degree$ clockwise = $p^1$
Carefully consider the cyclic structure of the permutation $\sigma$,
$$\begin{pmatrix}
1&2&3&4&5&6&7&8&9&10&11&12&13&14\\
2&3&4&5&6&7&8&9&10&11&12&13&14&1
\end{pmatrix}$$
$$\equiv (1\ 2\ 3\ 4\ 5\ 6\ 7\ 8\ 9\ 10\ 11\ 12\ 13\ 14) \ \ \text{[in cycle notation]}$$
If $\sigma$ fixes $x \in S$, then all vertex of the 14-gon must have the same color which is not the case, hence $fix(\sigma)=0$.
Clearly, $fix(\sigma=p^{13})=0$.
3)$\sigma=p^2$ i.e
$$\begin{pmatrix}
1&2&3&4&5&6&7&8&9&10&11&12&13&14\\
3&4&5&6&7&8&9&10&11&12&13&14&1&2
\end{pmatrix}$$
$$\equiv (1\ 3\ 5\ 7\ 9\ 11\ 13)(2\ 4\ 6\ 8\ 10\ 12\ 14) \ \ \text{[in cycle notation]}$$.
So, $\sigma$ has two cycles of length 7 and if you think carefully, for x to be in $fix(\sigma)$ all vertices in a single cycle will have same bead color. Which is not possible with 8 red, 3 blue and 3 brown beads. So, $fix(\sigma)=0$.
Clearly, $fix(\sigma=p^{12})=0$.
In the same fashion compute the $fix(\sigma)$ for the remaining by looking into the cycle structure and then use the burnside to get the required answer.
You should use Burnside's lemma.
There are $4$ rotations of order $12$. Each of these stabilizes $2$ colorings.
There are $2$ rotations of order $6$. Each of these stabilizes $4$ colorings.
There are $2$ rotations of order $4$. Each of these stabilizes $8$ colorings.
There are $2$ rotations of order $3$. Each of these stabilizes $16$ colorings.
There is $1$ rotation of order $2$. Each of these stabilizes $64$ colorings
There is $1$ rotation of order $1$. It stabilizes the $4096$ colorings.
We now apply Burnside and obtain:
$\frac{4\cdot2+2\cdot4+2\cdot8+2\cdot16+1\cdot64+1\cdot 4096}{12}=352$ necklaces.
Best Answer
You are correct, you want to use Burnside's lemma:
$$\mbox{# of Orbits}=\frac{1}{|G|}\sum_{g\in G}|fix(g)|.$$
You have an action of $G=D_{2m}$ on the set $X$ of $n$-colored necklaces with $m$ beads, and two such necklaces are the same if they are in the same orbit under this action. Therefore, counting the number of orbits counts the number of necklaces.
Let's set up some notation. We can regard a necklace as a regular $n$-gon with vertices numbered $\{1,\ldots,m\}$. Then, $G$ act on the vertices in the natural way. This corresponds to an embedding $G\hookrightarrow S_m$. Identifying an element $g\in G$ with its image in $S_m$, let $\psi(g)$ denote the number of disjoint cycles in the cycle decomposition of $g$.
Let $C$ be a set of $n$ colors (i.e. $|C|=n$). Then $G$ acts on $X=C^m$ by $$g.(c_1,\ldots,c_m)=(c_{g.1},\ldots,c_{g.m}).$$
Lemma: For the action of $G$ on $X$, we have $|fix(g)|=n^{\psi(g)}$.
proof: Since, for all $k$, $g^k.(c_1,\ldots,c_m)=(c_{g^k.1},\ldots,c_{g^k.m})$ we see that $(c_1,\ldots,c_m)$ is in $fix(g)$ if, and only if $c_i=c_{g^k.i}$ for all $i$ and $k$. This just says that $(c_1,\ldots,c_m)\in fix(g)$ if, and only if the coloring is constant on the cycles of $g$. There are $n$ choices to color each cycle, leading to $n^{\psi(g)}$ total colorings that are fixed by $g$.
Example: Suppose $m=4$ and $n=3$. The element of $D_8$ and their cycles types are chosen as follows: $$ \begin{array}{llll} 1=(1)(2)(3)(4)&r=(1234) &r^2=(13)(24)&r^3=(4321)\\ j=(24)(1)(3)&rj=(12)(34)&r^2j=(13)(2)(4)&r^3j=(14)(23) \end{array} $$ The corresponding orders of $fix(g)$ are given by $$ \begin{array}{llll} n^4&n&n^2&n\\ n^3&n^2&n^3&n^2 \end{array} $$ Therefore, the number of necklaces with $n$ colors is $$\frac{1}{8}(n^4+2n^3+3n^2+2n).$$ Substituting $n=3$ yields $21$ distinct necklaces.