The short answer is "because they don't work," but that's kind of a copout. This is actually quite a deep question. What you're referring to is sphere packing in two dimensions, specifically the kissing number, and sphere packing is actually quite a sophisticated and active field of mathematical research (in arbitrary dimensions).
Here's one answer, which isn't complete but which tells you why $6$ is a meaningful number in two dimensions. The packing you refer to is a special type of packing called a lattice packing, which means it comes from an arrangement of regularly spaced points; in this case, the hexagonal lattice. The number $6$ appears here because the hexagonal lattice has $6$-fold symmetry. So a natural question might be whether one can find lattices in two dimensions with, say $7$-fold or $8$-fold symmetry, since these might correspond to circle packings with more circles around a given circle. (Intuitively, we expect more symmetric lattices to give rise to denser packings and to packings where each circle has more neighbors.)
The answer is no: $6$-fold symmetry is the best you can do! This is a consequence of the crystallographic restriction theorem. The generalization of the theorem to $n$ dimensions says this: it is possible for a lattice to have $d$-fold symmetry only if $\phi(d) \le n$, where $\phi$ is Euler's totient function.
The generalization implies that you still cannot do better than $6$-fold symmetry in $3$ dimensions. There are two natural lattice packings in $3$ dimensions, which both occur in molecules and crystals in nature and which both have $6$-fold symmetry, and it turns out that these are the densest sphere packings in $3$ dimensions. It also turns out that they give the correct kissing number in $3$ dimensions, which is $12$ (see the wiki article).
In $4$ dimensions, the kissing number is $24$, and I believe the corresponding packing is a lattice packing coming from a lattice with $8$-fold symmetry, which is possible in $4$ dimensions. In higher dimensions, only two other kissing numbers are known: $8$ dimensions, where the $E_8$ lattice gives kissing number $240$, and $24$ dimensions, where the Leech lattice gives kissing number $196560$! These lattices are really mysterious objects and are related to a host of other mysterious objects in mathematics.
A great reference for this stuff, although it is a little dense, is Conway and Sloane's Sphere Packing, Lattices, and Groups (Wayback Machine). Edit: And for a very accessible and engaging introduction to symmetry in the plane and in general, I highly recommend Conway, Burgiel, and Goodman-Strauss's The Symmetries of Things.
Best Answer
If you connect the centers of two adjacent little circles and the center of the big one, you'll get a triangle. The sides of this triangle have lengths $r+R, r+R$ and $2r$. A little trigonometry will get you that the top angle is
$$\theta=2\arcsin\left(\frac{r}{r+R}\right) \; .$$
Since you want the small circles to form a closed ring around the big circle, this angle should enter an integer amount of times in $360°$ (or $2\pi$ if you work in radians). Thus,
$$\theta=360°/n \; .$$
From this, you can compute that
$$r=\frac{R \sin(180°/n)}{1-\sin(180°/n)} \; .$$
Here's a plot of the result for $n=14$:
Here's the code in Scilab: