After hard work on this problem, I could find an approach to the solution that I am posting here
In this case, let’s assume that each of 12 identical circles, with a flat radius r, is inscribed by each of 12 congruent regular pentagonal faces of $a$ regular dodecahedron with an edge length $a$ such that regular dodecahedron is concentric with the spherical surface having the center $O$ & a radius $R$.
Thus, all 30 points of tangency of the circles, lying on the spherical surface, are coincident with the mid-points of all 30 edges of a regular dodecahedron.
Now, consider one of the 12 identical circles with the center $C$ on the flat face & a flat radius r, touching five other circles at the points A, B, D, E & F (lying on the spherical surface as well as on the edges of the dodecahedron) and is inscribed by a regular pentagonal face of the dodecahedron with an edge length $a$ . (See the figure 1 below showing a regular pentagonal face of dodecahedron)
The flat radius $r$ of the circle inscribed by a regular pentagonal face with edge length $a$ is given as $$r=\frac{a}{2}\cot\frac{\pi}{5} \implies a=2r\tan\frac{\pi}{5}$$ $$\color {blue}{a=2r\sqrt{5-2\sqrt{5}}} \tag 1$$
Now, the radius $R$ of the spherical surface passing through all 12 identical vertices of a dodecahedron with edge length $a$ is given as $$\color {red}{R=\frac{\sqrt{3}(\sqrt{5}+1)a}{4}} $$ Now, the normal distance ($h=OC$) of each pentagonal face (having a circumcribed radius $\frac{a}{2}\sin\frac{\pi}{5}$) from the center O of the dodecahedron is given as $$h=\sqrt{\left(\frac{\sqrt{3}(\sqrt{5}+1)a}{4}\right)^2-\left(\frac{a}{2}\sin\frac{\pi}{5}\right)^2}$$$$=\frac{a}{2}\sqrt{\frac{25+11\sqrt{5}}{10}}$$ Now, substituting the value of $a$ in terms of $r$, we get $$h=\frac{2r\sqrt{5-2\sqrt{5}}}{2}\sqrt{\frac{25+11\sqrt{5}}{10}}$$$$\implies \color{blue} {h=OC=\frac{(1+\sqrt{5})r}{2}}$$
Draw the perpendicular OC from the centre O of the spherical surface (i.e. centre of the regular dodecahedron) to the centre C of the plane (flat) circle & join any of the points A, B, D, E & F of tangency of the plane circle say point A (i.e. mid-point of one of the edges of dodecahedron) to the centre O of the spherical surface (i.e. the centre of dodecahedron).
Thus, we obtain a right $\Delta OCA$ (as shown in the figure 2 above)
Applying Pythagoras Theorem in right $\Delta OCA$ as follows $$(OA)^2=(OC)^2+(CA)^2$$ $$\implies (R)^2=\left(\frac{(1+\sqrt{5})r}{2} \right)^2+(r)^2 $$ $$\implies \color {blue}{r=R\sqrt{\frac{5-\sqrt{5}}{10}}}$$ From the figure 2 above, we have $$\sin\theta=\frac{CA}{OA}=\frac{r}{R}=\sqrt{\frac{5-\sqrt{5}}{10}}$$ $$\implies \color{blue}{\theta=\sin^{-1}\sqrt{\frac{5-\sqrt{5}}{10}}}$$
$$\text{arc radius of each circle}=arc AC'=R\theta$$ $$\color {green}{=R\sin^{-1}\sqrt{\frac{5-\sqrt{5}}{10}}\approx 0.553574358\space R}$$
Best Answer
Let $R$ be the (known) radius of the large inscribed circle, $r$ the radius of the small inscribed circle, and $(x,r)$ the center of this small circle. Then one has the two equations $$x^2+(R-r)^2=(R+r)^2,\qquad\sqrt{x^2+r^2}+ r=2R$$ in the two unknowns $r$ and $x$.