I would like to know how I can get the coordinates of four control points of a Bézier curve that represents the best approximation of a circular arc, knowing the coordinates of three points of the corresponding circle. I would like at least to know the solution to this problem in the case where two of the known circle points are the two ends of a diameter of the circle.
[Math] Bézier curve approximation of a circular Arc
bezier-curvecirclesgeometry
Related Solutions
From the equation of a cubic Bézier curve, the midpoint of the curve is $\frac{1}{8}(P_1 + 3 P_2 + 3 P_3 + P_4)$. $x$ is the distance from that midpoint to $\frac{1}{2}(P_1 + P_4)$, i.e. $|\frac{3}{8}(-P_1 + P_2 + P_3 - P_4)|$.
As you're probably aware, you can put the control points wherever you like, so this answer isn't about where the control points should go, but about how to put them where you want them. If I understand correctly, you want them to be at the same distance from $B$ as the red and green $B_1$ and $B_2$, but not on the lines $AB$ and $BC$, but on a line perpendicular to the bisector of $\angle ABC$.
So you need a vector pointing along that bisector. Several operations will require normalizing vectors, so I'll denote by $n(\vec x)=\vec x/|\vec x|$ the unit vector along $\vec x$; in coordinates, this would be
$$x_1'=\frac{x_1}{\sqrt{x_1^2+x_2^2}}\;,$$ $$x_2'=\frac{x_2}{\sqrt{x_1^2+x_2^2}}\;.$$
If we denote by $\vec a, \vec b, \vec c$ the vectors for points $A$, $B$ and $C$, respectively, and the differences $\vec a - \vec b$ and $\vec c - \vec b$ by $\Delta\vec a$ and $\Delta\vec c$, respectively, then the vector $\vec m=n(n(\Delta \vec a)+n(\Delta \vec c))$ is a unit vector along the bisector of $\angle ABC$.
Now comes the tricky part. It's easy to get just any vector perpendicular to $\vec m$, but if I understand correctly you want to go different lengths to the two sides, corresponding to $1/6$ of the length of the corresponding line segment, so you have to get the signs right. You can do this by projecting $\vec m$ out of $\Delta \vec a$ and $\Delta \vec c$, respectively:
$$\vec m_{a}=n(\Delta\vec a-(\Delta\vec a\cdot\vec m)\vec m)\;,$$ $$\vec m_{c}=n(\Delta\vec c-(\Delta\vec c\cdot\vec m)\vec m)\;.$$
Then you just have to add the desired multiples of these vectors to $\vec b$:
$$\vec b_1 = \vec b + \frac{|\Delta\vec a|}6 \vec m_a\;,$$ $$\vec b_2 = \vec b + \frac{|\Delta\vec c|}6 \vec m_c\;.$$
[Edit in response to the comment:]
I'm not sure whether I fully understand what you're asking for since I don't know what "initial formula" refers to (and the comment is partly ungrammatical), but I gather that you're more comfortable with coordinate notation than with vector notation, so I'll write all the equations out in coordinate notation; I hope that will enable you to implement them.
I'd already done this for the normalization operation; so here again for completeness: The operation $n(\vec x)$ applied to the vector $\vec x$ with coordinates $(x_1,x_2)$ results in a unit vector $\vec x'=\vec x/|\vec x|$ with coordinates $(x_1',x_2')$ given by
$$x_1'=\frac{x_1}{\sqrt{x_1^2+x_2^2}}\;,$$ $$x_2'=\frac{x_2}{\sqrt{x_1^2+x_2^2}}\;.$$
The coordinates $(\Delta a_1,\Delta a_2)$ of the vector $\Delta\vec a$ are given by $\Delta a_1=a_1-b_1$ and $\Delta a_2=a_2-b_2$, respectively, and likewise $\Delta c_1=c_1-b_1$ and $\Delta c_2=c_2-b_2$ for the coordinates $(\Delta c_1,\Delta c_2)$ of the vector $\Delta\vec c$.
The scalar product $\vec x\cdot\vec s$ of vectors $\vec x$ and $\vec s$ with coordinates $(x_1,x_2)$ and $(s_1,s_2)$, respectively, is $x_1s_1+x_2s_2$, so e.g. the coordinates of $\Delta\vec a-(\Delta\vec a\cdot\vec m)\vec m$ are $\Delta a_1-(\Delta a_1 m_1+\Delta a_2m_2)m_1$ and $\Delta a_2-(\Delta a_1 m_1+\Delta a_2m_2)m_2$. The norm (length) $|\vec x|$ of a vector $\vec x$ with coordinates $(x_1,x_2)$ is given by $\sqrt{x_1^2+x_2^2}$, so the first coordinate of the first of the last two equations is
$$b_{11}=b_1+\frac{\sqrt{\Delta a_1^2+\Delta a_2^2}}6m_{a1}\;.$$
I hope that's roughly what you were looking for.
Best Answer
For a unit semi-circle centered at the origin, the points are $(1,0)$, $(1, \tfrac43)$, $(-1, \tfrac43)$, $(-1,0)$. Translate, rotate, and scale as needed.
If the end-points of the diameter are $\mathbf{P}$ and $\mathbf{Q}$, proceed as follows:
Let $\mathbf{U}$ be a vector obtained by rotating $\vec{\mathbf{P}\mathbf{Q}}$ through 90 degrees. Then the control points are $\mathbf{P}$, $\mathbf{P} + \tfrac23 \mathbf{U}$, $\mathbf{Q} + \tfrac23 \mathbf{U}$, $\mathbf{Q}$.
Pseudocode is as follows
For general circular arcs, complete details are given in "Good approximation of circles by curvature-continuous Bézier curves", by Tor Dokken, Morten Dæhlen Tom Lyche, Knut Mørken, Computer Aided Geometric Design Volume 7, Issues 1–4, June 1990, Pages 33-41.