When all else fails, get a bigger hammer
...which I did and found that the answer was $R=6$.
I won't explain why $AC$, $BD$ $PR$ and $QS$ concur at $F$. It's just a special case of
Brianchon's theorem. So let's start from there. Notice the angles $\angle CAB=\angle ACD=\alpha$, $\angle ABD=\angle BDC=\beta$ and segment $OF=x$. We'll use them all the time. For the sake of simplicity I will also introduce the following symbols: $AK=p_1$, $LC=p_2$, $BM=q_1$, $DN=q_2$. We know that $p_1p_2=16$ and $q_1q_2=9/4$.
First, let's try to find $p_1=AK$. The same approach will be used to find segments $p_2,q_1,q_2$
Take a look at quadrilateral $AQOK$:
$$AQ=FQ\cot\alpha=AK\cos \alpha+OK\sin\varphi$$
$$QO=AK\sin\alpha+OK\cos\varphi$$
This leads to the following equations:
$$p_1\cos\alpha+R\sin\varphi=(R+x)\cot\alpha$$
$$p_1\sin\alpha+R\cos\varphi=R$$
or:
$$R\sin\varphi=(R+x)\cot\alpha-p_1\cos\alpha$$
$$R\cos\varphi=R-p_1\sin\alpha$$
Square these two equations and add them. This will eliminate the angle $\varphi$ that we don't care about. You will end up with a quadratic equation with respect to $p_1$:
$$R^2=((R+x)\cot\alpha-p_1\cos\alpha)^2+(R-p_1\sin\alpha)^2$$
This equation has two solutions. The bigger one is actually equal to $AL$ and the smaller one is $AK=p_1$:
$$p_1=-\sqrt{R^2-\frac{1}{2} x^2 (1+\cos (2 \alpha))}+(R+x)\cos \alpha \cot \alpha +R \sin \alpha\tag{1}$$
There is no need to calculate $p_2$ in a similar way. Everything would look the same, except that we would use $R-x$ instead of $R+x$ along the way. So we can obtain an expression for $p_2$ just by replacing $x$ with $-x$:
$$p_2=-\sqrt{R^2-\frac{1}{2} x^2 (1+\cos (2 \alpha))}+(R-x)\cos \alpha \cot \alpha +R \sin \alpha\tag{2}$$
Now comes the hardest part: you have to multiply (1) and (2). It looks like we are going nowhere... After a lot of simplifications, we get:
$$p_1p_2=-\frac{1}{2} \csc ^2\alpha \left(\left(R^2+x^2\right)\cos (2 \alpha) +2 R \sin \alpha \sqrt{4 R^2-2 x^2 (1+\cos (2 \alpha))}-3 R^2+x^2\right)\tag{3}$$
At least we know it's 16 :)
The good thing is that we don't have to go through the same torture to find $q_1q_2$, the process is exactly the same except that we have $\beta$ instead of $\alpha$:
$$q_1q_2=-\frac{1}{2} \csc ^2\beta \left(\left(R^2+x^2\right)\cos (2 \beta) +2 R \sin \beta \sqrt{4 R^2-2 x^2 (1+\cos (2 \beta))}-3 R^2+x^2\right)\tag{4}$$
Like it's not difficult enough we have to introduce the fact that $x$, $\alpha$ and $\beta$ are not independent! There's a connection between them.
Take a look at triangle $\triangle ABC$:
$$AB=AQ+QB=(R+x)(\cot\alpha+\cot\beta)$$
$$BC=BR+RC=BQ+CS=(R+x)\cot\beta+(R-x)\cot\alpha$$
$$AC={2R \over \sin\alpha}$$
If you replace all that into a well-known equation:
$$BC^2=AB^2+AC^2-2 AB\cdot AC\cos\alpha$$
...you will get a very simple relation (trust me):
$$x^2=R^2(1-\tan\alpha\tan\beta)\tag{5}$$
Now replace (5) into (3) and (4) and we'll get something that we can easily deal with:
$$p_1p_2=R^2\left(2+\frac{\tan\beta}{\tan\alpha}-2\sqrt{1+\frac{\tan\beta}{\tan\alpha}}\right)$$
$$q_1q_2=R^2\left(2+\frac{\tan\alpha}{\tan\beta}-2\sqrt{1+\frac{\tan\alpha}{\tan\beta}}\right)$$
...or, if we introduce $z=\frac{\tan\alpha}{\tan\beta}$:
$$p_1p_2=R^2(2+z-2\sqrt{1+z})=R^2(\sqrt{1+z}-1)^2=16\tag{6}$$
$$q_1q_2=R^2\left(2+\frac1z-2\sqrt{1+\frac1z}\right)=R^2\left(\sqrt{1+\frac1z}-1\right)^2=\frac{9}{4}\tag{7}$$
Now divide (6) by (7) and you get:
$$\left(\frac{\sqrt{1+z}-1}{\sqrt{1+\frac1z}-1}\right)^2=\frac{p_1p_2}{q_1q_2}=\frac{16}{\frac94}=\frac{64}{9}$$
$$\frac{\sqrt{1+z}-1}{\sqrt{1+\frac1z}-1}=\frac{8}{3}$$
$$3(\sqrt{1+z}-1)=8\left(\sqrt{1+\frac1z}-1\right)\tag{8}$$
You should be able to show that equation (8) has only one solution:
$$z=\frac{16}9$$
Replace that into (6) and you get:
$$R^2=\frac{p_1p_2}{(\sqrt{1+z}-1)^2}=36\implies R=6$$
Best Answer
Let O is center of circumference, E is middle of AB, F is middle of CD, OA=$R$, OE=$x$. Then OF=$|\sqrt{3}-x|$.
Then $R^2=OA^2=AE^2+OE^2=4+x^2$, $R^2=OD^2=DF^2+OF^2=9+3-2\sqrt{3}x+x^2$.
Then $4+x^2=9+3-2\sqrt{3}x+x^2 \Rightarrow$ $2\sqrt{3}x=8 \Rightarrow$ $x=\frac{4}{\sqrt{3}} \Rightarrow$ $R^2=4+\frac{16}{3}=\frac{28}{3} \Rightarrow$ $R=2\sqrt{\frac{7}{3}}$.