This problem seems simple...but its not. For example, see here for a rather complex analysis for the prima facie simple case of ratios of normal rv and ratios of sums of uniforms.
In general, if your pairs are not from a bivariate gaussian, there is no nice formula for $E[R^2]$.
Note:
$$R_n=\frac{n\sum x_iy_i-\sum x_i\sum y_i}{n^2s_Xs_Y}$$
This mess will have some distribution $f_{R_n}(r)$ that will be very sensitive to $n$.
I think your best bet is to simulate this (Monte Carlo) for $n\in [2....N]$ using a large number of trials (you can check convergence by running each simulation twice, with randomly chosen seeds and comparing these results to each other and to results from $n-1$).
Once you have this data, you can fit a curve to the it or some transformation thereof. Your general equation looks reasonable in terms of how the curve will look, since:
$$E[R^2_n] \xrightarrow{p} 0$$ for correlations between independent variables
Possible Solution
Since your variables are independent, I realized that we are really looking for the variance of the sample correlation (i.e., the square of the expected value of the standard error of the correlation coefficient (see p.6):
$$se_{R_n}=\sqrt{\frac{1-R^2}{n-2}}$$. However, you already know the true value of $R^2$, so you can increase the df in the denominator to get:
But: $R^2=0$ for independent variables, so it reduces to:
$$(se_{R_n})^2=\sigma^2_{R_n}=E[R^2_n]=\frac{1}{n-1}$$
There you have it...it matches your empirical results. As per Wolfies, I should note that this is an asymptotic result, but sums of uniform RVs generally exhibit good convergence properties ala CLT, so this may explain the good fit.
For further reading, see @soakley's nice reference. I was able to pull the relevant page from JSTOR:
or, if you're really motivated, you can get this recent article (2005) about your exact problem.
Another is to write $X=\sqrt 3 Z + Y$ where $Z$ is independent of $Y$ and has the same distribution as $Y$ and has $EZ=0$ and $V(Z)=1$ (that is, $N(0,1)$).
Check: $V(X) = 3V(Z)+V(Y) = 3 + 1 = 4$, and $\text{Cov}(X,Y)= \text{Cov}(Y,Y) =1$. Then $X^2Y^2 = 3Z^2Y^2+2\sqrt 3 ZY^3 + Y^4$, whose expectation is $EZ^2EY^2+2\sqrt3 EZEY^3+EY^4=3+0+3=6$. (The 4th moment of $Y$ is $3$; see the Wikipedia page.)
Best Answer
Let $X = \min(\xi, \eta)$. There's a $1/2$ chance that $\eta = 0$, in which case $E(X|\eta = 0) = E(X^2|\eta = 0) = 0$, and there's a $1/2$ chance that $\eta = 1$, in which case $E(X | \eta = 1) = E(\xi) = 1/2$ and $E(X^2 | \eta = 1) = E(\xi^2) = \int_0^1 x^2\, dx = 1/3$. Putting these together gives $E(X) = 1/4$ and $E(X^2) = 1/6$, and the variance of $X$ is $E(X^2) - E(X)^2 = 5/48$.