Let's say we have two biased coins. The probability of tossing a head on the first coin is $\alpha$ and the probability of tossing a head on the second coin is $1-\alpha$. We toss both coins $n$ times and we say that there is a success when there is a head on both coins. If we denote this random variable by $X$ then
$$X\sim B(n,\alpha-\alpha^2).$$
The question is how to properly estimate $\alpha$. The problem with a 'standard' approach ($\alpha – \alpha^2=\frac{k}{n}$ where $k$ is number of successes) is that the number of successes might be greater than $0.25n$ and in such a case we obtain that $\alpha$ is a complex number. We can treat such cases as $\alpha=0.5$, but then this estimator is not unbiased anymore.
Best Answer
The estimator is biased, regardless.
Note first that $\alpha$ is not identifiable because you cannot distinguish between $\alpha$ and $1-\alpha$. Let's accommodate this problem by allowing that we don't care which coin is which and stipulating (arbitrarily, but with no loss of generality), that $0 \le \alpha \le 1/2$.
It's reasonable, and conventional, to fix the estimator $g$ as follows:
$$\eqalign{ g(k,n) =& \frac{1 - \sqrt{\delta}}{2} \cr \delta =& \max(0, 1 - 4 k / n) }$$
No matter what you do, though, this will be a nonlinear function of the outcome $k$ and therefore is certain to be biased for almost all $\alpha$.
A better approach is to search for estimators $h(k,n)$ among some functional class of estimators (such as ones that are linear in $k$) that minimize the expectation of some loss function. In many situations an estimator that works well for quadratic loss also works well for many reasonable types of losses, so let's look at this. What we're talking about, then, is (for each $n$) to minimize the expectation $\mathbb{E}[(h(k,n) - \alpha)^2]$ among all estimators $h$.
Let's look graphically at what's going on. The bias of any estimator $h$ of the parameter $\alpha$ is the difference between its expectation and the parameter, $\mathbb{E}[h(k,n) - \alpha]$. We can study any proposed estimator, then, by graphing its bias (if we really care about that) and its loss. For any value of $n$ they are functions of $\alpha$, which (of course) is unknown. That's why we have to look at the entire graph.
Here are the bias (blue, dashed) and square root of the expected quadratic loss (red) for $g$ when $n=16$:
(I use the root of the loss because this is directly comparable to the bias.) For example, $g$ is unbiased for $\alpha \approx 1/3$ but otherwise is biased, with the size of the bias largest for $\alpha = 1/2$. The root expected loss is roughly between 0.15 and 0.2 provided $\alpha$ exceeds $1/6$, approximately.
As an alternative, consider linear estimators $h_\lambda(k,n)$ of the form $h_\lambda(k,n) = \lambda(n) k/n$. Here is a plot of $h_2$ also for $n=16$ (but please note the change in scale on the vertical axis):
For most $\alpha$ its bias exceeds that of $g$, but for some $\alpha$ (near 0.4) it actually has less bias. For a wide range of $\alpha$, though, its root expected loss is less than that of $g$. Provided $\alpha \gt 1/5$ or so, this simple estimator clearly outperforms the "obvious" one!
This is not necessarily "the best" linear estimator, however. To illustrate, here is a plot of $h_{4/3}$:
It outperforms both $g$ and $h_2$ for $1/8 \lt \alpha \lt 3/8$, approximately. Note, though, that $g$ outperforms the $h_{\lambda}$ for sufficiently small $\alpha$.
These considerations suggest there is value in knowing something about what $\alpha$ might be: that will tell you which portions of the loss graphs to focus on in selecting among alternative estimators. If, in addition, you have a prior distribution for $\alpha$ you can compute the expected loss (this is now a single number) and use that to compare estimators: your task becomes one of finding an estimator with lowest possible expected loss. This, of course, is a Bayesian estimator.
Regardless, using plots of expected loss is a standard and effective way to compare estimators and to choose ones that are appropriate for any particular problem.