$0-1$ poker games have been studied by Borel, Von Neumann and Morgenstern, and others. The most detailed treatment I know is by Bill Chen and Jerod Ankenman, who wrote a series of posts in rec.gambling.poker in 2003, "The [0,1] game: Part 1-14." Some of this was included in their 2006 book, The Mathematics of Poker.
The game you mention is "[0,1] Game #9" on pages 198-203.
Before going into the solution, note that it is a common misconception that mixed strategies are ubiquitous. In fact, for games with finitely many options, mixed strategies are typically required for finitely many boundary cases. In a game with finitely many states like rock-paper-scissors that may be everything, but in a game like this with no atoms in the distribution of cards, mixed strategies are not required at all. Because of the hidden information, when you bet, your opponent still will not know whether you are betting for value (with a strong hand which is ahead on average when called) or with a bluffing hand (which is behind your entire calling range). At the Nash equilibrium, you may be indifferent between betting or checking with a range of hands, when checking will never win and betting as a bluff gives the same value as giving up. So, instead of choosing a function which gives you a probability of betting when dealt that card, you might as well partition the possibilities into sets for each action.
The following excerpts show the Nash equilibrium:
X's strategy consists of the following thresholds:
- $x_0$ between check-folding and bluffing.
- $x_1^*$ between check-folding and check-calling.
- $x_1$ between value betting and check-calling.
Y's strategy likewise consists of three thresholds:
- $y_0$ between checking and bluffing if X checks.
- $y_1^*$ between calling X's bet and folding to X's bet (if X bets)
- $y_1$ between value-betting and checking if $X$ checks.
...
$0 \lt x_1 \lt y_1 \lt x_1^*, y_1^* \lt y_0 \lt x_0 \lt 1$
[This follows the convention that lower cards are stronger, and corrects a typo.]
...
Solution:
$y_1 = (1-\alpha)/(1+\alpha)$
$ x_1 = (1-\alpha)^2/(1+\alpha)$
$y_0 = (1+\alpha^2)/(1+\alpha)$
$x_0 = 1 - \alpha(1-\alpha)^2/(1+\alpha)$
$y_1^* = 1 - \alpha$
$x_1^*= 1-\alpha$
where $\alpha$ is the probability with which you fold a bluff-catcher with a bet of that size relative to the pot to make someone indifferent to bluffing. Since you have a bet of $1$ into a pot of size $2a$, $\alpha = 2a / (1+2a)$.
Best Answer
Here is an answer to the updated question:
Suppose that there are two betting rounds. Darth Vader has three types of hands. Type 1 wins with probability 1. Type 2 is a draw that hits (becomes a winning hand) with probability $1/10$ between the betting rounds. You can't see whether type 2 hands hit. Vader has Type 1 $1/100$ of the time, and type 2 $99/100$ of the time.
You act first. The pot is $1$. The effective stack depth is $4$.
Option 1: Push all-in (bet $4$). Then Vader calls with type-1 hands and folds type-2 hands. You gain $1$ unit $99\%$ of the time but lose not just the pot but an extra $4$ units $1\%$ of the time, for a net gain of $95\%$ of the pot.
Option 2: Check to Vader. Vader can choose to check behind. Then Vader has a winning hand with probability $1/100 + 99/1000 = 109/1000$ on the river with an effective stack depth of $4$. The Nash equilibrium strategy is for you to check to Vader, who bets $4$ for value with all winning hands, and neutralizes your folds by bluffing $4$ times for every $5$ value bets. So, Vader bets $981/5000$ of the time with $10.9\%$ value bets and $8.72\%$ bluffs. This means you might as well fold all of the time, although you call $1/5$ of the time to neutralize Vader's bluffs. If you fold every time Vader bets, you win $80.38\%$ of the pot.
We have not considered all of Vader's possible actions. (Vader can get more by raising some of the time.) However, this is enough to say that checking is worse than pushing all-in.
Option 3: Bet $0.5$, then play to neutralize Vader's raises and later bets with type-2 hands. Before we do the calculations, as long as it is possible to neutralize drawing (type-2) hands, this is better hand-by-hand than pushing all in (option 1) because Vader might as well fold type-2 hands, and we might fold against type-1 hands, saving some of the remaining $3.5$ units.
Suppose when Vader raises to $x$ total, then bets $4-x$ all-in on the next street, you call the raise with probability $p(x)$ and you call the push with probability $q(x)$.
We can set the river calling frequency $q(x)$ so that these have the same equity, so that bluffing on the river has the same expected value as giving up. Vader risks $4-x$ to try to gain $2x+1$, so we call with probability $q(x) = \frac{2x+1}{x+5}$. That means if Vader raises with a draw and gets called, this costs Vader $x$ to get a situation worth $\frac{1}{10}(2x+1 + \frac{2x+1}{x+5}(4-x))$, a net cost of $\frac{10x^2+32x-9}{10x+50}$.
On the first round, Vader risks $\frac{10x^2+32x-9}{10x+50}$ to get a reward of $\frac{3}{2}$. We can neutralize his decision by calling with probability $p(x) =\frac{\text{reward}}{\text{risk+reward}} = \frac{15x+75}{10x^2+47x+66}$. This means Vader's payoff comes only from type 1 hands.
Suppose we call with probability $p(x) = \frac{15x+75}{10x^2+47x+66}$ on the first betting round and $q(x) = \frac{2x+1}{x+5}$ on the second round. Given a type-1 hand, Vader gets $\frac{3}{2}$ with probability $\frac{10x^2+32x-9}{10x^4+47x+66}$, $x+1$ with probability $\frac{60-15x}{10x^2+47x+66}$, and $5$ with probability $\frac{30x+15}{10x^2+47x+66}$. The average is $ \frac{486x+243}{20x^2+94x+132}$. If Vader knows your strategy, the best he can do is to set $x=\frac{3\sqrt{2}-1}{2}=1.62132$ and get $3.05944$ every time he has a type-1 hand, or $3.06\%$ of the pot. This is lower than the $5\%$ Vader gets when you push, and lower than the $18+\%$ Vader gets if you check, so betting less than all-in into Vader is better than checking or pushing, though another amount may be better than $0.5$.