I would like to know to which extent the theory developed for smooth projective varieties in the following articles
A. Białynicki-Birula, Some theorems on actions of algebraic groups.
Ann. of Math. (2) , 98:480–497, 1973.A. Białynicki-Birula, Some properties of the decompositions of
algebraic varieties determined by actions of a torus. Bull. Acad.
Polon. Sci. S ́er. Sci. Math. Astronom. Phys. , 24(9):667–674, 1976.
extends to the case of smooth non-complete (say, quasi-projective) varieties. Assume the ground field to be $\mathbb{C}$ if you want.
There are similar questions on MO, but I haven't found a satisfying answer: as Dori Bejleri comments below, my question is about "the decomposition into locally closed affine cells and the implications that has about the cohomology"; the question linked in the comment by Vivek Shende, on the other hand, "asks about the existence of an open cover by torus invariant affines, which while important to the proof of the BB decomposition, is not the same thing".
Best Answer
Everything holds for a smooth quasiprojective $X$ as long as there exists a $\mathbb{C}^*$ action so that
$$ \lim_{t \to 0} t \cdot x $$
exists for every $x \in X$. This is guaranteed when $X$ is projective but holds more generally. The theory really depends on a local analysis of torus actions at fixed points so as long as you have enough fixed points for the limits to exist (and $X$ is smooth), everything holds.
In the projective case, say you have a rank $r$ torus $T = (\mathbb{C}^*)^r$ acting on $X$, the one parameter subgroups $\mathbb{C}^* \cong T' \subset T$ form a lattice $N$ of rank $r$. For a projective $X$, any general enough $z \in N$ gives you a one parameter subgroup with the properties required to get a BB decomposition. Furthermore, scaling $z$ by a positive multiple does not change the decomposition. So you get a cone decomposition of $N \otimes_\mathbb{Z} \mathbb{R}$ that determine your BB decomposition.
As you vary within the interior of a top dimensional cone, the induced BB decomposition stays the same, and then when you cross the faces between the top dimensional cones (which correspond to degenerate choices of one parameter subgroup that do not give you enough independent conditions for the fixed points of $T'$ to be the same as those of $T$) you change the decomposition. Of course within each cone we need to pick a point that is scaled large enough.
So in the quasiprojective case, the difference is that now you have to stay within certain top dimensional cones where the limit above always exists, while the other regions of $N \otimes_\mathbb{Z} \mathbb{R}$ correspond to one parameter subgroups whose limit would be outside of the variety, say in an appropriate projective compactification.
Then once you have a decomposition into locally closed affine spaces you get the usual theorems about the cohomology vanishing in odd degree and being generated in even degree by the closures of the appropriate dimension affine cells, etc.
As an example consider $\operatorname{Hilb}^n(\mathbb{C}^2)$ which I will denote $H^n$ for convenience. $(\mathbb{C}^*)^2 = T$ acts on $\mathbb{C}^2$ in the obvious way and thus on $H^n$. The points of $H^n$ are ideals $I$ in $\mathbb{C}[x,y]$ so that $\dim_\mathbb{C}\mathbb{C}[x,y]/I = n$. The torus fixed points of this action correspond to the monomial ideals. If you pick a subtorus $(t^p,t^q)$ with $p,q > 0$, this gives a monomial order $w$ on $\mathbb{C}[x,y]$ and
$$ \lim_{t \to 0} (t^p,t^q)\cdot I = in_w(I) $$
the initial ideal with respect to this ordering, which for general orderings is always a monomial ideal in $H^n$ and so gives a BB decomposition. The special orderings where the initial ideal is not always a monomial ideal are the faces of the cones. However, if we pick $p,q < 0$, then this limit can be an ideal supported at infinity in $\mathbb{P}^2$ and so we do not get a BB decomposition of $H^n$.
See also this MO question: Cell decomposition for a variety not necessarily complete?.