Illusie's article about étale cohomology available here (in French) mentions that the standard definition of compactly supported cohomology (and higher direct images with compact support) does not give the right answers in the case of étale cohomology.
He says Grothendieck had the idea of defining them as follows instead:
$$Rf_! = Rg_* \circ j_!$$
where $f = gj$ is a compactification of $f$: $g$ is proper, and $j$ is an open immersion. It takes a bit of work to see that this is well defined (a theorem of Nagata guarantees the existence of this compactification, and the proper base change theorem shows that the result does not depend on which compactification is chosen).
What is the explanation here for the difference between Grothendieck's definition and the usual one, which amounts to $Rf_! = R(g_* j_!)$ ? What is it that allows us to say that either is the "right" one?
It seems to me that a satisfactory explanation should be more than just computing the groups in a special case and appealing to intuition (indeed, the compactly supported cohomology groups are often strange, at first sight, even in the topological situation).
For example, I could imagine that an explanation might involve derived categories: after all, one of the main uses of $Rf_!$ is Verdier duality, which implies many useful properties such as Poincaré duality. This maybe gives some hints as to which definition is preferred.
Best Answer
It is important in etale cohomology, as it is topology, to define cohomology groups with compact support --- we saw this already in the case of curves in Section 14. They should be dual to the ordinary cohomology groups.
The traditional definition (Greenberg 1967, p162) is that, for a manifold $U$, $ H_{c}^{r}(U,\mathbb{Z})=dlim_{Z}H_{Z}^{r}(U,\mathbb{Z}) $ where $Z$ runs over the compact subsets of $U$. More generally (Iversen 1986, III.1) when $\mathcal{F}$ is a sheaf on a locally compact topological space $U$, define $ \Gamma_{c}(U,\mathcal{F})=dlim_{Z}\Gamma_{Z}(U,\mathcal{F}) $ where $Z$ again runs over the compact subsets of $U$, and let $H_{c}% ^{r}(U,-)=R^{r}\Gamma_{c}(U,-)$.
For an algebraic variety $U$ and a sheaf $\mathcal{F}$ on $U_{\mathrm{et}}$, this suggests defining $ \Gamma_{c}(U,\mathcal{F})=dlim_{Z}\Gamma_{Z}(U,\mathcal{F}), $ where $Z$ runs over the complete subvarieties $Z$ of $U$, and setting $H_{c}^{r}(U,-)=R^{r}\Gamma_{c}(U,-)$. However, this definition leads to anomolous groups. For example, if $U$ is an affine variety over an algebraically closed field, then the only complete subvarieties of $U$ are the finite subvarieties, and for a finite subvariety $Z\subset U$, $ H_{Z}^{r}(U,\mathcal{F})=\oplus_{z\in Z}H_{z}^{r}(U,\mathcal{F}). $ Therefore, if $U$ is smooth of dimension $m$ and $\Lambda$ is the constant sheaf $\mathbb{Z}/n\mathbb{Z}$, then $ H_{c}^{r}(U,\Lambda)=dlim H_{Z}^{r}(U,\Lambda)=\oplus_{z\in U}H_{z}% ^{r}(U,\Lambda)=\oplus_{z\in U}\Lambda(-m)$ if $r=2m$, and it is 0 otherwise These groups are not even finite. We need a different definition...
If $j\colon\ U\rightarrow X$ is a homeomorphism of the topological space $U$ onto an open subset of a locally compact space $X$, then $ H_{c}^{r}(U,\mathcal{F})=H^{r}(X,j_{!}\mathcal{F}) $ (Iversen 1986, p184). We make this our definition.
From Section 18 of my notes: Lectures on etale cohomology.