The other direction is well known.
I think it is true and I was told by several other guys doing algebraic geometry that it is indeed true but they did not know how to prove. I am also wondering whether there is a general nonsense style proof. It looks like the statement can be proved by playing adjunctions in several categories. Note that a scheme $X$ is quasi compact iff the structure sheaf $O_X$ is a compact object in category of quasi coherent sheaves $Qcoh(X)$. We also know that if $X$ is noetherian scheme, then coherent sheaves are exactly the compact objects in $Qcoh(X)$, so it looks like this statement is equivalent to say if $f_*:Qcoh(X)\rightarrow Qcoh(Y)$ preseves compact objects, then $f$ itself is proper morphism of scheme. But this somehow means that $f^{-1}$ preserves compactness in the topological sense. (I think this can also be phrased into general nonsense language,say the compact objects in category of topological space.) Then one might be able play the commutativity of $Hom(M,?)$ functor and filtered colimits game to get proof.
All the comments are welcome
Thanks
Best Answer
Well, as already explained in the comment by Vivek Shende, the answer is no: there are non-proper morphisms with push-forward that preserves coherence. I've thought a bit on this the last couple of years and on the "positive" side we have:
Theorem 1. If $f\colon X\to Y$ is a universally closed morphism of finite type between noetherian schemes, then $f_*$ preserves coherence.
This is quite easy to show with Raynaud–Gruson's refined Chow Lemma (also when $X$ and $Y$ are algebraic spaces) and standard dévissage arguments. With a completely different proof I have also extended this to algebraic stacks.
The morphism $\mathbb{P}^2\setminus 0\to *$ does not preserve coherence as there exist non-proper curves. If $f_*$ preserves coherence after arbitrary noetherian base change, then $f$ is universally closed: the valuative criterion is easily seen to hold (as indicated by Piotr Achinger) since the fraction field of a DVR is not coherent. However, it turns out that it is not necessary to involve base change and it is also possible to characterize properness in terms of $R^1f_*$ (cf comment by Tom Graber).
Theorem 2. Let $f\colon X\to Y$ be a morphism of finite type between noetherian schemes. Then
(i) $f$ is universally closed if and only if $f_*$ preserves coherence, and
(ii) $f$ is proper if and only if $f_*$ and $R^1f_*$ preserves coherence.
The necessity of the conditions is Theorem 1 in (i) and well-known in (ii). To see that they are sufficient we need some lemmas that say that the valuative criteria can be checked using curves in $X$. A curve is a $1$-dimensional noetherian integral scheme. First we show that it is enough with curves of finite type over $Y$.
Lemma 1. Let $f\colon X\to Y$ be a morphism of finite type between noetherian schemes. If $f$ is not universally closed (resp. not separated) then there exists
(i) a separated curve C together with a morphism $C\to Y$ of finite type; and
(ii) a closed subscheme $C'$ of $X\times_Y C$ such that $C'$ is a curve and $C'\to C$ is birational and not surjective (resp. not separated).
Proof. By the valuative criterion, there exists a DVR $D$, a morphism $Y_0:=\operatorname{Spec} D\to Y$ and an integral closed subscheme $Z_0\subseteq X_0:=X\times_Y Y_0$ such that $Z_0\to Y_0$ is an open immersion: the inclusion of the generic point (resp. not separated). In the latter case, we can assume that there are two different sections of $Z_0\to Y_0$ that only agree over the generic point $U_0=\operatorname{Spec} K(D)$.
We now approximate the map $Y_0\to Y$ and the data $U_0\subseteq Y_0$, $Z_0\to Y_0$: there exists
Moreover, we can arrange so that $Z'\to Y'$ is equal to $U'\to Y'$ (resp. there are $2$ sections of $Z'\to Y'$ which only coincide over $U'$).
Pick a closed point $y'$ in $Y'\setminus U'$ and a generization $u'\in U'$ such that $C:=\overline{u'}$ has dimension $1$. Let $C'=Z\cap X'\times_{Y'} C$. QED
Now we refine Lemma 1 and show that it is enough to consider either "vertical curves" (contained in a fiber) or "horizontal curves" (birational to a curve in the base).
Lemma 2. Let $f\colon X\to Y$ be a morphism of finite type between noetherian schemes. If $f$ is not universally closed (resp. not separated) then there exists a birational morphism of curves $C'\to C$ which is not universally closed (resp. not separated) such that either:
(i) there is a finite morphism $C'\to X_y$ for a point $y\in Y$ (of finite type) and $C$ is a projective curve over $y$; or
(ii) there is a finite morphism $C\to \operatorname{Spec} \mathcal{O}_{Y,y}$ and $C'$ is a closed subscheme of $X\times_Y C$.
Proof. Let $C\to Y$ be a curve as in Lemma 1. The image is either a point or the morphism is quasi-finite.
Case (i): the image of $C$ is a point $y\in Y$. We may replace $C$ by a compactification and assume that $C\to y$ is projective. If the image of $C'\to X_y$ is a point, then it is a closed point and it follows that $C'\to C$ is finite which gives a contradiction since $C'\to C$ is not proper. Thus the image of $C'\to X_y$ is $1$-dimensional and $C'\to X_y$ is quasi-finite and proper, hence finite.
Case (ii): $C\to Y$ is quasi-finite. We may replace $Y$ with the schematic image of $C$ and localize at the image of a suitable closed point of $C$. Then $Y$ is integral and $1$-dimensional. Using Zariski's main theorem, we have $C\to \overline{C}\to Y$ where $C\to \overline{C}$ is an open immersion and $\overline{C}\to Y$ is finite. We may replace $C$ with $\overline{C}$ and $C'$ with its closure in $X\times_Y \overline{C}$. QED
Remark. The localization in (ii) is only necessary when $Y$ is not Jacobson.
The third lemma shows that curves as in Lemma 2 give rise to non-coherent cohomology.
Lemma 3. Let $f\colon C'\to C$ be a birational morphism of finite type of curves. If $f$ is not universally closed (resp. not separated), then $f_*\mathcal{O}_{C'}$ is not coherent (resp. $R^1f_*\mathcal{O}_{C'}$ is not coherent). If $C$ is projective over $k$, then in addition $\Gamma(C',\mathcal{O}_{C'})$ (resp. $H^1(C',\mathcal{O}_{C'})$) is infinite-dimensional.
Proof. For the first statement, it is enough to show that the sheaf is not coherent after passing to the henselization of a suitable point $c$ in $C$. We may thus assume that $C$ is local and henselian (although not necessarily irreducible). If $f$ is not universally closed, then (after replacing $C'$ with a connected component if it is reducible) $C'\to C$ is an open immersion. Then $f_*\mathcal{O}_{C'}$ is not coherent.
If $f$ is not separated, then (after replacing $C'$ with a connected component if it is reducible) $C'\to C$ is not separated and there is an open covering of $C'$ consisting of the local rings at the points above $c$. A Cech calculation gives that $H^0(C',\mathcal{O}_{C'})$ is coherent whereas $H^1(C',\mathcal{O}_{C'})$ is not coherent.
For the second statement, we note that there is a dense open subscheme $U\subseteq C$ such that $f_*\mathcal{O}_{C'}$ is coherent over $U$ and $R^1f_*\mathcal{O}_{C'}$ is zero over $U$. If $f_*\mathcal{O}_{C'}$ is not coherent, then choose a coherent subsheaf $\mathcal{F}\subseteq f_*\mathcal{O}_X$ that is an isomorphism over $U$. Then the cokernel $\mathcal{Q}$ is a non-coherent sheaf that is zero over $U$. It follows that $H^0(C,\mathcal{Q})$ is infinite, hence that $H^0(C,f_*\mathcal{O}_{C'})$ is infinite since $H^1(C,\mathcal{F})$ is finite. If $R^1f_*\mathcal{O}_{C'}$ is not coherent, then $H^0(C,R^1f_*\mathcal{O}_{C'})$ is infinite. QED.
Theorem 2 is an easy consequence of the lemmas above:
Proof of Theorem 2. We have seen that the conditions are necessary. Conversely, assume that $f$ is not universally closed (resp. not separated). We have to show that $f_*$ does not preserve coherence (resp. $R^1f_*$ does not preserve coherence). Let $C'\to C$ be as in Lemma 2 and choose a coherent sheaf $\mathcal{F}\in \mathrm{Coh}(X)$ restricting to the push-forward of $\mathcal{O}_{C'}$ in $\mathrm{Coh}(X\times \operatorname{Spec} \mathcal{O}_{Y,y})$. By Lemma 3, we have that $f_*\mathcal{F}$ (resp. $R^1f_*\mathcal{F}$) is not coherent at $y$. QED.
Remark 1. With small modifications, the proof of Theorem 2 also holds for algebraic spaces and Deligne–Mumford stacks: first one takes a finite cover of $Y$ to reduce to $Y$ a scheme, then uses the valuative criterion for stacks. An alternative proof for schemes, algebraic spaces and Deligne–Mumford stacks would be to use Raynaud–Gruson's Chow lemma to reduce the question to morphisms of the form $f\colon X\to \mathbb{P}^n_Y \to Y$ where $f\colon X\to \mathbb{P}^n_Y$ is étale and birational and $Y$ is a scheme.
Remark 2. The situation for algebraic stacks with infinite-dimensional stabilizers is more subtle. Theorem 1 holds as mentioned above and Theorem 2 (i) is probably ok. Theorem 2 (ii) however has to be modified. If $X$ has a good moduli space then it follows from Theorem 2 that $X$ has coherent cohomology if and only if $X_\mathrm{gms}$ is proper. For example, note that $BGL_n$ has coherent cohomology but is not separated. This is all well-known and one may argue that such stacks are "almost proper" (cf. paper by Halpern-Leistner–Preygel, arXiv:1402.3204).