There are smooth counterexamples. Let $S_0$ be a smooth separated irreducible scheme over a field $k$ with dimension $d > 1$, and $s_0 \in S_0(k)$. Blow up $s_0$ to get another such scheme $S_1$ with a $\mathbf{P}^{d-1}_k$ over $s_0$. Blow up a $k$-point $s_1$ over $s_0$ to get $S_2$, and keep going. Get pairs $(S_n, s_n)$ so that the open complement $U_n$ of $s_n$ in $S_n$ is open in $U_ {n+1}$ and is strictly contained in it. Glue them together in the evident manner, to get a smooth irreducible $k$-scheme. It is locally of finite type, but is not quasi-compact (since the $U_n$ are an open cover with no finite subcover). This is separated (either by direct consideration of affine open overlaps, or by using the valuative criterion).
I think an initial object exists if you work with integral excellent schemes (maybe integral is not really necessarily, but then require that $X$ be schematically dense in $Z$).
So suppose $X, Y$ are integral and excellent. Consider all possible factorizations $X\to Z_{\alpha} \to Y$ with $Z_{\alpha}$ integral. Then $K(Z_{\alpha})=K(X)$. For any pair $Z_{\alpha}, Z_{\beta}$, the closure $Z_{\gamma}$ of $X$ in $Z_{\alpha}\times_Y Z_{\beta}$ gives a factorization $X\to Z_{\gamma}\to Y$ with $Z_{\gamma}$ dominating $Z_{\alpha}$ and $Z_{\beta}$, finite over $Y$, and $X\to Z_{\gamma}$ is an open immersion (one checks that $X\to Z_{\gamma}$ is an immersion, hence open in some closed subscheme $F$, but $X$ is birational to $Z_{\gamma}$, so $F=Z_{\gamma}$). Thus we can consider the projective limite $Z$ of the $Z_{\alpha}$'s.
By construction $Z$ is affine and integral over $Y$. As $Z_{\alpha}$ is dominated by the normalization $\widetilde{Y}$ of $Y$ in $K(X)$ and $\widetilde{Y}$ is finite over $Y$ by excellent hypothesis, $Z$ is finite over $Y$. It remains to see that the canonical map $X\to Z$ is an open immersion. This property is local over $Y$. So we suppose $Y$ is affine. Cover $X$ by principal affine open subsets $D(h)$'s of some $Z_{\alpha_0}$. Then $D(h) \to D_Z(h)$ is a closed immersion because $D(h)\to D_{Z_{\alpha_0}}(h)$ is, and it is birational, so it is an isomorphism and we are done.
It would interesting to compute explicitely the projective limite in some concrete situations. For exemple, consider a surface $S$, finite over $\mathbb A^2_{\mathbb C}$, with non-normal locus $\Delta$. Let $X$ be an open subset of $S$ with $\Delta\cap X$ non-empty and not equal to $\Delta$. The inclusion $X\to S=Z_{\alpha_0}$ is a factorisation. But what is the $Z$ constructed as above ?
Best Answer
Let $k$ be an algebraically closed field of characteristic zero and let $X$ be a projective variety over $k$. To try and answer your first two questions, let us try formalize the property that Hom-schemes $\mathrm{Hom}(Y,X)$ are finite unions of quasi-projective schemes.
Definition. We say that $X$ is bounded over $k$ if, for every normal projective variety $Y$ over $k$, the Hom-scheme $\mathrm{Hom}(Y,X)$ is of finite type over $k$.
We can then show the following result; see [1] and [2].
(I think that your question is only concerned with 1 and 6 in the following result, but the rest might also be useful to you.)
Theorem. The following are equivalent.
Boundedness is most likely equivalent to "hyperbolicity". In fact, it is implied by hyperbolicity (Kobayashi, Brody, ...). More precisely:
This should answer your first question. For your second question: if $X$ has no rational curves, then every Hom-scheme $Hom(Y,X)$ has projective components (but it can have infinitely many components). Conversely, if all Hom-schemes $Hom(Y,X)$ have (only) projective components, then $X$ has no rational curves. (Use that the components of $Hom(\mathbb{P}^1,\mathbb{P}^1)$ are affine varieties of increasing dimension.)
Abelian varieties have no rational curves and give examples of non-finite type Hom-schemes with each component projective. Note that non-trivial abelian varieties are far from being hyperbolic, so there's no contradiction to the above conjecture.
References.
[1] R. van Bommel, A. Javanpeykar, L. Kamenova. Boundedness in families with applications to arithmetic hyperbolicity https://arxiv.org/abs/1907.11225
[2] A. Javanpeykar and L. Kamenova Demailly's notion of algebraic hyperbolicity: geometricity, boundedness, moduli of maps https://arxiv.org/abs/1807.03665