The key theorem for this exercise in the context of Hartshorne chapter I is theorem I.5.7A:
Theorem (Elimination Theory). Let $f_1,\cdots,f_r$ be homogeneous polynomials in $x_0,\cdots,x_n$, having indeterminate coefficients $a_{ij}$. Then there is a set $g_1,\cdots,g_t$ of polynomials in the $a_{ij}$, with integer coefficients, which are homogeneous in the coefficients of each $f_i$ separately, with the following property: for any field $k$, and for any set of special values of the $a_{ij}\in k$, a necessary and sufficient condition for the $f_i$ to have a common zero different from $(0,0,\cdots,0)$ is that the $a_{ij}$ are a common zero of the polynomials $g_j$.
This lets us prove that any map out of a projective variety is closed. $\renewcommand{\PP}{\mathbb{P}}$
To do this, we need to show that $\PP^n\times Y\to Y$ is a closed map for any variety $Y$.
By covering $Y$ with affines, we may reduce to the case that $Y$ is affine.
Now suppose $Z\subset \PP^n\times Y$ is closed: this means that it's cut out by a list of equations $f_1,\cdots,f_r$ which are homogeneous polynomials in the homogeneous coordinates on $\PP^n$ with coefficients taken from $A(Y)$.
By theorem I.5.7A, we may find polynomials $g_1,\cdots,g_s$ in the coefficients of these $f_i$ which vanish iff all the $f_i$ simultaneously vanish.
But this exactly means that the image of $Z\subset \PP^n\times Y$ under the projection $\PP^n\times Y\to Y$ is closed, or that the projection $\PP^n\times Y\to Y$ is closed.
Now consider a morphism $f:Z\to Y$ where $Z$ is projective and $i:Z\to \PP^n$ is the inclusion of $Z$ in to some projective space (guaranteed by the definition of projective).
We can factor $f$ as $Z\to \PP^n\times Y\to Y$ where the first map is $z\mapsto (i(z),f(z))$.
To show the image of this map is closed, cover $\PP^n\times Y$ by affine open subsets of the form $U_i\times Y$ for $U_i$ the standard affine opens.
Then on each of these opens, the set of points $(i(z),f(z))$ is closed because it's cut out by the equations for $Z\cap U_i$ and then also the equalities $y_i=f_i(z)$ where $y_i$ are the coordinate functions on $Y$ and $f_i$ are the components of the map $f$.
So the image of $Z$ in $\PP^n\times Y$ is closed and therefore the image of $Z$ in $Y$ is closed since the projection $\PP^n\times Y\to Y$ is closed.
To apply this to our problem, write $i:X\to Y$ for the inclusion.
Then by the above, $i(X)$ is closed in $Y$.
Question: "I know I can use 6.7 and 6.8 to extend the morphism f to such a ϕ. But I don't know how to prove it's surjective."
Answer: Let $C$ be an irreducible smooth projective curve over an algebraically closed field $k$ and let $K:=K(C)$ be the function field of $C$ with $x\in K$ a "non-constant" rational function. Let $P(t)\in k[t]$ and assume $P(x)=0$. Since $k$ is algebraically closed this implies $x\in k$ a contradiction, hence $x$ is "transcendental over $k$", meaning the subring $k[x] \subseteq K$ generated by $k$ and $x$ is isomorphic to a polynomial ring. You get an embedding $k \subseteq k(x) \subseteq K$ where $x$ behaves like an independent variable over $k$. This gives (by the proof of HH.I.6.12) a surjective morphism
$$\phi: C \rightarrow \mathbb{P}^1_k.$$
Since $C$ is projective and irreducible, the image is closed and irreducible and hence $Im(\phi)=\mathbb{P}^1_k$. The inverse image $\phi^{-1}(x)$ of a closed point $x\in \mathbb{P}^1_k$ is a strict closed subvariety of $C$ (since $\phi$ is non-constant) and since $C$ is irreducible it follows $\phi^{-1}(x)$ must be a finite set of points.
Best Answer
$\def\PP{\mathbb{P}}\def\ol{\overline}\def\pd{\partial}\def\AA{\mathbb{A}}$
You're right, there's a lot of handwaving when one says "$X$ is birational to the cone on $Y$". (And if that statement can be made rigorous in a simple way, the fact that I'm complaining about it and posting a longer solution will surely provoke someone in to demonstrating this fact.)
Without loss of generality, suppose $P\in D(x_0)$ and consider the rational map $Y\times\PP^1\to X$ given on $(D(x_0)\cap Y)\times\AA^1$ by $(Q,t)\mapsto P+Qt$. This surjects on to an open dense set of the collection of points $R$ which are on a line of the form $\ol{PQ}$ for $Q\neq P$, of which $X$ is the closure. As $Y\times\PP^1$ is irreducible, $(D(x_0)\cap Y)\times\AA^1$ is irreducible, the image of $(D(x_0)\cap Y)\times\AA^1$ is irreducible, and the closure of its image is irreducible, so $X$ is irreducible and this rational map from $Y\times \PP^1$ is dominant. Thus we have an inclusion of function fields $k(X)\hookrightarrow k(Y\times\PP^1)$, so $\dim X\leq \dim Y+1$. On the other hand, $Y\subset X$, so $\dim X \geq \dim Y$. All we need to do to show that $\dim X=\dim Y+1$ is to find a point in $X$ not in $Y$. (Many claimed solutions to this problem say something like 'we can construct a rational inverse' to the map defined above and skip over the existence of such a point - this is problematic, because without this point, one can't actually define such an inverse!)
We will prove the claim by induction on $\dim Y$. If $\dim Y=1$, then I claim that for any $Q\in Y$ distinct from $P$, the intersection $\ol{PQ}\cap Y$ is a proper subset of $\ol{PQ}$. If it is not, then $\ol{PQ}\subset Y$, and by irreducibility, $Y=\ol{PQ}$, which is a linear variety. But by exercise I.7.6, this implies $Y$ has degree one, which is not the case. So $\ol{PQ}$ contains a point not in $Y$, and thus $X$ contains a point not in $Y$.
For the rest of the argument, we will show that if $\dim Y>1$, then we can find a hyperplane $H$ through $P$ so that $Y\cap H$ is smooth at $P$. Then either $Y\cap H$ is irreducible, in which case we will show that $i(Y,H;Y\cap H)=1$ and so we have that $Y\cap H$ is a variety of dimension $r-1$, degree $d$, and $P\in Y\cap H$ is a smooth point, or $Y\cap H$ is reducible, in which case we can find a point in $X\setminus Y$. The combination of these two statements will prove our claim by induction, and the central difficulty will be in showing $i(Y,H;Y\cap H)=1$.
If $\dim Y > 1$, we first note that $r<n$: otherwise $Y=\PP^n$, which is linear, contradicting the fact that $\deg Y>1$. Now, up to a change of coordinates on $\PP^n$ we may assume that $P=[1:0:\cdots:0]$ and work in the affine open $U_0$. Then $Y\cap U_0$ is cut out by an ideal $(g_1,\cdots,g_m)\subset k[x_1,\cdots,x_n]$, and the Jacobian criterion says that $P$ is smooth if the Jacobian matrix at $P$, $\frac{\pd(g_1,\cdots,g_m)}{\pd(x_1,\cdots,x_n)}(P)$, has rank $n-r$. As $r<n$, we can find a vector $(a_1,\cdots,a_n)\in k^n$ so that after appending this to our matrix, we get a matrix of rank $n-r+1$. Letting $f=\sum a_ix_i$, $\ol{f}=\sum a_i X_i$, and $H=V(\ol{f})$ where the $x_i$ are coordinates on $U_0$ and $X_i$ are homogeneous coordinates on $\PP^n$, I claim that $H\cap Y$ is smooth at $P$.
As we can check smoothness in local coordinates, it suffices to verify that $(k[Y\cap U_0]/\sqrt{(f)})_{\mathfrak{m}_P}$ is a regular local ring. Since this is just the quotient of $(k[Y\cap U_0]/(f))_{\mathfrak{m}_P}$ by the nilradical, it suffices to show that $(k[Y\cap U_0]/(f))_{\mathfrak{m}_P}$ is already a regular local ring: this will show that the nilradical is zero because regular local rings are domains, and so $(f)=\sqrt{(f)}$ as ideals of $k[Y\cap U_0]_{\mathfrak{m}_P}$. But we've already done this, more or less by construction of $f$: the Jacobian matrix of $(g_1,\cdots,g_m,f)$ has rank $n-r+1$ by construction, so by theorem I.5.1, the local ring $(k[x_1,\cdots,x_n]/(g_1,\cdots,g_m,f))_{\mathfrak{m}_P}$ is a regular local ring, hence a domain, and thus $(f)_{\mathfrak{m}_P}$ is radical and so this is the local ring of $Y\cap H\cap U_0=V(f)\subset Y\cap U_0$ at the point $P$. So $k[V(f)\subset Y\cap U_0]_{\mathfrak{m}_p}$ is a regular local ring of dimension $r-1$, and thus $V(f)\subset Y\cap U_0$ is smooth at $P$, so $Y\cap V(\ol{f})$ is smooth at $P$. In particular, there is only one irreducible component of $Y\cap H$ through $P$ by exercise I.3.11, the fact that regular local rings are domains, and the correspondence between irreducible components and minimal primes.
Now I claim that this computation implies that $i(Y,H; Z)=1$ where $Z$ is the unique irreducible component of $Y\cap H$ through $P$. To do this, we show that we can compute intersection multiplicity from the data of an affine open via a lemma.
Lemma. Suppose $S$ is a noetherian graded ring and $M$ a graded $S$-module, $\mathfrak{p}$ a minimal homogeneous prime of $M$, and $f\in S_1$ a homogeneous element not in $\mathfrak{p}$. Then the length of $M_{(\mathfrak{p})}$ over $S_{(\mathfrak{p})}$ is the same as the length of $M_\mathfrak{p}$ over $S_\mathfrak{p}$.
Proof. First we show that the graded localization functor $M\mapsto M_{(f)}$ is exact. Given an exact sequence of graded $S$-modules $0\to M'\to M\to M''\to 0$, we have that localization at a homogeneous element $f$ gives an exact sequence of graded modules $0\to M'_f \to M_f \to M''_f\to 0$ because localization is exact and commutes with direct sums: the $d^{th}$ graded piece of $M_f$ is exactly the elements $\frac{m}{f^n}$ where $\deg m =d+n$. As our maps are maps of graded modules and therefore preserve degree, we can restrict to the elements of any fixed degree to get an exact sequence $0\to (M'_f)_d \to (M_f)_d \to (M''_f)_d \to 0$, so we have the claim.
Next, I claim that if $\mathfrak{p}\subset S$ is a graded prime ideal and $f$ is a homogeneous element not in $\mathfrak{p}$, then $M_{(\mathfrak{p})} = (M_{(f)})_{\mathfrak{p}'}$ where $\mathfrak{p}'$ is $(\mathfrak{p}_{(f)})_0$. This is not hard to see: given an element $\frac{x}{g}\in M_{(\mathfrak{p})}$, we can write it as $(xg^{\deg(f) -1}/f^{\deg(x)})/(g^{\deg(f)}/f^{\deg(g)})$ because $\deg(x)=\deg(g)$ and $g^{\deg(f)}/f^{\deg(g)}$ isn't in $\mathfrak{p}'$. Conversely, if we have an element $(x/f^n)/(g/f^m)\in (M_{(f)})_{\mathfrak{p}'}$, we can write it as $(xf^m)/(gf^n)$ and the composition both ways is the identity.
Finally, we see that because the length of $M_\mathfrak{p}$ is the number of times that $S/\mathfrak{p}$ appears in the filtration promised by proposition I.7.4 and the functor $M\mapsto M_{(\mathfrak{p})}$ sends $S/\mathfrak{p}$ to $(S/\mathfrak{p})_{(\mathfrak{p})}\cong S_{(\mathfrak{p})}/\mathfrak{p}_{(\mathfrak{p})}$. So there is a filtration of $M_{(\mathfrak{p})}$ by $S_{(\mathfrak{p})}$-modules so that the number of times $S_{(\mathfrak{p})}/\mathfrak{p}_{(\mathfrak{p})}$ shows up is the length of $M_\mathfrak{p}$ over $S_\mathfrak{p}$. On the other hand, by the usual Jordan-Holder theorem that proposition I.7.4 is the analogue for, the length of $M_{(\mathfrak{p})}$ over $S_{(\mathfrak{p})}$ is exactly this number as well. $\blacksquare$
The upshot of this is that $i(Y,H;Z)$ is equal to the length of $(k[Y\cap U_0]/(f))_{I_{Z\cap U_0}}$ over $k[Y\cap U_0]_{I_{Z\cap U_0}}$. But since $I_{Z\cap U_0}\subset \mathfrak{m}_P$ and localization is transitive, we have that $$(k[Y\cap U_0]/(f))_{I_{Z\cap U_0}}\cong ((k[Y\cap U_0]/(f))_{\mathfrak{m}_P})_{I_{Z\cap U_0}} \cong (k[Z \cap U_0]_{\mathfrak{m}_P})_{I_{Z\cap U_0}}\cong k[Z \cap U_0]_{I_{Z\cap U_0}},$$ which clearly has length one over itself, and thus we've proven that $i(Y,H;Z)=1$. By an application of theorem I.7.7, this shows that if $Y\cap H$ is irreducible, then $Y\cap H$ is a variety of dimension $r-1$ and degree $d$ with a smooth point $P$, fulfilling our promise from earlier.
It remains to check that if $Y\cap H$ is reducible then we can find a point in $X$ not in $Y$. Let $Z_1$ be the unique irreducible component of $Y\cap H$ containing $P$ and let $Z_2$ be an irreducible component of $Y\cap H$ not containing $P$: as $Z_1\cap Z_2$ is a proper closed subset of $Z_1$ and $Z_2$, we can pick a point $Q\in Z_2\setminus Z_1$. Now, $Y\cap \ol{PQ} = (Y\cap H)\cap \ol{PQ}$, and we can write $\ol{PQ}\cap Y$ as a union of the closed sets $\ol{PQ}\cap Z_i$. If $Y\cap \ol{PQ}=\ol{PQ}$, then it must be the case that one of these closed sets $\ol{PQ}\cap Z_i$ is all of $\ol{PQ}$ by irreducibility. But this is not the case: it can't be $Z_j$ for $j>1$, as no $Z_j$ with $j>1$ contains $P$, and $Z_1$ doesn't contain $Q$ by our choice of $Q$. So $Y\cap \ol{PQ}$ is a reducible subset of $\ol{PQ}$, which means it's a proper subset of $\ol{PQ}$ and we're finished.