$\def\PP{\mathbb{P}}\def\ol{\overline}\def\pd{\partial}\def\AA{\mathbb{A}}$
You're right, there's a lot of handwaving when one says "$X$ is birational to the cone on $Y$". (And if that statement can be made rigorous in a simple way, the fact that I'm complaining about it and posting a longer solution will surely provoke someone in to demonstrating this fact.)
Without loss of generality, suppose $P\in D(x_0)$ and consider the rational map $Y\times\PP^1\to X$ given on $(D(x_0)\cap Y)\times\AA^1$ by $(Q,t)\mapsto P+Qt$.
This surjects on to an open dense set of the collection of points $R$ which are on a line of the form $\ol{PQ}$ for $Q\neq P$, of which $X$ is the closure.
As $Y\times\PP^1$ is irreducible, $(D(x_0)\cap Y)\times\AA^1$ is irreducible, the image of $(D(x_0)\cap Y)\times\AA^1$ is irreducible, and the closure of its image is irreducible, so $X$ is irreducible and this rational map from $Y\times \PP^1$ is dominant.
Thus we have an inclusion of function fields $k(X)\hookrightarrow k(Y\times\PP^1)$, so $\dim X\leq \dim Y+1$.
On the other hand, $Y\subset X$, so $\dim X \geq \dim Y$.
All we need to do to show that $\dim X=\dim Y+1$ is to find a point in $X$ not in $Y$.
(Many claimed solutions to this problem say something like 'we can construct a rational inverse' to the map defined above and skip over the existence of such a point - this is problematic, because without this point, one can't actually define such an inverse!)
We will prove the claim by induction on $\dim Y$.
If $\dim Y=1$, then I claim that for any $Q\in Y$ distinct from $P$, the intersection $\ol{PQ}\cap Y$ is a proper subset of $\ol{PQ}$.
If it is not, then $\ol{PQ}\subset Y$, and by irreducibility, $Y=\ol{PQ}$, which is a linear variety.
But by exercise I.7.6, this implies $Y$ has degree one, which is not the case.
So $\ol{PQ}$ contains a point not in $Y$, and thus $X$ contains a point not in $Y$.
For the rest of the argument, we will show that if $\dim Y>1$, then we can find a hyperplane $H$ through $P$ so that $Y\cap H$ is smooth at $P$.
Then either $Y\cap H$ is irreducible, in which case we will show that $i(Y,H;Y\cap H)=1$ and so we have that $Y\cap H$ is a variety of dimension $r-1$, degree $d$, and $P\in Y\cap H$ is a smooth point, or $Y\cap H$ is reducible, in which case we can find a point in $X\setminus Y$.
The combination of these two statements will prove our claim by induction, and the central difficulty will be in showing $i(Y,H;Y\cap H)=1$.
If $\dim Y > 1$, we first note that $r<n$: otherwise $Y=\PP^n$, which is linear, contradicting the fact that $\deg Y>1$.
Now, up to a change of coordinates on $\PP^n$ we may assume that $P=[1:0:\cdots:0]$ and work in the affine open $U_0$.
Then $Y\cap U_0$ is cut out by an ideal $(g_1,\cdots,g_m)\subset k[x_1,\cdots,x_n]$, and the Jacobian criterion says that $P$ is smooth if the Jacobian matrix at $P$, $\frac{\pd(g_1,\cdots,g_m)}{\pd(x_1,\cdots,x_n)}(P)$, has rank $n-r$.
As $r<n$, we can find a vector $(a_1,\cdots,a_n)\in k^n$ so that after appending this to our matrix, we get a matrix of rank $n-r+1$.
Letting $f=\sum a_ix_i$, $\ol{f}=\sum a_i X_i$, and $H=V(\ol{f})$ where the $x_i$ are coordinates on $U_0$ and $X_i$ are homogeneous coordinates on $\PP^n$, I claim that $H\cap Y$ is smooth at $P$.
As we can check smoothness in local coordinates, it suffices to verify that $(k[Y\cap U_0]/\sqrt{(f)})_{\mathfrak{m}_P}$ is a regular local ring.
Since this is just the quotient of $(k[Y\cap U_0]/(f))_{\mathfrak{m}_P}$ by the nilradical, it suffices to show that $(k[Y\cap U_0]/(f))_{\mathfrak{m}_P}$ is already a regular local ring: this will show that the nilradical is zero because regular local rings are domains, and so $(f)=\sqrt{(f)}$ as ideals of $k[Y\cap U_0]_{\mathfrak{m}_P}$.
But we've already done this, more or less by construction of $f$: the Jacobian matrix of $(g_1,\cdots,g_m,f)$ has rank $n-r+1$ by construction, so by theorem I.5.1, the local ring $(k[x_1,\cdots,x_n]/(g_1,\cdots,g_m,f))_{\mathfrak{m}_P}$ is a regular local ring, hence a domain, and thus $(f)_{\mathfrak{m}_P}$ is radical and so this is the local ring of $Y\cap H\cap U_0=V(f)\subset Y\cap U_0$ at the point $P$.
So $k[V(f)\subset Y\cap U_0]_{\mathfrak{m}_p}$ is a regular local ring of dimension $r-1$, and thus $V(f)\subset Y\cap U_0$ is smooth at $P$, so $Y\cap V(\ol{f})$ is smooth at $P$.
In particular, there is only one irreducible component of $Y\cap H$ through $P$ by exercise I.3.11, the fact that regular local rings are domains, and the correspondence between irreducible components and minimal primes.
Now I claim that this computation implies that $i(Y,H; Z)=1$ where $Z$ is the unique irreducible component of $Y\cap H$ through $P$.
To do this, we show that we can compute intersection multiplicity from the data of an affine open via a lemma.
Lemma. Suppose $S$ is a noetherian graded ring and $M$ a graded $S$-module, $\mathfrak{p}$ a minimal homogeneous prime of $M$, and $f\in S_1$ a homogeneous element not in $\mathfrak{p}$.
Then the length of $M_{(\mathfrak{p})}$ over $S_{(\mathfrak{p})}$ is the same as the length of $M_\mathfrak{p}$ over $S_\mathfrak{p}$.
Proof. First we show that the graded localization functor $M\mapsto M_{(f)}$ is exact.
Given an exact sequence of graded $S$-modules $0\to M'\to M\to M''\to 0$, we have that localization at a homogeneous element $f$ gives an exact sequence of graded modules $0\to M'_f \to M_f \to M''_f\to 0$ because localization is exact and commutes with direct sums: the $d^{th}$ graded piece of $M_f$ is exactly the elements $\frac{m}{f^n}$ where $\deg m =d+n$.
As our maps are maps of graded modules and therefore preserve degree, we can restrict to the elements of any fixed degree to get an exact sequence $0\to (M'_f)_d \to (M_f)_d \to (M''_f)_d \to 0$, so we have the claim.
Next, I claim that if $\mathfrak{p}\subset S$ is a graded prime ideal and $f$ is a homogeneous element not in $\mathfrak{p}$, then $M_{(\mathfrak{p})} = (M_{(f)})_{\mathfrak{p}'}$ where $\mathfrak{p}'$ is $(\mathfrak{p}_{(f)})_0$.
This is not hard to see: given an element $\frac{x}{g}\in M_{(\mathfrak{p})}$, we can write it as $(xg^{\deg(f) -1}/f^{\deg(x)})/(g^{\deg(f)}/f^{\deg(g)})$ because $\deg(x)=\deg(g)$ and $g^{\deg(f)}/f^{\deg(g)}$ isn't in $\mathfrak{p}'$.
Conversely, if we have an element $(x/f^n)/(g/f^m)\in (M_{(f)})_{\mathfrak{p}'}$, we can write it as $(xf^m)/(gf^n)$ and the composition both ways is the identity.
Finally, we see that because the length of $M_\mathfrak{p}$ is the number of times that $S/\mathfrak{p}$ appears in the filtration promised by proposition I.7.4 and the functor $M\mapsto M_{(\mathfrak{p})}$ sends $S/\mathfrak{p}$ to $(S/\mathfrak{p})_{(\mathfrak{p})}\cong S_{(\mathfrak{p})}/\mathfrak{p}_{(\mathfrak{p})}$.
So there is a filtration of $M_{(\mathfrak{p})}$ by $S_{(\mathfrak{p})}$-modules so that the number of times $S_{(\mathfrak{p})}/\mathfrak{p}_{(\mathfrak{p})}$ shows up is the length of $M_\mathfrak{p}$ over $S_\mathfrak{p}$.
On the other hand, by the usual Jordan-Holder theorem that proposition I.7.4 is the analogue for, the length of $M_{(\mathfrak{p})}$ over $S_{(\mathfrak{p})}$ is exactly this number as well. $\blacksquare$
The upshot of this is that $i(Y,H;Z)$ is equal to the length of $(k[Y\cap U_0]/(f))_{I_{Z\cap U_0}}$ over $k[Y\cap U_0]_{I_{Z\cap U_0}}$.
But since $I_{Z\cap U_0}\subset \mathfrak{m}_P$ and localization is transitive, we have that $$(k[Y\cap U_0]/(f))_{I_{Z\cap U_0}}\cong ((k[Y\cap U_0]/(f))_{\mathfrak{m}_P})_{I_{Z\cap U_0}} \cong (k[Z \cap U_0]_{\mathfrak{m}_P})_{I_{Z\cap U_0}}\cong k[Z \cap U_0]_{I_{Z\cap U_0}},$$ which clearly has length one over itself, and thus we've proven that $i(Y,H;Z)=1$.
By an application of theorem I.7.7, this shows that if $Y\cap H$ is irreducible, then $Y\cap H$ is a variety of dimension $r-1$ and degree $d$ with a smooth point $P$, fulfilling our promise from earlier.
It remains to check that if $Y\cap H$ is reducible then we can find a point in $X$ not in $Y$.
Let $Z_1$ be the unique irreducible component of $Y\cap H$ containing $P$ and let $Z_2$ be an irreducible component of $Y\cap H$ not containing $P$: as $Z_1\cap Z_2$ is a proper closed subset of $Z_1$ and $Z_2$, we can pick a point $Q\in Z_2\setminus Z_1$.
Now, $Y\cap \ol{PQ} = (Y\cap H)\cap \ol{PQ}$, and we can write $\ol{PQ}\cap Y$ as a union of the closed sets $\ol{PQ}\cap Z_i$.
If $Y\cap \ol{PQ}=\ol{PQ}$, then it must be the case that one of these closed sets $\ol{PQ}\cap Z_i$ is all of $\ol{PQ}$ by irreducibility.
But this is not the case: it can't be $Z_j$ for $j>1$, as no $Z_j$ with $j>1$ contains $P$, and $Z_1$ doesn't contain $Q$ by our choice of $Q$.
So $Y\cap \ol{PQ}$ is a reducible subset of $\ol{PQ}$, which means it's a proper subset of $\ol{PQ}$ and we're finished.
Best Answer
I think there's an easier way to show this. I'll present a hint and then detail how to make it work under some spoiler text so you can try for yourself before looking at the answer.
Hint: Consider the two lines $V(x)$ and $V(y)$ in $\Bbb P^2$. What does the projection map do to each of these lines? What can you infer about the possible extensions of this map to the point $[0:0:1]$ via these extensions?