Here is an outline. I'll just assume $Y$ is affine to make the problem easier.
We'll have to use the second part of the lemma here. This says:
$$
A(Y)_{\mathfrak{m}_P} = \bigcap_{\mathfrak q \text{ height 1 prime of $A(Y)_{\mathfrak m_P}$}} (A(Y)_{\mathfrak{m}_P})_{\mathfrak{q}}
= \bigcap_{\mathfrak q \text{ height 1 prime of $A(Y)$ contained in $\mathfrak m_P$}} A(Y)_{\mathfrak q}
$$
Note all these calculations are occuring in $K(Y)$, the field of rational functions. Our given $f$ is a rational function so must be of the form $g/h$, where $g$ and $h$ lie in $A(Y)$ and $h \neq 0$. It remains to show that $h$ is not in any height 1 prime $\mathfrak q \subset \mathfrak m_P$, because then $f$ will lie in $A(Y)_{\mathfrak m_P}$ so $f$ will be regular at $P$. If $h$ were in such a prime $\mathfrak q$, then $h$ would vanish on $Z(\mathfrak q)$. We can find a point $Q \neq P$ in $Z(\mathfrak q)$ because $\dim Y \geq 2$ (basically $\mathfrak q \subsetneq \mathfrak m_P$ by Hartshorne Thm 3.2c). But $f$ is regular at $Q$ so $h$ cannot vanish at $Q$. So $h \notin \mathfrak q$, as desired.
This is a response to Mingfeng's question below.
We have a ring isomorphism from the field of fractions of $A(Y)$ to the function field $K(Y)$ (note by Hartshorne's definition, this function field is really equivalence classes of pairs of regular functions defined on some nonempty open set). This map sends a fraction $f/g$ to the class of the pair ($f/g$, $\{ g \neq 0 \}$). This map is injective because this is a nonzero map out of a field. To see surjectivity, take a rational function, and take a representative, which is a regular function $h$ defined on an open set $U$. By definition, regular functions locally look like $f/g$. So shrink $U$ so we have the representation $f/g$. Then map the fraction $f/g$ to the pair ($f/g$, $\{ g \neq 0 \}$). This pair is equivalent to $(h, U)$. Hence this map is an isomorphism.
Notice that $A(Y)_{m_p} \subset frac (A(Y))$, and $\mathcal O_p \subset K(Y)$. I am thinking of $\mathcal O_p$ as on page 16 in Hartshorne. Then, the above map restricts to an isomorphism $A(Y)_{m_p} \to \mathcal O_p$.
Now back to our problem. $f$ is defined on $Y-0$, so I can think of $f$ as a rational function. Under the isomorphism, suppose $f$ corresponds to $g/h$. My goal is to show $g/h$ actually lies in $A(Y)_{m_p}$, because then $f$ will actually lie in $\mathcal O_p$, so $f$ is actually regular at $p$.
We can see $h(Q) \neq 0$ as follows. Since $f$ is regular at $Q$, $f \in \mathcal O_Q$. This means $g/h$ lies in $A(Y)_{m_Q}$. So $h \notin m_Q$, so $h(Q) \neq 0$.
Hartshorne's claim is true when $Y$ is projective. The key fact is that every point in a variety has an affine open neighborhood (see proposition I.4.3, for instance) - combining this with the fact that the local ring of a point can be calculated from any open neighborhood, we have that the result for the affine case carries over to the projective case. (The key error in your calculation is that you haven't justified why you should be able to replace $A(Y)$ with $S(Y)$ and have everything else work.)
Best Answer
Since the problem is local around $p$, you can assume that $Y=Spec(A)$ where $A$ is a noetherian domain (quasi-projectiveness is irrelevant).
Clearly, every point $\mathfrak q \in Spec(A)$ of height one is distinct from $p$ (since $\mathfrak q$ it corresponds to a subvariety of codimension $1$). So every function $f$ defined on $Spec(A)\setminus \lbrace p\rbrace $ is defined at $\mathfrak q $.
You can then conclude that $f\in A$, that is $f$ extends regularly through $p$, thanks to the formula valid for a noetherian normal domain (Matsumura, Commutative ring theory, Theorem 11.5, page 81)
$$ A=\bigcap_{ht(\mathfrak q)=1} A_ \mathfrak q $$
A general result in this vein is that if $X$ is a locally noetherian normal integral scheme and $Y\subset X$ a closed subset of codimension $\geq 2$, the restriction morphism $\mathcal O_X(X)\to O_X(X\setminus F)$ is bijective.