Let $X$ be an integral scheme and let $Y$ be a non-empty open subscheme of $X$. $X$ is irreducible, so $Y$ is dense in $X$. I claim the induced homomorphism $i^\flat : \mathscr{O}_X \to i_* \mathscr{O}_Y$ is monic but not necessarily epic. Indeed, the claim is local on $X$ and $Y$, so we may take $X$ to be affine and $Y$ to be a distinguished open subscheme of $X$. But then all we have is a localisation of an integral domain, and this is always injective but not necessarily surjective.
The point is that there isn't one notion of subscheme which gives rise to open and closed subschemes; rather, there are two.
Open subschemes are a special case of the following construction: if $X$ is a locally ringed space and $Y$ is any subset of $X$, we can make $Y$ into a locally ringed space by pulling back $\mathscr{O}_X$ along the inclusion $i : Y \hookrightarrow X$. Unfortunately, there's no guarantee that $(Y, i^{-1} \mathscr{O}_X)$ is a scheme even if $(X, \mathscr{O}_X)$ is.
On the other hand, the definition of a closed subscheme depends a lot on the scheme structure. If $Y$ is a closed subset of $X$, that means $Y \cap U$ is closed in $U$ for every open subset $U$ – but we know that closed subsets of $\operatorname{Spec} A$ are homeomorphic to $\operatorname{Spec} A / \mathfrak{a}$ for some suitable ideal $\mathfrak{a}$, and in essence the structure of $Y$ as a closed subscheme of $X$ is defined so that we have an exact sequence
$$0 \longrightarrow i^{-1} \mathscr{I}_{Y \mid X} \longrightarrow i^{-1} \mathscr{O}_X \longrightarrow \mathscr{O}_Y \longrightarrow 0$$
or equivalently, so that we have the "fundamental exact sequence":
$$0 \longrightarrow \mathscr{I}_{Y \mid X} \longrightarrow \mathscr{O}_X \longrightarrow i_* \mathscr{O}_Y \longrightarrow 0$$
But the story for varieties is more subtle. For the purposes of this discussion, I mean "variety" in the sense of a reduced scheme of finite type over an algebraically closed field $k$. Because varieties have enough closed points, the structure sheaf of a variety $X$ is isomorphic to a subsheaf of the sheaf of continuous functions $X(k) \to \mathbb{A}^1(k)$. (Henceforth, I will pretend non-closed points don't exist.) Thus, there is a canonical way of restricting regular functions on (any open subset of) $X$ to any (not necessarily open or closed!) subset $Y$ of $X$. If $Y$ is open, this recovers the open subscheme structure, and if $Y$ is closed, this recovers the closed subscheme structure.
Let's look at this more closely. We define $\mathscr{I}_{Y \mid X}$ to be the subsheaf of $\mathscr{O}_X$ consisting of those regular functions which vanish on $Y$, i.e.
$$\mathscr{I}_{Y \mid X} (U) = \{ f \in \mathscr{O}_X : \forall y \in Y . \, f (y) = 0 \}$$
and we define, for each open subset $V$ of $Y$,
$$\mathscr{O}_Y (V) = \varinjlim_{U \supseteq V} \mathscr{O}_X (U) / \mathscr{I}_{Y \mid X} (U)$$
This is a sheaf because $Y$ is quasicompact. (Every subset of $X$ is quasicompact!) If $Y$ is open, then $V$ is open in $X$, so we are taking the direct limit over a directed system with a terminal object – hence $\mathscr{O}_Y (V) \cong \mathscr{O}_X (V) / \mathscr{I}_{Y \mid X} (V) \cong \mathscr{O}_X (V)$ in this case.
For $Y$ closed something weird happens as well. Let $U$ be an open affine subset of $X$. Then, $V = U \cap Y$ is a closed subset of $U$ and an open subset of $Y$. Suppose $f \in \mathscr{O}_X (U)$ does not vanish on $V$. Then, the Nullstellensatz implies $f$ is already invertible in $\mathscr{O}_X (U) / \mathscr{I}_{Y \mid X} (U)$ – and this implies that the directed system is constant! In particular, we get $i_* \mathscr{O}_Y \cong \mathscr{O}_X / \mathscr{I}_{Y \mid X}$.
In general, however, we don't get anything nice. There is a natural left exact sequence of groups
$$0 \longrightarrow \mathscr{I}_{Y \mid X} (U) \longrightarrow \mathscr{O}_X (U) \longrightarrow \mathscr{O}_Y (U \cap Y)$$
and therefore a left exact sequence of sheaves on $X$:
$$0 \longrightarrow \mathscr{I}_{Y \mid X} \longrightarrow \mathscr{O}_X \longrightarrow i_* \mathscr{O}_Y$$
We have just seen that this extends to a short exact sequence when $Y$ is closed. When $Y$ is open and $X$ is irreducible, the homomorphism $\mathscr{O}_X \to i_* \mathscr{O}_Y$ is monic but in general not epic. (Consider a point $x \in X \setminus Y$: the stalk of $i_* \mathscr{O}_Y$ at $x$ gives the fraction field of the local ring $\mathscr{O}_{X, x}$.)
Best Answer
This proof is not true, and the following counterexample is good to keep in mind:
Let $X$ be two copies of $\Bbb A^1$ and let $Y$ be $V(xy)\subset \Bbb A^2$, with the map $f$ acting as $t\mapsto (0,t)$ and $u\mapsto (u,0)$ where $t,u$ are coordinates on the different copies of $\Bbb A^1$. Then certainly removing the origins of both lines in $X$ and the crossing point in $Y$ gives an isomorphism, so this is a birational map. But one can see immediately that the stalk of $f_*\mathcal{O}_X$ is rank 2 at the crossing point of $Y$.
To get at the particular point where your proof breaks, consider the map $g:X\to \Bbb A^1$, the map taking the value $1$ on one line and $0$ on the other. Following your procedure, you obtain a function $f$ on $Y$ without the crossing point which is $0$ on one axis and $1$ on the other - this cannot be extended to a function on the whole variety. So your claim here is false.
The essential difficulty here is that sections of a line bundle on a normal variety can be extended over a codimension two set, but not necessarily a codimension one set. For your claim to be true, you'll need to enforce exactly the conditions of Zariski's Main Theorem.
I should also mention that the question of$X$ is smooth and $f:X\to Y$ is birational implying $f_*\mathcal{O}_X \cong \mathcal{O}_Y$ is awfully close to the condition of rational singularities. If you end up studying more algebraic geometry, this class is a particularly well-behaved bunch of singularities.