If the map from $X$ to the maxspec of $\mathcal O(X)$ is a bijection, then $X$ is indeed affine.
Here is an argument:
By assumption $X \to $ maxspec $\mathcal O(X)$ is bijective, thus quasi-finite,
and so by (Grothendieck's form of) Zariski's main theorem, this map factors as an open embedding of $X$ into a variety that is finite over maxspec $\mathcal O(X)$.
Any variety finite over an affine variety is again affine, and hence $X$ is an open subset of an affine variety, i.e. quasi-affine. So we are reduced to considering the case when $X$ is quasi-affine, which is well-known and straightforward.
(I'm not sure that the full strength of ZMT is needed, but it is a natural tool
to exploit to get mileage out of the assumption of a morphism having finite fibres, which is what your bijectivity hypothesis gives.)
In fact, the argument shows something stronger: suppose that we just assume
that the morphism $X \to $ maxspec $\mathcal O(X)$ has finite non-empty fibres,
i.e. is quasi-finite and surjective.
Then the same argument with ZMT shows that $X$ is quasi-affine. But it is standard that the map $X \to $ maxspec $\mathcal O(X)$ is an open immersion when $X$ is quasi-affine,
and since by assumption it is surjecive, it is an isomorphism.
Note that if we omit one of the hypotheses of surjectivity or quasi-finiteness, we can find a non-affine $X$ satisfying the other hypothesis.
E.g. if $X = \mathbb A^2 \setminus \{0\}$ (the basic example of a quasi-affine,
but non-affine, variety), then maxspec $\mathcal O(X) = \mathbb A^2$, and the open immersion $X \to \mathbb A^2$ is evidently not surjective.
E.g. if $X = \mathbb A^2$ blown up at $0$, then maxspec $\mathcal O(X) =
\mathbb A^2$, and $X \to \mathbb A^2$ is surjective, but has an infinite fibre
over $0$.
Caveat/correction: I should add the following caveat, namely that it is not always true, for a variety $X$ over a field $k$, that $\mathcal O(X)$ is finitely generated over $k$, in which case maxspec may not be such a good construction to apply, and the above argument may not go through. So in order to conclude that $X$ is affine, one should first insist that $\mathcal O(X)$ is finitely generated over $k$, and then that futhermore the natural map $X \to $ maxspec $\mathcal O(X)$ is quasi-finite and surjective.
(Of course, one could work more generally with arbitrary schemes and Spec rather than
maxspec, but I haven't thought about this general setting: in particular, ZMT requires some finiteness hypotheses, and I haven't thought about what conditions might guarantee that the map $X \to $ Spec $\mathcal O(X)$ satisfies them.)
Incidentally, for an example of a quasi-projective variety with non-finitely generated ring of regular functions, see this note of Ravi Vakil's
Best Answer
In principle, the $O_Y$ attached to $Y$ depends on the closed set $X$: indeed, given a locally closed subset $Y$, we know that there is a closed subset $X$ in $P^n$ such that $Y$ is open in $X$, and we define $O_Y$ as the restriction of $Y$ of $O_X$.
Now, there are in general many closed subsets of $P^n$ of which $Y$ is an open subset, and the whole point of the remark is that $O_Y$ does not depend on the choice of $X$: if $X'$ is another closed subset of $P^n$ such that $Y$ is an open subset of $X'$, the restriction $O_{X'}|_Y$ is isomorphic to $O_X|_Y$ as sheaves of rings on $Y$.
You should try to prove this yourself, even if it takes a long time. It is a great exercise in handling the definitions of everything.