If the map from $X$ to the maxspec of $\mathcal O(X)$ is a bijection, then $X$ is indeed affine.
Here is an argument:
By assumption $X \to $ maxspec $\mathcal O(X)$ is bijective, thus quasi-finite,
and so by (Grothendieck's form of) Zariski's main theorem, this map factors as an open embedding of $X$ into a variety that is finite over maxspec $\mathcal O(X)$.
Any variety finite over an affine variety is again affine, and hence $X$ is an open subset of an affine variety, i.e. quasi-affine. So we are reduced to considering the case when $X$ is quasi-affine, which is well-known and straightforward.
(I'm not sure that the full strength of ZMT is needed, but it is a natural tool
to exploit to get mileage out of the assumption of a morphism having finite fibres, which is what your bijectivity hypothesis gives.)
In fact, the argument shows something stronger: suppose that we just assume
that the morphism $X \to $ maxspec $\mathcal O(X)$ has finite non-empty fibres,
i.e. is quasi-finite and surjective.
Then the same argument with ZMT shows that $X$ is quasi-affine. But it is standard that the map $X \to $ maxspec $\mathcal O(X)$ is an open immersion when $X$ is quasi-affine,
and since by assumption it is surjecive, it is an isomorphism.
Note that if we omit one of the hypotheses of surjectivity or quasi-finiteness, we can find a non-affine $X$ satisfying the other hypothesis.
E.g. if $X = \mathbb A^2 \setminus \{0\}$ (the basic example of a quasi-affine,
but non-affine, variety), then maxspec $\mathcal O(X) = \mathbb A^2$, and the open immersion $X \to \mathbb A^2$ is evidently not surjective.
E.g. if $X = \mathbb A^2$ blown up at $0$, then maxspec $\mathcal O(X) =
\mathbb A^2$, and $X \to \mathbb A^2$ is surjective, but has an infinite fibre
over $0$.
Caveat/correction: I should add the following caveat, namely that it is not always true, for a variety $X$ over a field $k$, that $\mathcal O(X)$ is finitely generated over $k$, in which case maxspec may not be such a good construction to apply, and the above argument may not go through. So in order to conclude that $X$ is affine, one should first insist that $\mathcal O(X)$ is finitely generated over $k$, and then that futhermore the natural map $X \to $ maxspec $\mathcal O(X)$ is quasi-finite and surjective.
(Of course, one could work more generally with arbitrary schemes and Spec rather than
maxspec, but I haven't thought about this general setting: in particular, ZMT requires some finiteness hypotheses, and I haven't thought about what conditions might guarantee that the map $X \to $ Spec $\mathcal O(X)$ satisfies them.)
Incidentally, for an example of a quasi-projective variety with non-finitely generated ring of regular functions, see this note of Ravi Vakil's
Simply put, the evolution of the foundational definitions in Algebraic Geometry was driven not by the search of new objects, but of a better understanding of classical objects. In fact, the crowning achievement of Grothendieck's school was precisely to rigorously state and prove a myriad of classical problems. The exemplary case are the Weil conjectures. Weil himself had a dab at the foundations of Algebraic Geometry so as to justify his work on the conjectures. Eventually the problem was closed first by Grothendieck's reinterpretation in terms of étale cohomology, and Deligne's proof of that version.
To illustrate the point just made, let me sketch a (hi)story of the definitions:
The starting point is, of course, the study of the zero set of some polynomials. This goes back to Descartes, and his discovery (invention?) of analytic geometry. Now, as time goes on, it is clear that this works well not for real polynomials, but for complex ones. More generally, given an algebraically closed field $k=\overline{k}$, you have a tight relationship between the algebra of the polynomial ring $R:=k[x_1,...,x_n]$ and the geometry of the zero sets in $\mathbb{A}^n_k$ (Hilbert's Nullstellensatz). In particular, these zero sets are found to be in correspondence to radical ideals. In a first iteration, then, an (affine) variety is the zero locus of a radical ideal $I=\sqrt{I}$. A few addendums are in order: (a) one quickly notes that the algebra of the coordinate ring $R/I$ distinguishes between irreducible components determined by minimal primes (Scheinnullstellensatz). Just as topology naturally focuses on connected spaces, one can focus on irreducible varieties; one will then speak of 'algebraic set' for radical ideals, and 'varieties' for prime ones. (b) a natural variation is to consider also open subsets of varieties, the quasi-affine varieties (c) with the discovery projective space, all this discussion can be reproduced for homogeneous ideals and (quasi)projective verieties. A very good reference for this very classical point of view is the first chapter of Hartshorne's Algebraic Geometry.
The original fauna is, then, one unified more by a heuristic and formal similarity rather than a unified conceptual framework. Quasi-affine varities are not affine, and quasi-projective is not projective. (The counterexample for both negative statements is just the affine plane without the origin.) It is true that (quasi)affine is quasi-projective, but there are great advantages is considering the affine structure of a variety. One way to unify these concepts is by defining a variety as the glueing of (a finite number of) affine varieties. It is, of course, a strictly more general definition, but this was all for the better: the Jacobian variety wasn't originally known to be projective, but it did have affine charts. (In fact, I have the vague notion that Weil introduced this definition precisely because of the Jacobian variety.) A complication deriving from this general methodology is the possibility of glueing things 'wrong', the affine line with double origin being the standard example. To prevent this, we add a separation axiom that is formally the same as the Hausdorff axiom: the diagonal $X\hookrightarrow X\times X$ is closed. (Take a pause to ponder why this does not imply that a variety is Hausdorff.)
As things progress, it becomes clear that the local rings should be part of the data of a variety. Varieties are defined as a topological set of zeros, but there is a gulf between what one thinks of a morphism of varieties, and a general continuous morphism; local rings allow us to make precise the distinction is a way that is less awkward than the more classical definition (see Hartshorne Chapter I Exs.3.2 and 3.3, for example). The difficulty here is how to organize to organize all the data. Thankfully, sheaf theory was there to help and lead Serre to his definition: a variety is a locally ringed space that is locally isomorphic to an affine variety (as a locally ringed space). The separation axiom is of course still imposed.
One could draw a direct line from here to schemes, and in fact most people do that. Whereas the connection is indisputable, there really is a piece of the puzzle missing. All that we said above applies to algebraically closed fields. But if we try to extend any of it to more general fields, thing go awry pretty fast. The standard example is the ideal generated by $x^2+1$ in $\mathbb{R}[x]$, which doesn't correspond to any point in the real line, despite the fact that it generates a prime (indeed maximal) ideal. More generally, any of the polynomials $x^n+x^{n-1}+...+1$ for $n$ even are irreducible, but still don't correspond to any real points. They do correspond, of course, to the complex $n$th-roots of unity. Essentially, for non-algebraically closed fields, 'varieties' are somehow determined by points in larger fields. But mind you that in the example of $n$th roots we deal only with algebraic extensions. In general, if we want to persist in the idea of fixing a large field $K$ over which to o geometry over $k$, we need to go beyond the algebraic closure into some extension of infinite trascendence degree. Weil worked precisely along these lines in his Foundations of Algebraic Geometry. An alternative is not to fix any such (incredibly) large field, but rather to try to keep track of the points over an arbitrary field. It may sound dizzying to do that, but there was at hand a means to do that just at the right moment: we can see the variety as determining a functor from the category of fields to the category of sets, assigning to each fields the points of the variety defined over it.
We finally get to Grothendieck and his school. It turns out that the previous two points are in close connection with each other, especially when we realize that we can keep track of points of a variety over an arbitrary $k$-algebra. The connection here are schemes, which on one hand are locally ringed spaces locally isomorphic to a spectrum of a ring; but on the other, are representatives of functors which obey a descent condition, and admit an atlas. Under this definition, we not only unify the whole set of disparate examples (points 1 and 2), making precise the difference with topology (point 3), but include even the case of non-algebraically closed fields (in fact, any base ring!). The last point is an important one: the scheme $X_\mathbb{R}$ corresponding to $x^2+1$ over $\mathbb{R}$ has one (closed) point, and the local ring of that point has residue field $\mathbb{C}$, precisely the field over which the polynomial splits. Over $\mathbb{C}$, of course, $X_\mathbb{C}$ has two closed points, with residue fields $\mathbb{C}$. There is a map $X_\mathbb{C}\to X_\mathbb{R}$ that is a Galois cover. On the other hand, if $X_\mathbb{R}$ had any points defined over $\mathbb{R}$, they'd show up as fixed points of the Galois action in $X_\mathbb{C}$ (if we'd taken $x^3-1$, for example, there would be in both $X_\mathbb{R}$ and $X_\mathbb{C}$ an extra point corresponding to $1$). In this very concrete way, the scheme keeps track of all points over all fields, while still distinguishing which come from where. I am belaboring the point because it is perhaps the most elementary, but still powerful illustration of how the power of schemes is not so much in enlarging our set of objects, but in clarifying the framework for classical ones. The elegance of the framework only grows as we dig deeper into the subject, with its concise definitions and simpler proofs (compare completeness to projectivity, for example).
That said, speaking of schemes does enlarge our fauna, and by a lot. One obvious examples are non-reduced schemes. But there's much to say on the use of even those non-classical schemes as tools in the study of classical ones. The canonical example: $k[\epsilon]/(\epsilon^2)$ allows us to define (Zarisky) tangent spaces; more generally, Artinian rings are the foundation of deformation theory. On the other hand, these new animals do show up naturally (as some moduli spaces, for example). One important side note here is the existence of primary decomposition, which expresses a scheme as a union of integral schemes and some embedded components with only nilpotents.
After all this, it is fair to ask whither are varieties gone. Elegant and useful as the framework may be, we seem to have lost track of them. This gives us our final definition (so-far?): a variety if a finite-type integral scheme over an (arbitrary) field. Indeed, there is a functor embedding the category of varieties in the category of schemes which makes this precise. In the algebraically closed field, this is in Hartshorne's book; for more general fields, it is in Demazure-Gabriel's.
I seem to have only said standard and well-known facts, so I am not sure that I addressed your question very well. And of course, this should not be taken as a historically accurate description. But just as writing it down helped me put some order in my head, I hope it helps you in some measure.
Best Answer
Yes, there are many examples of proper (thus separated) varieties which are not projective and therefore not quasi-projective. Here's a list of some examples: