One way to think of a manifold is as a family of of open subsets $U_i \subset \mathbb{R}^n$, together with distinguished subsets $V_{ij} \subset U_i$ and isomorphisms $\psi_{ij}: V_{ij} \to V_{ji}$ that satisfy the cocycle condition. This may not be useful practically, but occasionally it might be an intuitive crutch. Now, just as manifolds are obtained by gluing together subsets of $\mathbb{R}^n$, a scheme is obtained by gluing together affines. In other words, we have open affines $U_i = \mathrm{Spec} A_i$ for suitable rings $A_i$, open subsets $V_{ij} \subset U_i$, and isomorphisms $\psi_{ij}$ (of locally ringed spaces) as before.
However, the open subsets $V_{ij}$ need not be themselves affine.
Question: Is it possible to formulate this definition such that the sets $V_{ij}$ are affine? I know this can be done if the scheme is separated (because the intersection of open affines is affine).
One of the nice things about this is that wouldn't have to worry about the isomorphisms $\psi_{ij}$ being isomorphisms of locally ringed spaces, just isomorphisms of the corresponding rings.
This is probably a bad way of thinking of schemes in general; the only reason I was interested in it was because then the fibered product could perhaps be thought of more "explicitly."
The following (related) question also occurred to me when I was thinking about this.
Question': Is there an easy way to tell when the complement of $V(\mathfrak{a}) \subset \mathrm{Spec} A$ is affine? Of course, this is true when $\mathfrak{a}$ is principal. (Answered: see the comments of Matthew Emerton and David Speyer.)
Best Answer
You can, if you use a slightly more general notion of gluing. (The notion of gluing you present is "wrong", or at least simplistic, in roughly the same way that it is "wrong" to require that a basis for a topology be closed under intersections. E.g., if you do this, then the set of open balls in $\mathbb{R}^n$ for $n > 1$ does not form a "basis.")
Let $X$ be a scheme. Consider the diagram whose objects are open affine subschemes of $X$, and whose morphisms are inclusions $U \hookrightarrow V$ such that $U$ is a distinguished open subset of $V$. Whenever $U$ and $V$ are two objects and $x \in U \cap V$, there exists an object $W \subset U \cap V$ such that $x \in W$ and $W \hookrightarrow U$, $W \hookrightarrow V$ are both morphisms: Since the distinguished open subsets of $U$ form a basis for the topology, there is a distinguished open $W'$ in $U$ such that $x \in W' \subset U \cap V$. Similarly, there is a section $f$ over $V$ such that $x \in V_f \subset W'$. But then $V_f = W'_f$ is a distinguished open subset of both $U$ and $V$, so we let $W = W'_f$.
It is also not too hard to show that whenever we have a category as above, we can glue things together to form a scheme (i.e., the diagram has a unique colimit in the category of schemes). If someone asks for a precise statement of this, I'll try to cook one up, but it's not particularly nice. (Not quite horrendous, but not very nice either.)
In particular, the fiber product is obtained by gluing together schemes of the form $\mathrm{Spec} A \otimes_C B$, where $\mathrm{Spec} C$ contains the images of both $\mathrm{Spec} A$ and $\mathrm{Spec} B$, with "overlap inclusions" specified by morphisms $A \otimes_C B \to A_f \otimes_C B_g$. An important note here: if $C \to D$ is a ring epimorphism (e.g., corresponds to an open immersion), and $A, B$ are $D$-algebras, then $A \otimes_C B$ is naturally isomorphic to $A \otimes_D B$.