What is a rigorous way to prove this graph is non-planar? I have some vague memories of setting the edges within the boundary of the graph as vertices and then use some adjacency rule to check to see if it's bipartite (or something like that to check for planarity). Could someone tell me how to do it properly?
[Math] a rigorous way to prove that this graph is non-planar
graph theoryplanar-graphs
Related Solutions
Let's look at bicubic planar graphs. As you say, there must be $2n$ vertices, $3n$ edges, and $n+2$ faces for some $n$. Suppose there are $F_j$ faces of length $j$; then we want lower and upper bounds on $F_4$ in terms of $n$.
Note that every 4-cycle in the graph must be a face in the planar embedding, because any chord in the 4-cycle would give you cycles of length 3, impossible in a bipartite graph. The graph you have at the end of the question is not a counterexample because it is not cubic (in a 3-regular graph, there would be two triangles on the bottom.)
Upper bound
The upper bound is $n+2$. This is obviously the maximum possible because it is the total number of faces, and it is achieved by the cube graph, where $n = 4$.
This is the only graph that achieves the bound, because it is the only cubic planar graph consisting only of squares. If all the faces are squares, then 4 times the number of faces must be 3 times the number of vertices, or $$ 4(n+2) = 6n $$ so $n = 4$.
If we exclude this single graph, then the upper bound is $n$, and graphs with $F_4 = n$ can be found for arbitrarily large $n$ (see the prism graph at the end of this answer.)
To prove this, we need to show that we cannot have $n + 1$ 4-sided faces and a single $j$-sided face, with $j > 4$. In this case $$ 6n = 4F_4 + jF_j = 4(n+1) + j $$ so $j = 2n - 4$. The same logic as this answer by Henning Makholm applies:
Now start by drawing the face with $2n - 4$ sides. That uses up $2n -4$ nodes. Since the graph is cubic, each of these nodes have an extra neighbor. But if those neighbors are all different, then there would be at least $4n - 8$ nodes in the graph which is too many.
Therefore, there must be a pair of corners of the $j$-gon that share a neighbor outside the $j$-gon. Since [all other faces are 4-cycles], this is only possible if these two nodes are separated by one corner in the $j$-gon. But then it's impossible for the extra leg of the corner between them to connect to anything, which is absurd. So the graph cannot exist.
(It's not that absurd to have an edge going to another graph component contained within a face, but such an edge would be a cut-edge, which cannot exist in a regular bipartite graph.)
Lower bound
The lower bound is an arbitrarily small multiple of $n$. We can see this by applying a bitruncation operation, which replaces every vertex by a hexagon and leaves a smaller version of each face (with the same number of sides). Thus, starting from the cube, we get a bitruncated cube (aka truncated octahedron) with 6 squares and 8 hexagons ($n = 12$).
Bitruncating that gives a polyhedron with 6 squares and 32 hexagons ($n = 36$).
Bitruncating that gives a polyhedron with 6 squares and 104 hexagons ($n = 108$).
Here are the corresponding planar graphs:
In general, bitruncating yields a graph with 3 times as many vertices and the same number of squares, so we can keep going and get $F_4$ to be less than $\epsilon n$ for any $\epsilon > 0$.
Another option would be a chamfering operation, which replaces every edge by a hexagon and every face by a smaller face of the same number of sides. Starting from the cube, this gives you 4 times as many vertices with each step, so $n$ grows even faster, while always keeping 6 squares.
Absolute bounds
As you mention, any 3-regular planar bipartite graph must have at least 6 squares (assuming you forbid digons, that is, two edges between the same vertices). One way to show this is that 3 times the number of vertices equals the sum of each face's length, so
$$\begin{align} 6n &= 4F_4 + 6F_6 + 8F_8 + \ldots \\ &\geq 4F_4 + 6(n+2-F_4) \\ &= 6n + 12 - 2F_4 \\ 2F_4 &\geq 12 \\ F_4 &\geq 6. \end{align}$$
(If you allow digons, you can get this guy with two squares: )
There is no absolute upper bound on $F_4$ (independent of $n$), since you can have prism graphs with arbitrarily many squares:
These all have $F_4 = n$ and are bipartite for any even $n$.
Your cases are not exhaustive. You can still have a subgraph of $K_{4,4}$ that is planar and is not a subgraph of $K_{1,7}$, $K_{2,6}$, or $C_8$. Think, for example, of the graph of a cube. Also, you seem to be confusing edges and vertices. The question asks for a graph with $8$ vertices and $13$ edges, but you seem to be talking about graphs with $8$ edges.
While the statement is correct; a planar graph cannot have $8$ vertices and $13$ edges, a better approach is to use Euler's formula. Let $G$ be a planar graph with $8$ vertices and $13$ edges. Without loss of generality, $G$ is connected, so $v + f - e = 2$ (assuming $G$ is connected). This gives us $f = 7$. Now if $G$ is $2$-colorable, then every face has at least $4$ edges on it. So we can "recount" the vertices: we have $7$ faces, each with at least $4$ edges on them, and each edge appears in at most two face. So $e \ge 7 \cdot 4 / 2 = 14$, a contradiction.
Best Answer
Kuratowski's Theorem provides a rigorous way to classify planar graphs. To show that your graph, $G$, is non-planar, it suffices to show that it contains a subdivision of $K_{3,3}$ as a subgraph.
But the following graph is a subdivision of $K_{3,3}$ and a subgraph of $G$, so we're done.