So first, the convention in crystallography is to write the Fourier series with a $2\pi$ in the phase, i.e. to replace your $G$ with $2\pi G$, which I will do in the following. I will also drop the index $m$ on $G_m$ because it plays no role. So your equation (1) is equivalent to require that
$$G.(m \vec{a}_1 + n \vec{a}_2 + o\vec{a}_3)$$
is an integer for any integer $m$, $n$, and $o$. By taking the three following $mno$: 100, 010, 001, we get the so-called Laue equations:
$$\begin{aligned}
G.\vec{a}_1 &= h\\
G.\vec{a}_2 &= k\\
G.\vec{a}_3 &= l
\end{aligned}$$
for some integers $h$, $k$, $l$.
The most traditional way to proceed from here I would say is to use the existence of an unique base $(\vec{a}^*_1, \vec{a}^*_2, \vec{a}^*_3)$ dual to the base $(\vec{a}_1, \vec{a}_2, \vec{a}_3)$, which has the essential property that for any vector $\vec{H}$,
$$\vec{H}=(\vec{H}.\vec{a}_1)\vec{a}^*_1 + (\vec{H}.\vec{a}_2)\vec{a}^*_2 + (\vec{H}.\vec{a}_3)\vec{a}^*_3.\tag{3}$$
The Laue equations then immediately give
$$\vec{G}=h\vec{a}^*_1 + k\vec{a}^*_2 + l\vec{a}^*_3, $$
proving that $\vec{G}$ can is a linear combination of the $\vec{a}^*_i$'s with integer coefficients.
Your eq. (2) holds for this dual base (without the factor $2\pi$),
$$\vec{a}_i\cdot\vec{a}^*_j = \delta_{ij}\tag{2}$$
as this is another characterisation of it, but it does not come as a consequence of (1) in this approach: instead it is a general and fundamental result of linear algebra (as dual bases exist in any dimension). In dimension 3, the simplest approach is to construct the dual base as
$$\vec{a}^*_1 = \frac{\vec{a}_2\times\vec{a}_3}{\det(\vec{a}_1, \vec{a}_2, \vec{a}_3)}$$
and circular permutation of indices. Then (2) easily follow, from which (3) is then obvious.
You don't need to. In fact, you can work directly in 2D and solve things explicitly, since the condition for the reciprocal basis that $b_i\cdot a_j = 2\pi\delta_{ij}$ reads in matrix notation
$$
\begin{pmatrix}
b_{1x} & b_{1y} \\ b_{2x} & b_{2y}
\end{pmatrix}
\begin{pmatrix}
a_{1x} & a_{2x} \\ a_{1y} & a_{2y}
\end{pmatrix}
=
2\pi
\begin{pmatrix}
1 & 0 \\ 0 & 1
\end{pmatrix}
,
$$
so all you need to do is multiply with the explicit matrix inverse on the right to get
$$
\begin{pmatrix}
b_{1x} & b_{1y} \\ b_{2x} & b_{2y}
\end{pmatrix}
=
\frac{2\pi}{a_{1x}a_{2y}-a_{1y}a_{2x}}
\begin{pmatrix}
a_{2y} & -a_{2x} \\ -a_{1y} & a_{1x}
\end{pmatrix}
,
$$
or, transposing this for clarity so that you can read off the matrix equation column-by-column to get the $b_i$,
$$
\begin{pmatrix}
b_{1x} & b_{2x} \\ b_{1y} & b_{2y}
\end{pmatrix}
=
\frac{2\pi}{a_{1x}a_{2y}-a_{1y}a_{2x}}
\begin{pmatrix}
a_{2y} & -a_{1y} \\ -a_{2x} & a_{1x}
\end{pmatrix}
.
$$
You can then check by hand that this matches the projection on the $x,y$ plane of the results you get from the 3D formulas by taking the third vector along $z$.
If, on the other hand, you do want to stick to the three-dimensional formalism (which then lets you use the same formulas for both cases) then you need to supplement your basis of the plane with a third vector to make the span three-dimensional. This third vector needs to have a nonzero $z$ component (or it would be linearly dependent on $a_1$ and $a_2$), but you need additional restrictions to specify it uniquely.
These restrictions come from the fact that
- you want $b_3\cdot a_1=b_3\cdot a_2=0$, so you want $b_3$ along the $z$ axis, and more importantly,
- you want $a_3\cdot b_1=a_3\cdot b_2=0$, where you want $b_1$ and $b_2$ to lie in the $xy$ plane because all your physics is two-dimensional, and this requires $a_3$ to lie along the $z$ axis.
As to the precise magnitude of $a_3$, it is irrelevant ─ it is easy to see that changing $a_3\mapsto \lambda a_3$ by any $\lambda \neq 0$ does not affect in any way the 3D $b_1$ and $b_2$ expressions you quote.
Best Answer
They are not necessary, but with this choice you can interpret them as a basis for the wave numbers $\mathbf k$ (as in $e^{i\mathbf k\cdot\mathbf r}$).