Welcome to MSE!
So with all these similarities between the commutator and the vector cross product, is it possible to use the commutator product used as the replacement for the cross product in geometric algebra in place of the wedge product?
First, the vector cross product and the geometric algebra commutator use the same symbol $\times$ for their operation. This similarity, not mentioned in your post, could be potentially very confusing when most of the mathematicians and physicists learn vector algebra before they learn geometric algebra, as the commutator of two vectors (as you said) would yield a bivector, while the cross product would only yield a vector.
Another subtle difference between the vector cross product and the commutator that could cause confusion is that the result from the triple vector cross product is the negative of the corresponding triple product for the commutator in $G^{3,0}$. For vectors $a$, $b$, $c$, $d$, and $e$, let $d = a \times (b \times c)$ and $e = b \times c$, where $\times$ represents the vector cross product. Now if $\times$ were to be the geometric algebra commutator, $b \times c$ would instead equal $Ie$, and $a \times (b \times c) = a \times Ie = I(a \times e) = I^2 d = -d$, due to the properties of the pseudoscaler in $G^{3,0}$, causing another potential source of confusion and error.
Due to the possible confusion between the two operators, in geometric algebra the commutator is typically only used for bivectors, while the outer product/wedge product is used for vectors. Perhaps in a future when vector algebra has been superseded by geometric algebra, the commutator can be more widely used (but see below).
And likewise for the symmetric product extension of the dot product $A \cdot B = \frac{1}{2}(AB + BA)$ in place of the typical contraction extension?
In a geometric algebra of dimension less than three, this definition would yield a pure-grade multivector, but once you reach four dimensions, this product of two bivectors would yield a mixed-grade multivector, one of grade 0 and another of grade 4. Typically, we only wish to have one of the two multivector terms - the inner product/contraction would yield the grade 0 term, while the outer product would yield the grade 4 term. Thus, for GA applications such as 3D Conformal Geometry ($G^{4,1}$) or Spacetime Algebra ($G^{3, 1}$), the contraction and outer product would be more useful in that sense.
What are the potential disadvantages of using the commutator and the symmetric dot product in place of the wedge product and contraction dot product in geometric algebra
There are actually two contractions, the left contraction and the right contraction, which do have the less-commonly-used symbols $\lfloor$ and $\rfloor$ as well, but due to historical reasons and the relationships with the vector dot product and the inner product in linear algebra, the $\cdot$ symbol is typically used for the left contraction. So, technically, you're after defining two new binary operations in geometric algebra, which could be used alongside the contractions and the outer product.
But as a replacement for the contractions and the outer product? One of the big advantages of using geometric algebra is that there is a clear intuition between the algebraic objects and operators in geometric algebra and the corresponding geometric objects in the defined space of any dimension. The contractions and outer product have a distinctive geometric interpretation (as do blades) - the contraction of two blades $A$ and $B$ being the part of $A$ most unlike $B$, according to Leo Dorst in his textbook Geometric Algebra for Computer Science, while the outer product of $A$ and $B$ is the blade $C$ representing the subspace whose basis is the union of the the bases of the subspaces representing $A$ and $B$. But it is unclear what the geometric interpretation of either the commutator or your defined symmetric dot product should result in. Thus using your two products in place of the ones currently used in geometric algebra would lose some of the intuitive geometric interpretation of geometric algebra that makes it such a powerful tool in geometry, physics, and engineering.
The commutator product is important for bivectors in geometric algebra primarily not for its associations with geometry, but rather because the bivectors equipped with the commutator product are associated with particular Lie Algebras from abstract algebra, which are used greatly in more advanced physics.
So, while it is possible to use the commutator in place of the wedge product, the wedge product offers many more advantages to geometric algebra compared to the commutator.
especially in 3 dimensions?
I make a separate case for three dimensions, as it seems that, if a different symbol were to be used for the commutator, your algebra does seem to be a midway point between traditional vector algebra and full-blown $G^{3,0}$. The main issue with the output of the cross product is that it fails to distinguish between polar and axial vectors, or vectors and pseudovectors, but by introducing an commutative pseudoscalar (complex imaginary) unit whose square is negative one, and defining pseudovectors as the product of a vector and the pseudoscalar, that issue with the output of the cross product is resolved, and furthermore it highlights a link between quaternions and vector algebra. However, this is only the case in three dimensions, and does not generalise to other dimensions, as geometric algebra is intended to, and so suffers from the same limitations as traditional vector algebra.
Edit: That was quite the long response. I wasn't expecting that.
Best Answer
The equation $ab = a\cdot b + a\wedge b$ is neither a definition of the geometric product nor even valid in general. When it is first introduced in Doran and Lasenby's Geometric Algebra for Physicists $a$ and $b$ are assumed to be vectors. Similarly, the equation $$ a\wedge b = \frac12(ab - ba) $$ is valid for vectors but is not valid in general. It is also not stated that this is a definition of the exterior product of vectors, merely that it can be if the geometric product is already defined.
You should consider reading Chapter 4 "Foundations of Geometric Algebra" if you have not already.
One way to define the exterior product is via grade projections of the geometric product.
If $A_r$ is an $r$-vector and $B_s$ is an $s$-vector then $$ A_r\wedge B_s := \langle A_rB_s\rangle_{r+s} $$ where $\langle\cdot\rangle_{r+s}$ is the $(r+s)$-vector component (and is of course zero if there is no such component). Because every multivector is a sum of homogeneous multivectors, we simply extend this definition by linearity: if $n$ is the dimension of our base vector space and $$ A = \sum_{i=0}^nA_i,\quad B = \sum_{i=0}^nB_i $$ where $A_i, B_i$ are $i$-vectors, then $$ A\wedge B := \sum_{i=0}^n\sum_{j=0}^n\langle A_iB_j\rangle_{i+j}. $$
For your example, this looks like $$\begin{aligned} &(a_1 e_1 + a_2 e_2) \wedge (b_3 e_1 e_2) \\ &= a_1 b_3 e_1 \wedge (e_1 e_2) + a_2 b_3 e_2 \wedge (e_1 e_2) \\ &= a_1b_3\langle e_1e_1e_2\rangle_3 + a_2b_3\langle e_2e_1e_2\rangle_3 \end{aligned}$$ then $$ \langle e_1e_1e_2\rangle_3 = e_1^2\langle e_2\rangle_3 = 0, $$$$ \langle e_2e_1e_2\rangle_3 = -e_2^2\langle e_1\rangle_3 = 0 $$ so the whole expression is $0$. I've used the fact that $e_i^2$ is a scalar and that $e_ie_j = -e_je_i$ when $i\ne j$.
Practically, what might help the most are the following facts:
In fact, these properties completely characterize the exterior product. We can derive two more key facts: