Welcome to MSE!
So with all these similarities between the commutator and the vector cross product, is it possible to use the commutator product used as the replacement for the cross product in geometric algebra in place of the wedge product?
First, the vector cross product and the geometric algebra commutator use the same symbol $\times$ for their operation. This similarity, not mentioned in your post, could be potentially very confusing when most of the mathematicians and physicists learn vector algebra before they learn geometric algebra, as the commutator of two vectors (as you said) would yield a bivector, while the cross product would only yield a vector.
Another subtle difference between the vector cross product and the commutator that could cause confusion is that the result from the triple vector cross product is the negative of the corresponding triple product for the commutator in $G^{3,0}$. For vectors $a$, $b$, $c$, $d$, and $e$, let $d = a \times (b \times c)$ and $e = b \times c$, where $\times$ represents the vector cross product. Now if $\times$ were to be the geometric algebra commutator, $b \times c$ would instead equal $Ie$, and $a \times (b \times c) = a \times Ie = I(a \times e) = I^2 d = -d$, due to the properties of the pseudoscaler in $G^{3,0}$, causing another potential source of confusion and error.
Due to the possible confusion between the two operators, in geometric algebra the commutator is typically only used for bivectors, while the outer product/wedge product is used for vectors. Perhaps in a future when vector algebra has been superseded by geometric algebra, the commutator can be more widely used (but see below).
And likewise for the symmetric product extension of the dot product $A \cdot B = \frac{1}{2}(AB + BA)$ in place of the typical contraction extension?
In a geometric algebra of dimension less than three, this definition would yield a pure-grade multivector, but once you reach four dimensions, this product of two bivectors would yield a mixed-grade multivector, one of grade 0 and another of grade 4. Typically, we only wish to have one of the two multivector terms - the inner product/contraction would yield the grade 0 term, while the outer product would yield the grade 4 term. Thus, for GA applications such as 3D Conformal Geometry ($G^{4,1}$) or Spacetime Algebra ($G^{3, 1}$), the contraction and outer product would be more useful in that sense.
What are the potential disadvantages of using the commutator and the symmetric dot product in place of the wedge product and contraction dot product in geometric algebra
There are actually two contractions, the left contraction and the right contraction, which do have the less-commonly-used symbols $\lfloor$ and $\rfloor$ as well, but due to historical reasons and the relationships with the vector dot product and the inner product in linear algebra, the $\cdot$ symbol is typically used for the left contraction. So, technically, you're after defining two new binary operations in geometric algebra, which could be used alongside the contractions and the outer product.
But as a replacement for the contractions and the outer product? One of the big advantages of using geometric algebra is that there is a clear intuition between the algebraic objects and operators in geometric algebra and the corresponding geometric objects in the defined space of any dimension. The contractions and outer product have a distinctive geometric interpretation (as do blades) - the contraction of two blades $A$ and $B$ being the part of $A$ most unlike $B$, according to Leo Dorst in his textbook Geometric Algebra for Computer Science, while the outer product of $A$ and $B$ is the blade $C$ representing the subspace whose basis is the union of the the bases of the subspaces representing $A$ and $B$. But it is unclear what the geometric interpretation of either the commutator or your defined symmetric dot product should result in. Thus using your two products in place of the ones currently used in geometric algebra would lose some of the intuitive geometric interpretation of geometric algebra that makes it such a powerful tool in geometry, physics, and engineering.
The commutator product is important for bivectors in geometric algebra primarily not for its associations with geometry, but rather because the bivectors equipped with the commutator product are associated with particular Lie Algebras from abstract algebra, which are used greatly in more advanced physics.
So, while it is possible to use the commutator in place of the wedge product, the wedge product offers many more advantages to geometric algebra compared to the commutator.
especially in 3 dimensions?
I make a separate case for three dimensions, as it seems that, if a different symbol were to be used for the commutator, your algebra does seem to be a midway point between traditional vector algebra and full-blown $G^{3,0}$. The main issue with the output of the cross product is that it fails to distinguish between polar and axial vectors, or vectors and pseudovectors, but by introducing an commutative pseudoscalar (complex imaginary) unit whose square is negative one, and defining pseudovectors as the product of a vector and the pseudoscalar, that issue with the output of the cross product is resolved, and furthermore it highlights a link between quaternions and vector algebra. However, this is only the case in three dimensions, and does not generalise to other dimensions, as geometric algebra is intended to, and so suffers from the same limitations as traditional vector algebra.
Edit: That was quite the long response. I wasn't expecting that.
Best Answer
The best way to view all of these identities is consequences of grade selection. We say that scalars have grade 0, vectors grade 1, bivectors are grade 2, and so forth. Examples, assuming that $ \left\{ { \mathbf{e}_1, \mathbf{e}_2, \cdots } \right\} $ is an orthonormal Euclidean basis, then $$-3, \pi, 7, 9, \cdots$$ are all scalars with grade 0, $$ \mathbf{e}_1, \mathbf{e}_2, 3 \mathbf{e}_3 + \pi \mathbf{e}_7, \cdots$$ are all vectors with grade 1, $$ \mathbf{e}_1 \mathbf{e}_2, \mathbf{e}_2 \mathbf{e}_3, 3 \mathbf{e}_3 \mathbf{e}_7 + \pi \mathbf{e}_7 \mathbf{e}_1, \cdots$$ are all bivectors and have grade 2. We write $$ {\left\langle{{A}}\right\rangle}_{{k}}$$ as the selection of all grade $ k $ elements from the multivector $ A $. For example, if $$ A = -3 + \pi+ 7+ 9 + \mathbf{e}_1+ \mathbf{e}_2+ 3 \mathbf{e}_3 + \pi \mathbf{e}_7 + \mathbf{e}_1 \mathbf{e}_2+ \mathbf{e}_2 \mathbf{e}_3+ 3 \mathbf{e}_3 \mathbf{e}_7 + \pi \mathbf{e}_7 \mathbf{e}_1,$$ then $$ {\left\langle{{A}}\right\rangle}_{{0}} = \pi+ 13,$$ $$ {\left\langle{{A}}\right\rangle}_{{1}} = \mathbf{e}_1+ \mathbf{e}_2+ 3 \mathbf{e}_3 + \pi \mathbf{e}_7 ,$$ $$ {\left\langle{{A}}\right\rangle}_{{2}} = \mathbf{e}_1 \mathbf{e}_2+ \mathbf{e}_2 \mathbf{e}_3+ 3 \mathbf{e}_3 \mathbf{e}_7 + \pi \mathbf{e}_7 \mathbf{e}_1,$$ and $ {\left\langle{{A}}\right\rangle}_{{k}} $ for all $ k > 2 $ is zero (in this case.)
Now, we can consider the geometric product of two vectors $ a = \sum_i x_i \mathbf{e}_i $ and $ b = \sum_i y_i \mathbf{e}_i $ $$\begin{aligned} a b &= \sum_{i,j} x_i \mathbf{e}_i y_j \mathbf{e}_j \\ &= \sum_{i = j} x_i y_i \mathbf{e}_i^2 + \sum_{i \ne j} x_i y_j \mathbf{e}_i \mathbf{e}_j \\ &= \sum_{i} x_i y_i + \sum_{i \ne j} x_i y_j \mathbf{e}_i \mathbf{e}_j \\ &= \sum_{i} x_i y_i + \sum_{i < j} \left( { x_i y_j - x_j y_i } \right) \mathbf{e}_i \mathbf{e}_j \\ \end{aligned}$$ Observe that the first sum is a scalar, and the second sum has only grade two terms. This means that for any product of two vectors we must have $$ a b = {\left\langle{{a b}}\right\rangle}_{{0}} + {\left\langle{{a b}}\right\rangle}_{{2}}.$$ Clearly we wish to identify the scalar component as the dot product, and we define the bivector component to be the wedge product: $$ a \cdot b = {\left\langle{{ a b }}\right\rangle}_{{0}},$$ $$ a \wedge b = {\left\langle{{ a b }}\right\rangle}_{{2}}.$$
You should be able to convince yourself that a product of a vector and r-blade $ A_r $ (an entity with only grade r components) must have only grades $ r-1 $ and $ r + 1 $, so $$ a A_r = {\left\langle{{a A_r}}\right\rangle}_{{r-1}} + {\left\langle{{a A_r}}\right\rangle}_{{r+1}},$$ so we define $$ a \cdot A_r = {\left\langle{{ a A_r }}\right\rangle}_{{r-1}},$$ $$ a \wedge A_r = {\left\langle{{ a A_r }}\right\rangle}_{{r+1}},$$ so that $$ a A_r = a \cdot A_r + a \wedge A_r.$$
Still more generally, answering your final question, one can define the dot and wedge products of two blades $ A_r $, $ B_s $ with grades $ r, s $ respectively, as $$ A_r \cdot B_s = {\left\langle{{ A_r B_s }}\right\rangle}_{{\left\lvert {r-s} \right\rvert}},$$ $$ A_r \wedge B_s = {\left\langle{{ A_r B_s }}\right\rangle}_{{r+s}}.$$ Note that there is some variability in conventions and notations for dot product like operators, but these are definitions that are consistent with those used in New Foundations.