Some authors define the geometric product in terms of the dot and wedge product, which are introduced separately. I think that accentuates an apples vs oranges view. Suppose instead you expand a geometric product in terms of coordinates, with $ \mathbf{a} = \sum_{i = 1}^N a_i \mathbf{e}_i, \mathbf{b} = \sum_{i = 1}^N b_i \mathbf{e}_i $, so that the product is
$$\mathbf{a} \mathbf{b}= \sum_{i, j = 1}^N a_i b_j \mathbf{e}_i \mathbf{e}_j= \sum_{i = 1}^N a_i b_i \mathbf{e}_i \mathbf{e}_i+ \sum_{1 \le i \ne j \le N}^N a_i b_j \mathbf{e}_i \mathbf{e}_j.$$
An axiomatic presentation of geometric algebra defines the square of a vector as $ \mathbf{x}^2 = \left\lVert {\mathbf{x}} \right\rVert^2 $ (the contraction axiom.). An immediate consequence of this axiom is that $ \mathbf{e}_i \mathbf{e}_i = 1$. Another consequence of the axiom is that any two orthogonal vectors, such as $ \mathbf{e}_i, \mathbf{e}_j $ for $ i \ne j $ anticommute. That is, for $ i \ne j $
$$\mathbf{e}_i \mathbf{e}_j = - \mathbf{e}_j \mathbf{e}_i.$$
Utilizing these consequences of the contraction axiom, we see that the geometric product splits into two irreducible portions
$$\mathbf{a} \mathbf{b}=\sum_{i = 1}^N a_i b_i+ \sum_{1 \le i < j \le N}^N (a_i b_j - b_i a_j) \mathbf{e}_i \mathbf{e}_j.$$
The first sum (the symmetric sum) is a scalar, which we recognize as the dot product $ \mathbf{a} \cdot \mathbf{b}$, and the second (the antisymmetric sum) is something else. We call this a bivector, or identify it as the wedge product $\mathbf{a} \wedge \mathbf{b}$.
In this sense, the dot and wedge product sum representation of a geometric product, are just groupings of terms of a larger integrated product.
Another way of reconciling the fact that we appear able to add two unlike entities, is to recast the geometric product in polar form. To do so, consider a decomposition of a geometric product in terms of constituent unit vectors
$$\mathbf{a} \mathbf{b} = \left\lVert {\mathbf{a}} \right\rVert \left\lVert {\mathbf{b}} \right\rVert \left( { \hat{\mathbf{a}} \cdot \hat{\mathbf{b}} + \hat{\mathbf{a}} \wedge \hat{\mathbf{b}} } \right),$$
and assume that we are interested in the non-trivial case where $ \mathbf{a} $ and $ \mathbf{b} $ are not colinear (where the product reduces to just $ \mathbf{a} \mathbf{b} = \left\lVert {\mathbf{a}} \right\rVert \left\lVert {\mathbf{b}} \right\rVert $). It can be shown that the square of a wedge product is always non-positive, so it is reasonable to define the length of a wedge product like so
$$\left\lVert {\hat{\mathbf{a}} \wedge \hat{\mathbf{b}}} \right\rVert = \sqrt{-(\hat{\mathbf{a}} \wedge \hat{\mathbf{b}})^2}.$$
We can use this to massage the dot plus wedge unit vector sum above into
$$\mathbf{a} \mathbf{b} = \left\lVert {\mathbf{a}} \right\rVert \left\lVert {\mathbf{b}} \right\rVert \left( { \hat{\mathbf{a}} \cdot \hat{\mathbf{b}} +\frac{\hat{\mathbf{a}} \wedge \hat{\mathbf{b}} }{\left\lVert {\hat{\mathbf{a}} \wedge \hat{\mathbf{b}}} \right\rVert}\left\lVert {\hat{\mathbf{a}} \wedge \hat{\mathbf{b}}} \right\rVert} \right).$$
The sum has two scalar factors of interest, the dot product $ \hat{\mathbf{a}} \cdot \hat{\mathbf{b}} $ and the length of the wedge product $ \left\lVert {\hat{\mathbf{a}} \wedge \hat{\mathbf{b}}} \right\rVert $. Viewed geometrically, these are the respective projections onto two perpendicular axes, as crudely sketched in the figure
That is, we can make the identifications
$$\hat{\mathbf{a}} \cdot \hat{\mathbf{b}} = \cos\theta$$
$$\left\lVert { \hat{\mathbf{a}} \wedge \hat{\mathbf{b}} } \right\rVert = \sin\theta.$$
(Aside: Admittedly, I've pulled this sine/wedge identification out of a black hat, but it follows logically from study of projection and rejection in geometric algebra. The black hat magic trick may at least be verified by computing the length of the "rejection" component of the vector $\hat{\mathbf{a}}$, that is, $\hat{\mathbf{a}} - \hat{\mathbf{b}} \left( {\hat{\mathbf{a}} \cdot \hat{\mathbf{b}}} \right)$, which has squared length $ 1 - \left( {\hat{\mathbf{a}} \cdot \hat{\mathbf{b}}} \right)^2$. Expanding $ -\left( { \hat{\mathbf{a}} \wedge \hat{\mathbf{b}} } \right)^2 = -\left( { \hat{\mathbf{a}} \wedge \hat{\mathbf{b}} } \right) \cdot \left( { \hat{\mathbf{a}} \wedge \hat{\mathbf{b}} } \right) = -\hat{\mathbf{a}} \cdot \left( { \hat{\mathbf{b}} \cdot \left( { \hat{\mathbf{a}} \wedge \hat{\mathbf{b}} } \right) } \right) $ produces the same result.)
Inserting the trigonometric identification of these two scalars into the expansion of the geometric product, we now have
$$\mathbf{a} \mathbf{b} = \left\lVert {\mathbf{a}} \right\rVert \left\lVert {\mathbf{b}} \right\rVert \left( { \cos\theta +\frac{\hat{\mathbf{a}} \wedge \hat{\mathbf{b}} }{\left\lVert {\hat{\mathbf{a}} \wedge \hat{\mathbf{b}}} \right\rVert}\sin\theta} \right).$$
This has a complex structure that can be called out explicitly by making the identification
$$\mathbf{i} \equiv\frac{\hat{\mathbf{a}} \wedge \hat{\mathbf{b}} }{\left\lVert {\hat{\mathbf{a}} \wedge \hat{\mathbf{b}}} \right\rVert},$$
where by our definition of the length of a wedge product $ \mathbf{i}^2 = -1 $.
With such an identification, we see that the multivector factor
of a geometric product
has a complex exponential structure
$$\begin{aligned}\mathbf{a} \mathbf{b}= \left\lVert {\mathbf{a}} \right\rVert \left\lVert {\mathbf{b}} \right\rVert \left( { \cos\theta + \mathbf{i} \sin\theta } \right)= \left\lVert {\mathbf{a}} \right\rVert \left\lVert {\mathbf{b}} \right\rVert e^{\mathbf{i} \theta }.\end{aligned}$$
In this view of the geometric product, while we initially added two apparently dissimilar objects, this was really no less foreign than adding real and imaginary portions of a complex number, and we see that the geometric product can be viewed as a scaled rotation operator operating in the plane spanned by the two vectors.
In 3D, the wedge and the cross products are related by what is called a duality relationship, relating a bivector that can be interpreted as an oriented plane, and the normal to that plane. Algebraically, this relationship is
$$\mathbf{a} \wedge \mathbf{b} = I (\mathbf{a} \times \mathbf{b}),$$
where $ I = \mathbf{e}_1 \mathbf{e}_2 \mathbf{e}_3 $ is a unit trivector (often called the 3D pseudoscalar), which also satisfies $ I^2 = -1 $. With the usual normal notation for the cross product $ \mathbf{a} \times \mathbf{b} = \hat{\mathbf{n}} \left\lVert {\mathbf{a}} \right\rVert \left\lVert {\mathbf{b}} \right\rVert \sin\theta $ we see our unit bivector $\mathbf{i}$, is related to the cross product normal-direction by $\mathbf{i} = I \hat{\mathbf{n}} $. A rough characterization of this is that $ \mathbf{i} $ is a unit (oriented) plane that is spanned by $ \mathbf{a}, \mathbf{b} $ normal to $ \hat{\mathbf{n}}$.
The intuition of that the geometric product and the Lagrange identity are related is on the mark. There is a wedge product generalization of the Lagrange identity in geometric algebra. The 3D form stated in the question follows from the duality relationship of the wedge and cross products.
Welcome to MSE!
So with all these similarities between the commutator and the vector cross product, is it possible to use the commutator product used as the replacement for the cross product in geometric algebra in place of the wedge product?
First, the vector cross product and the geometric algebra commutator use the same symbol $\times$ for their operation. This similarity, not mentioned in your post, could be potentially very confusing when most of the mathematicians and physicists learn vector algebra before they learn geometric algebra, as the commutator of two vectors (as you said) would yield a bivector, while the cross product would only yield a vector.
Another subtle difference between the vector cross product and the commutator that could cause confusion is that the result from the triple vector cross product is the negative of the corresponding triple product for the commutator in $G^{3,0}$. For vectors $a$, $b$, $c$, $d$, and $e$, let $d = a \times (b \times c)$ and $e = b \times c$, where $\times$ represents the vector cross product. Now if $\times$ were to be the geometric algebra commutator, $b \times c$ would instead equal $Ie$, and $a \times (b \times c) = a \times Ie = I(a \times e) = I^2 d = -d$, due to the properties of the pseudoscaler in $G^{3,0}$, causing another potential source of confusion and error.
Due to the possible confusion between the two operators, in geometric algebra the commutator is typically only used for bivectors, while the outer product/wedge product is used for vectors. Perhaps in a future when vector algebra has been superseded by geometric algebra, the commutator can be more widely used (but see below).
And likewise for the symmetric product extension of the dot product $A \cdot B = \frac{1}{2}(AB + BA)$ in place of the typical contraction extension?
In a geometric algebra of dimension less than three, this definition would yield a pure-grade multivector, but once you reach four dimensions, this product of two bivectors would yield a mixed-grade multivector, one of grade 0 and another of grade 4. Typically, we only wish to have one of the two multivector terms - the inner product/contraction would yield the grade 0 term, while the outer product would yield the grade 4 term. Thus, for GA applications such as 3D Conformal Geometry ($G^{4,1}$) or Spacetime Algebra ($G^{3, 1}$), the contraction and outer product would be more useful in that sense.
What are the potential disadvantages of using the commutator and the symmetric dot product in place of the wedge product and contraction dot product in geometric algebra
There are actually two contractions, the left contraction and the right contraction, which do have the less-commonly-used symbols $\lfloor$ and $\rfloor$ as well, but due to historical reasons and the relationships with the vector dot product and the inner product in linear algebra, the $\cdot$ symbol is typically used for the left contraction. So, technically, you're after defining two new binary operations in geometric algebra, which could be used alongside the contractions and the outer product.
But as a replacement for the contractions and the outer product? One of the big advantages of using geometric algebra is that there is a clear intuition between the algebraic objects and operators in geometric algebra and the corresponding geometric objects in the defined space of any dimension. The contractions and outer product have a distinctive geometric interpretation (as do blades) - the contraction of two blades $A$ and $B$ being the part of $A$ most unlike $B$, according to Leo Dorst in his textbook Geometric Algebra for Computer Science, while the outer product of $A$ and $B$ is the blade $C$ representing the subspace whose basis is the union of the the bases of the subspaces representing $A$ and $B$. But it is unclear what the geometric interpretation of either the commutator or your defined symmetric dot product should result in. Thus using your two products in place of the ones currently used in geometric algebra would lose some of the intuitive geometric interpretation of geometric algebra that makes it such a powerful tool in geometry, physics, and engineering.
The commutator product is important for bivectors in geometric algebra primarily not for its associations with geometry, but rather because the bivectors equipped with the commutator product are associated with particular Lie Algebras from abstract algebra, which are used greatly in more advanced physics.
So, while it is possible to use the commutator in place of the wedge product, the wedge product offers many more advantages to geometric algebra compared to the commutator.
especially in 3 dimensions?
I make a separate case for three dimensions, as it seems that, if a different symbol were to be used for the commutator, your algebra does seem to be a midway point between traditional vector algebra and full-blown $G^{3,0}$. The main issue with the output of the cross product is that it fails to distinguish between polar and axial vectors, or vectors and pseudovectors, but by introducing an commutative pseudoscalar (complex imaginary) unit whose square is negative one, and defining pseudovectors as the product of a vector and the pseudoscalar, that issue with the output of the cross product is resolved, and furthermore it highlights a link between quaternions and vector algebra. However, this is only the case in three dimensions, and does not generalise to other dimensions, as geometric algebra is intended to, and so suffers from the same limitations as traditional vector algebra.
Edit: That was quite the long response. I wasn't expecting that.
Best Answer
$ \newcommand\grade[1]{\langle#1\rangle} \newcommand\lcontr{{\rfloor}} \newcommand\rcontr{{\lfloor}} $
I will answer the first question in two ways: the way they did it, and the better way. By then we will have the answer to the second question.
How they did it
$A$ and $B$ are bivectors, and $x$ is a vector. We get $$ A\cdot(x\wedge(x\cdot B)) = \grade{A(x\wedge(x\cdot B))} = \grade{A\,x{\wedge}(x\cdot B)} $$ since we know the result is a scalar, and then we drop the first set of parentheses since we give the geometric product less precedence. Now, the geometric product $x(x\cdot B)$ has a grade 0 part $x\cdot(x\cdot B)$ and a grade 2 part $x\wedge(x\cdot B)$. But $\grade{A(\text{not grade 2})} = 0$ because $A$ is grade 2 (the product of a $k$-vector and an $l$-vector gives a least possible grade of $|k-l|$). So in this case it's perfectly safe to replace $x\wedge(x\cdot B)$ with $x(x\cdot B)$: $$ \grade{A\,x{\wedge}(x\cdot B)} = \grade{Ax(x\cdot B)} = \grade{(Ax)(x\cdot B)}. $$ The last equality is by associativity of the geometric product (which is key in a lot of manipulations like this). The term $x\cdot B$ is grade 1, so $\grade{(\text{not grade 1})(x\cdot B)} = 0$, meaning we only get the grade 1 part of $Ax$ which is $A\cdot x$. Hence $$ \grade{(Ax)(x\cdot B)} = \grade{(A\cdot x)(x\cdot B)}. $$ I don't know why they decided to write $\grade{(A\cdot x)xB}$ instead, since what I would next is $$ \grade{(A\cdot x)(x\cdot B)} = (A\cdot x)\cdot(x\cdot B) = -(x\cdot A)\cdot(x\cdot B). $$ The first equality comes from the fact that $A\cdot x$ and $x\cdot B$ are vectors. Now we see that this expression is symmetric in $A$ and $B$ (since the dot of two vectors is), so we can simply swap them in the original expression: $$ A\cdot(x\wedge(x\cdot B)) = B\cdot (x\wedge(x\cdot A)). $$
The better way
Read The Inner Products of Geometric Algebra (2002) by Leo Dorst. The left and right contractions $\lcontr$ and $\rcontr$ are variants of Doran an Lasenby's dot product which are better theoretically and practically. They could be defined as $$ A_r\lcontr B_s = \begin{cases} \grade{A_rB_s}_{s-r} &\text{if } s \geq r, \\ 0 &\text{otherwise}, \end{cases} $$$$ A_r\rcontr B_s = \begin{cases} \grade{A_rB_s}_{r-s} &\text{if } r \geq s, \\ 0 &\text{otherwise}, \end{cases} $$ where here subscripts indicate grade. The important identities for our purposes are $$ (A\wedge B)\lcontr C = A\lcontr(B\lcontr C),\quad A\rcontr(B\wedge C) = (A\rcontr B)\rcontr C, $$ which hold for all multivectors $A, B, C$. We translate the dots into contractions and then everything is simple. The dot with $B$ could be $\lcontr$ or $\rcontr$, but we judiciously choose $\rcontr$. $$ A\cdot(x\wedge(x\cdot B)) = A\rcontr(x\wedge(x\lcontr B)) = (A\rcontr x)\rcontr(x\lcontr B) = (A\rcontr x)\cdot(x\lcontr B). $$ In the last step I switch $\rcontr$ to $\cdot$ since the arguments have the same grade and I like to emphasize the symmetry. Same as before we get $$ (A\rcontr x)\cdot(x\lcontr B) = -(x\lcontr A)\cdot(x\lcontr B), $$ which shows that $A$ and $B$ are symmetric in this expression, hence $$ A\rcontr(x\wedge(x\lcontr B)) = B\rcontr(x\wedge(x\lcontr A)). $$
Question 2
The expression $$ (Ia)\cdot(x\wedge(x\cdot(Ia))) $$ is the same one as before with $A = B = Ia$, and I've shown in two ways that $$ (Ia)\cdot(x\wedge(x\cdot(Ia))) = -(x\cdot(Ia))\cdot(x\cdot(Ia)) $$ But then we see $$ -(x\cdot(Ia))\cdot(x\cdot(Ia)) = -(x\cdot(Ia))^2, $$ and there's a minus sign in front the integral which cancels with this one.