You aren't going wrong, but I'd start in a slightly different way so that you get just two terms in your sum to start with. First expand
$$\begin{aligned}a \wedge \left( { b \wedge c } \right)&=\frac{1}{{2}} \left( { a \left( { b \wedge c } \right)+\left( { b \wedge c } \right) a } \right) \\ &=\frac{1}{{2}} \left( { a \left( { b c - b \cdot c } \right)+\left( { b c - b \cdot c } \right) a } \right) \\ &=\frac{1}{{2}} \left( { a b c + b c a } \right) - a \left( { b \cdot c } \right) \\ &=\frac{1}{{2}} \left( { a b c + b c a } \right) - \frac{1}{{2}} a \left( { b c + c b } \right) \\ &=\frac{1}{{2}} \left( { b c a - a c b } \right).\end{aligned}$$
Now note that we can toggle any pair of vectors using $ y x = 2 x \cdot y - x y $, so
$$\begin{aligned}b c a - a c b&=b \left( { 2 c \cdot a - a c } \right) - \left( { 2 a \cdot c - c a } \right) b \\ &=c a b - b a c,\end{aligned}$$
or
$$\begin{aligned}b c a - a c b&=\left( { 2 b \cdot c - c b } \right) a - a \left( { 2 c \cdot b - b c } \right) \\ &=a b c - c b a.\end{aligned}$$
That is
$$b c a - a c b = c a b - b a c = a b c - c b a.$$
We can plug this into our first expansion of the wedge, by writing
$$\begin{aligned}a \wedge \left( { b \wedge c } \right)&=\frac{1}{{2}} \left( { b c a - a c b } \right) \\ &=\frac{1}{{3 \times 2}} 3 \left( { b c a - a c b } \right) \\ &=\frac{1}{{3!}} \left( { \left( { b c a - a c b } \right) + \left( { c a b - b a c } \right) + \left( { a b c - c b a } \right)} \right) \\ &=\frac{1}{{3!}}\left( { a b c + b c a + c a b - a c b - b a c - c b a } \right).\end{aligned}$$
Observe that we have all the permutations of the products in this sum, each weighted by the sign of the permutation, as desired.
$
\newcommand\grade[1]{\langle#1\rangle}
\newcommand\lcontr{{\rfloor}}
\newcommand\rcontr{{\lfloor}}
$
I will answer the first question in two ways: the way they did it, and the better way. By then we will have the answer to the second question.
How they did it
$A$ and $B$ are bivectors, and $x$ is a vector. We get
$$
A\cdot(x\wedge(x\cdot B)) = \grade{A(x\wedge(x\cdot B))} = \grade{A\,x{\wedge}(x\cdot B)}
$$ since we know the result is a scalar, and then we drop the first set of parentheses since we give the geometric product less precedence. Now, the geometric product $x(x\cdot B)$ has a grade 0 part $x\cdot(x\cdot B)$ and a grade 2 part $x\wedge(x\cdot B)$. But $\grade{A(\text{not grade 2})} = 0$ because $A$ is grade 2 (the product of a $k$-vector and an $l$-vector gives a least possible grade of $|k-l|$). So in this case it's perfectly safe to replace $x\wedge(x\cdot B)$ with $x(x\cdot B)$:
$$
\grade{A\,x{\wedge}(x\cdot B)} = \grade{Ax(x\cdot B)} = \grade{(Ax)(x\cdot B)}.
$$
The last equality is by associativity of the geometric product (which is key in a lot of manipulations like this). The term $x\cdot B$ is grade 1, so $\grade{(\text{not grade 1})(x\cdot B)} = 0$, meaning we only get the grade 1 part of $Ax$ which is $A\cdot x$. Hence
$$
\grade{(Ax)(x\cdot B)} = \grade{(A\cdot x)(x\cdot B)}.
$$
I don't know why they decided to write $\grade{(A\cdot x)xB}$ instead, since what I would next is
$$
\grade{(A\cdot x)(x\cdot B)}
= (A\cdot x)\cdot(x\cdot B)
= -(x\cdot A)\cdot(x\cdot B).
$$
The first equality comes from the fact that $A\cdot x$ and $x\cdot B$ are vectors. Now we see that this expression is symmetric in $A$ and $B$ (since the dot of two vectors is), so we can simply swap them in the original expression:
$$
A\cdot(x\wedge(x\cdot B)) = B\cdot (x\wedge(x\cdot A)).
$$
The better way
Read The Inner Products of Geometric Algebra (2002) by Leo Dorst. The left and right contractions $\lcontr$ and $\rcontr$ are variants of Doran an Lasenby's dot product which are better theoretically and practically. They could be defined as
$$
A_r\lcontr B_s = \begin{cases}
\grade{A_rB_s}_{s-r} &\text{if } s \geq r, \\
0 &\text{otherwise},
\end{cases}
$$$$
A_r\rcontr B_s = \begin{cases}
\grade{A_rB_s}_{r-s} &\text{if } r \geq s, \\
0 &\text{otherwise},
\end{cases}
$$
where here subscripts indicate grade.
The important identities for our purposes are
$$
(A\wedge B)\lcontr C = A\lcontr(B\lcontr C),\quad
A\rcontr(B\wedge C) = (A\rcontr B)\rcontr C,
$$
which hold for all multivectors $A, B, C$. We translate the dots into contractions and then everything is simple. The dot with $B$ could be $\lcontr$ or $\rcontr$, but we judiciously choose $\rcontr$.
$$
A\cdot(x\wedge(x\cdot B))
= A\rcontr(x\wedge(x\lcontr B))
= (A\rcontr x)\rcontr(x\lcontr B)
= (A\rcontr x)\cdot(x\lcontr B).
$$
In the last step I switch $\rcontr$ to $\cdot$ since the arguments have the same grade and I like to emphasize the symmetry. Same as before we get
$$
(A\rcontr x)\cdot(x\lcontr B) = -(x\lcontr A)\cdot(x\lcontr B),
$$
which shows that $A$ and $B$ are symmetric in this expression, hence
$$
A\rcontr(x\wedge(x\lcontr B)) = B\rcontr(x\wedge(x\lcontr A)).
$$
Question 2
The expression
$$
(Ia)\cdot(x\wedge(x\cdot(Ia)))
$$
is the same one as before with $A = B = Ia$, and I've shown in two ways that
$$
(Ia)\cdot(x\wedge(x\cdot(Ia))) = -(x\cdot(Ia))\cdot(x\cdot(Ia))
$$
But then we see
$$
-(x\cdot(Ia))\cdot(x\cdot(Ia)) = -(x\cdot(Ia))^2,
$$
and there's a minus sign in front the integral which cancels with this one.
Best Answer
This is technically an error because he introduces a new convention of separating two products he wants to perform left-to-right with a space, but never explicitly states the convention. What he really means is
$$n\cdot a_\perp=\langle (n n) n\wedge a\rangle=\langle n \wedge a\rangle = 0,$$
Or, to be more precise by not skipping steps:
$$n\cdot a_\perp=\langle n(n(n\wedge a))\rangle=\langle (n n) \text{ } n\wedge a\rangle=\langle n \wedge a\rangle = 0,$$
With the second equality following from the associative law.