$\newcommand{\Brak}[1]{\langle #1 \rangle}\DeclareMathOperator{\proj}{proj}$If $A$ and $B \neq 0$ are vectors (in an arbitrary inner product space, with the inner product denoted by angle brackets), there exists a unique pair of vectors that are (respectively) parallel to $B$ and orthogonal to $B$, and whose sum is $A$. These vectors are, indeed, given by explicit formulas:
$$
\proj_{B}(A) = \frac{\Brak{A, B}}{\Brak{B, B}}\, B,\qquad
\proj_{B^{\perp}}(A) = A - \proj_{B}(A)
$$
The first is sometimes called the component of $A$ along $B$, and the second is the component of $A$ perpendicular/orthogonal to $B$.
The point is, the component of $A$ perpendicular to $B$ is unique (unles you have a definition that explicitly says otherwise) so "no", you need not/should not take both choices of sign.
The dot product is the special case of a more general concept, the inner product. If you have a vector space $ V $ over the reals or the complex numbers, then an inner product is a map $ f : V \times V \to \mathbb{C} $ or $ f : V \times V \to \mathbb{R} $ which is conjugate symmetric, positive definite, and linear in its first argument. We usually write $ f(u, v) = \langle u, v \rangle $, in which case these properties can be summed up as follows:
- Conjugate symmetry: $ \overline{\langle u, v \rangle} = \langle v, u \rangle $, where $ \bar{z} $ denotes complex conjugation. Note that this implies $ \langle u, u \rangle $ is always real for any vector $ u $.
- Positive definiteness: $ \langle v, v \rangle \geq 0 $ for any $ v \in V $, with equality holding iff $ v = 0 $.
- Linearity in the first argument: $ \langle \alpha u + \beta v, w \rangle = \alpha \langle u, w \rangle + \beta \langle v, w \rangle $ where $ u, v, w \in V $ and $ \alpha, \beta $ are in the field of scalars.
If $ V = \mathbb{R}^n $, then we can fix a basis $ B = \{ b_i \in \mathbb{R}, 1 \leq i \leq n \} $ and define $ \langle b_i, b_i \rangle = 1 $ and $ \langle b_i, b_j \rangle = 0 $ for $ i \neq j $. Extending this to all of $ \mathbb{R}^n $ by linearity gives us
$$ \left \langle \sum_{k=1}^{n} c_k b_k, \sum_{j=1}^{n} d_j b_j \right \rangle = \sum_{1 \leq k, j \leq n} d_k c_j \langle b_i, b_j \rangle = \sum_{i=1}^{n} c_i d_i $$
where positive definiteness is readily verified. You will recognize this expression as the definition of the dot product. Indeed, if we take our basis $ B $ to be the standard basis of $ \mathbb{R}^n $, then this inner product is the dot product.
Why is this formalism more powerful? A result about the inner product is the Cauchy-Schwarz inequality, which says that $ |\langle u, v \rangle| \leq |u| |v| $ where $ |u| = \sqrt{\langle u, u \rangle} $. This tells us that
$$ -1 \leq \frac{\langle u, v \rangle}{|u| |v|} \leq 1 $$
assuming that our field of scalars is $ \mathbb{R} $. We then see that the arccosine of this expression is well-defined, so we can define the angle between nonzero vectors $ u $ and $ v $ as
$$ \theta = \arccos \left( \frac{\langle u, v \rangle}{|u| |v|} \right) $$
The properties we expect to be true are then easily verified. This notion extends to infinite dimensional vector spaces over $ \mathbb{R} $, where defining angle is not at all obvious. It is then trivially true that we have $ \langle u, v \rangle = |u| |v| \cos(\theta) $, since that is how $ \theta $ was defined.
The cross product is an entirely separate concept which allows us to find a vector orthogonal to two given vectors in $ \mathbb{R}^3 $. In addition, its magnitude also gives the area of the parallelogram spanned by the vectors. These properties can be taken as the definition of the cross product (with appropriate care for orientation), or they can be derived as theorems starting from the algebraic definition.
Best Answer
Consider the plane with normal $\mathbf n$ and containing the point $A$. The equation of this plane, as you have worked out earlier, is $$\mathbf r\cdot\mathbf n=-16$$ The asked-for decomposition requires the perpendicular component to reach from the origin to a point on this plane – in other words, $k\mathbf n\cdot\mathbf n=-16$. Solving, we get $30k=-16$ and hence $k=-\frac8{15}$. Then $$\mathbf b=\vec{OA}-k\mathbf n=(3,-4,2)+\tfrac8{15}(2,5,-1)=(\tfrac{61}{15},-\tfrac43,\tfrac{22}{15})$$ and we can verify that $\mathbf b\cdot\mathbf n=0$, i.e. they are perpendicular as stipulated in the question.