[Math] Confusion on terminology: “vector rejection”, “vector projection”, and “orthogonal projection”

linear algebramatricesorthogonal matricesprojectionprojection-matrices

I have not found a satisfying answer to what is the difference between these three, intuitively:

1) vector projection

2) vector rejection

3) orthogonal projection

I know that "vector rejection of a onto b" will give me the vector that is orthogonal to the "vector projection of a onto b". So, it would make sense to me that "vector rejection" is synonymous with "orthogonal projection". right?

But according to other sources, (e.g. (difference between parallel and orthogonal projection)) "orthogonal projection" is somehow included in the definition of vector projection.

If that is the case, then I do not understand what is a projection that is not an orthogonal projection?

In your answer, feel free to show math – I can understand math. But I'm having difficulty connecting the math to the definitions I see here on Stack Exchange.


Best Answer

The general case of projections is this:

Suppose a vector space $V$ decomposes as a direct sum of subspaces $U$ and $W$, i.e., $V = U \oplus W$. Then every $v \in V$ can be expressed uniquely as $v = u + w$, where $u \in U$ and $w \in W$. We say that the vector $u$ is the projection of the vector $v$ onto the subspace $U$ (with respect to the decomposition $U \oplus W$).

Notice that this definition has nothing to do with orthogonality! We can project onto any subspace with respect to any decomposition.

Now let's look at orthogonal projections. Assume, as above, $V = U \oplus W$, but now additionally we assume that $U$ and $W$ are orthogonal. (Here I must assume that $V$ is now an inner product space, so we have a notion of orthogonality. If you are not familiar with such spaces, you can just think of $\mathbb{R}^n$ with the usual notion of orthogonality.)

We designate one of the subspaces (say $U$) as the "parallel" one we are projecting onto, and the other one ($W$) as the "orthogonal" one. (I am using the terminology of the question you linked to.) Then, in your examples 1-3 above:

  1. Vector projection is projection onto $U$.
  2. Vector rejection is projection onto $W$.
  3. Orthogonal projection is projection onto $W$ (same as 2).