This is a tremendously common confusion to have, and in my experience, people are notoriously bad at explaining this concept. I'm sorry that you had to deal with people who were abrasive in addition to poor expositors.
In an arbitrary vector space, you cannot talk about components. They actually don't exist. Now, you can impose them on a finite-dimensional space by providing a bijective linear transformation from the arbitrary vector space to $F^n$, but then they're just that: an imposition, because any other bijective linear transformation will choose different would-be "components".
Components exist in $F^n$ because of the actual nature of the objects involved. So you don't need a basis, you can just look at an arbitrary object $(a,b,\dots,n)$, and find any of its components, because they're built into the object. This can be confusing because we also write coordinate vectors in this way, and when the basis is the standard basis, there is no difference between the components and the coordinates. However, in any other basis, there will be a difference.
(Edit: Val made an important point in the comments. I should have been more careful when I said there was "no difference". The fact is that coordinates and components are never conceptually the same, but I meant to say that in the standard basis case they will be numerically equal.)
Lacking a basis at all, you might want to say that $F^n$ still has coordinates implied by its components. But, in my opinion, this seems silly, since you cannot do the same in other spaces.
So the short answer is: Yes, there is a difference, because components are part of the objects.
As for your "collection of vectors" notion, they are basically the same. But it is easy to imagine a collection of vectors which is not a vector space: for example the circle in $\mathbb{R}^2$. This is definitely a collection, and the objects in it are definitely vectors, but it is not a vector space.
What I assume you meant by "collection" was what we might call a "meaningfully structured collection", and the meaningful structure is described precisely as an abelian group over which elements can be scaled by objects in a field. In that sense, your notion is correct, though a bit less transparent.
The general case of projections is this:
Suppose a vector space $V$ decomposes as a direct sum of subspaces $U$ and $W$, i.e., $V = U \oplus W$. Then every $v \in V$ can be expressed uniquely as $v = u + w$, where $u \in U$ and $w \in W$. We say that the vector $u$ is the projection of the vector $v$ onto the subspace $U$ (with respect to the decomposition $U \oplus W$).
Notice that this definition has nothing to do with orthogonality! We can project onto any subspace with respect to any decomposition.
Now let's look at orthogonal projections. Assume, as above, $V = U \oplus W$, but now additionally we assume that $U$ and $W$ are orthogonal. (Here I must assume that $V$ is now an inner product space, so we have a notion of orthogonality. If you are not familiar with such spaces, you can just think of $\mathbb{R}^n$ with the usual notion of orthogonality.)
We designate one of the subspaces (say $U$) as the "parallel" one we are projecting onto, and the other one ($W$) as the "orthogonal" one. (I am using the terminology of the question you linked to.) Then, in your examples 1-3 above:
- Vector projection is projection onto $U$.
- Vector rejection is projection onto $W$.
- Orthogonal projection is projection onto $W$ (same as 2).
Best Answer
In geometric terms ...
In a parallel projection, points are projected (onto some plane) in a direction that is parallel to some fixed given vector.
In an orthogonal projection, points are projected (onto some plane) in a direction that is normal to the plane.
So, all orthogonal projections are parallel projections, but not vice versa. A parallel projection that is not an orthogonal projection is called an "oblique" projection.
This could all be translated into the language of linear algebra, I suppose, but I don't think that would make it any clearer.