I find your use of the term "orientation" slightly confusing, since it seems to oscillate between an absolute sense (roughly synonymous with "direction") and a relative sense (roughly synonymous with "alignment"). Let me paraphrase how I understand your question:
You have triangles for more than one cylinder in a file, and you don't know which triangle belongs to which cylinder. The normals of the triangles are all roughly orthogonal to the axis of their cylinder. For a given direction, you want to know how well the axes of the cylinders are aligned with this direction. In an extreme case, the axes of all cylinders might have the same direction; in that case, the test with that direction should yield "fully aligned" and a test with any direction orthogonal to it should yield "fully non-aligned". The problem is that though the cylinders may be fully aligned, the normals of the triangles form different angles with any given direction.
It seems to me the "incorrect way" that you're using isn't quite so bad. I don't fully understand how you arrive at $0$ and $\pi/4$, respectively, but an indicator that ranges between $0$ for aligned and $\pi/4$ for non-aligned seems useful. I gather that what you don't like about it is that you want it to be some sort of average angle.
You could probably find a function that maps your current indicator to something like an average angle, but generally speaking, averaging trigonometric functions tends to be easier and to have nicer properties than averaging angles (or absolute values of angles, which seems to be what you're doing), so I'll do something similar to what you did, but using trigonometric functions whose averages we can easily calculate exactly, so we'll get an expression in closed form relating the values of the indicator to something like an average angle.
For this to work, the triangles for each cylinder need to be roughly equidistributed about its axis. If the normals of the triangles tend to preferentially point in one of the directions orthogonal to the axis, that will make the cylinder appear more aligned with the third direction (orthogonal to the axis and the preferred normal direction) than it actually is.
So consider a cylinder with unit vector $\vec a$ along its axis, a unit vector $\vec u$ along which we want to measure the cylinder's alignment, and unit vectors $\vec n_1$ and $\vec n_2$ orthogonal to $\vec a$, which we choose such that $\vec n_1$ is also orthogonal to $\vec u$ (which we can always do). We can write the normal of a triangle of the cylinder as $\vec n=\cos\phi\,\vec n_1+\sin\phi\,\vec n_2$, and equidistribution of the normals around the cylinder's axis amounts to equidistribution of $\phi$ over $[0,2\pi]$. We have $\vec u\cdot\vec n_1=0$ (since we chose $\vec n_1$ orthogonal to $\vec u$) and $\vec u\cdot\vec n_2=\pm\sin\theta$, were $\theta$ is the angle between $\vec a$ and $\vec u$ (you may want to draw a sketch to see why that's the case). Thus $\vec u\cdot\vec n=\pm\sin\theta\sin\phi$. Now if we average this over all the equidistributed normals, we'll just get $0$ from the average over $\phi$, independent of $\theta$; that's not what we want. However, if we square before averaging, we get the average of $\sin^2\theta\sin^2\phi$ over $\phi$, which is $\frac12\sin^2\theta$. That's good, because we shouldn't be distinguishing between negative and positive values of $\theta$ or between $\theta$ and $\pi-\theta$ anyway.
Now to turn this into something like an average angle, we just have to apply the inverse operations. That is, if $\langle(\vec u\cdot\vec n)^2\rangle$ denotes the average of the square of the scalar product of the alignment direction $\vec u$ with the triangle normals $\vec n$, the "average angle" $\hat\theta$ would be derived from $\langle(\vec u\cdot\vec n)^2\rangle=\langle\frac12\sin^2\theta\rangle$ as
$$
\hat\theta=\arcsin\sqrt{2\langle(\vec u\cdot\vec n)^2\rangle}\;.
$$
You'll need to clamp any arguments above $1$ for the arcsine to $1$; this might occur if the normals aren't perfectly equidistributed.
If all the cylinders are perfectly aligned along $\vec u$, then all the normals will be orthogonal to $\vec u$, so $\langle(\vec u\cdot\vec n)^2\rangle$ will be zero, and so will $\hat\theta$. On the other hand, if all the cylinders are perfectly aligned orthogonal to $\vec u$ and their normals are equidistributed about their axes, then $\langle(\vec u\cdot\vec n)^2\rangle$ will be $1/2$, so $\hat\theta$ will be $\pi/2$ as desired. More generally, if all the cylinder axes form an angle $\theta$ with $\vec u$, then $\hat\theta$ will be $\theta$. If the cylinders form different angles with $\vec u$, then $\hat\theta$ will be an average angle in the sense that $\sin^2\theta$ (or equivalently $\cos^2\theta$) is averaged.
If you want to calculate the angle between two vectors (say, $u$ and $v$) in high-dimensional space, the formula is
$$
A = \cos^{-1}\left(\frac{u \cdot v}{\|u\|\|v\|}\right)
$$
Where $u\cdot v$ denotes the dot product. In MATLAB:
A = acos((u'*v)/(norm(u)*norm(v)))
So, to get the angles of the projections on the the first $N-1$ axes, let's say, you could do something like this:
v = b-a;
for i = 1:N-1
angles(i) = acos(v(i)/norm(v));
end
Or, if you want to skip the for loop,
angles = acos(v/norm(v));
Best Answer
Exterior algebra fulfills this need. If two planes passing through the origin are generated by vectors $\boldsymbol{v}_1,\boldsymbol{v}_2$ and $\boldsymbol{v}_3,\boldsymbol{v}_4$, then $\boldsymbol{v}_1\wedge\boldsymbol{v}_2$ and $\boldsymbol{v}_3\wedge\boldsymbol{v}_4$ are blades representing the planes that they generate, $$\langle \boldsymbol{v}_1\wedge \boldsymbol{v}_2 ,\boldsymbol{v}_3\wedge \boldsymbol{v}_4\rangle=\begin{vmatrix} \langle\boldsymbol{v}_1,\boldsymbol{v}_3\rangle &\langle\boldsymbol{v}_1,\boldsymbol{v}_4\rangle\\ \langle\boldsymbol{v}_2,\boldsymbol{v}_3\rangle &\langle\boldsymbol{v}_2,\boldsymbol{v}_4\rangle \end{vmatrix}$$ is an inner product, and $$\cos\theta=\frac{\langle \boldsymbol{v}_1\wedge \boldsymbol{v}_2 ,\boldsymbol{v}_3\wedge \boldsymbol{v}_4\rangle}{\sqrt{\langle \boldsymbol{v}_1\wedge \boldsymbol{v}_2 ,\boldsymbol{v}_1\wedge \boldsymbol{v}_2\rangle \langle \boldsymbol{v}_3\wedge \boldsymbol{v}_4 ,\boldsymbol{v}_3\wedge \boldsymbol{v}_4\rangle}}\text{.}$$ gives the angle between the planes.