Taking the absolute values is probably not a good idea, and the $x\to1-x$ transform at the end also doesn't look promising.
I'll assume that by "least aligned" you mean that ideally your vector would be orthogonal to all the normals (and thus in the planes of all the triangles), not that it should be opposite to the normals (and thus pointing into the planes); in the latter case you should just add up all the normals and use the direction opposite to the sum.
A first attempt might be to add up the normals and then take a vector orthogonal to the sum, but then you'd have to arbitrarily choose one of the vectors in the plane orthogonal to the sum, and any information which of these might be best would be lost.
Here's a systematic way of doing this. You want your vector $x$ to be orthogonal to the normals $n_i$, so you want the absolute value of the dot product $x\cdot n_i$ to be as small as possible. A natural ansatz would then be to minimize the sum of the squares of the dot products:
$$
\sum_i(x\cdot n_i)^2\to\min\;,
$$
subject to the constraint that $x$ is a unit vector. We can write this more suggestively:
$$
\sum_ix^\top n_in_i^\top x=x^\top\left(\sum_i n_in_i^\top \right)x\to\min\;.
$$
Adding a Lagrange multiplier term for the normalization constraint yields the objective function
$$
x^\top\left(\sum_i n_in_i^\top \right)x-\lambda x^\top x\;,
$$
and requiring the gradient with respect to $x$ to vanish yields
$$
\left(\sum_i n_in_i^\top \right)x-\lambda x=0\;.
$$
This is the eigenvalue problem for the positive semidefinite matrix $\sum_in_in_i^\top$. That's just a $3\times3$ matrix, so you can readily find its eigensystem. The direction you want is the eigenvector corresponding to its least eigenvalue.
I find your use of the term "orientation" slightly confusing, since it seems to oscillate between an absolute sense (roughly synonymous with "direction") and a relative sense (roughly synonymous with "alignment"). Let me paraphrase how I understand your question:
You have triangles for more than one cylinder in a file, and you don't know which triangle belongs to which cylinder. The normals of the triangles are all roughly orthogonal to the axis of their cylinder. For a given direction, you want to know how well the axes of the cylinders are aligned with this direction. In an extreme case, the axes of all cylinders might have the same direction; in that case, the test with that direction should yield "fully aligned" and a test with any direction orthogonal to it should yield "fully non-aligned". The problem is that though the cylinders may be fully aligned, the normals of the triangles form different angles with any given direction.
It seems to me the "incorrect way" that you're using isn't quite so bad. I don't fully understand how you arrive at $0$ and $\pi/4$, respectively, but an indicator that ranges between $0$ for aligned and $\pi/4$ for non-aligned seems useful. I gather that what you don't like about it is that you want it to be some sort of average angle.
You could probably find a function that maps your current indicator to something like an average angle, but generally speaking, averaging trigonometric functions tends to be easier and to have nicer properties than averaging angles (or absolute values of angles, which seems to be what you're doing), so I'll do something similar to what you did, but using trigonometric functions whose averages we can easily calculate exactly, so we'll get an expression in closed form relating the values of the indicator to something like an average angle.
For this to work, the triangles for each cylinder need to be roughly equidistributed about its axis. If the normals of the triangles tend to preferentially point in one of the directions orthogonal to the axis, that will make the cylinder appear more aligned with the third direction (orthogonal to the axis and the preferred normal direction) than it actually is.
So consider a cylinder with unit vector $\vec a$ along its axis, a unit vector $\vec u$ along which we want to measure the cylinder's alignment, and unit vectors $\vec n_1$ and $\vec n_2$ orthogonal to $\vec a$, which we choose such that $\vec n_1$ is also orthogonal to $\vec u$ (which we can always do). We can write the normal of a triangle of the cylinder as $\vec n=\cos\phi\,\vec n_1+\sin\phi\,\vec n_2$, and equidistribution of the normals around the cylinder's axis amounts to equidistribution of $\phi$ over $[0,2\pi]$. We have $\vec u\cdot\vec n_1=0$ (since we chose $\vec n_1$ orthogonal to $\vec u$) and $\vec u\cdot\vec n_2=\pm\sin\theta$, were $\theta$ is the angle between $\vec a$ and $\vec u$ (you may want to draw a sketch to see why that's the case). Thus $\vec u\cdot\vec n=\pm\sin\theta\sin\phi$. Now if we average this over all the equidistributed normals, we'll just get $0$ from the average over $\phi$, independent of $\theta$; that's not what we want. However, if we square before averaging, we get the average of $\sin^2\theta\sin^2\phi$ over $\phi$, which is $\frac12\sin^2\theta$. That's good, because we shouldn't be distinguishing between negative and positive values of $\theta$ or between $\theta$ and $\pi-\theta$ anyway.
Now to turn this into something like an average angle, we just have to apply the inverse operations. That is, if $\langle(\vec u\cdot\vec n)^2\rangle$ denotes the average of the square of the scalar product of the alignment direction $\vec u$ with the triangle normals $\vec n$, the "average angle" $\hat\theta$ would be derived from $\langle(\vec u\cdot\vec n)^2\rangle=\langle\frac12\sin^2\theta\rangle$ as
$$
\hat\theta=\arcsin\sqrt{2\langle(\vec u\cdot\vec n)^2\rangle}\;.
$$
You'll need to clamp any arguments above $1$ for the arcsine to $1$; this might occur if the normals aren't perfectly equidistributed.
If all the cylinders are perfectly aligned along $\vec u$, then all the normals will be orthogonal to $\vec u$, so $\langle(\vec u\cdot\vec n)^2\rangle$ will be zero, and so will $\hat\theta$. On the other hand, if all the cylinders are perfectly aligned orthogonal to $\vec u$ and their normals are equidistributed about their axes, then $\langle(\vec u\cdot\vec n)^2\rangle$ will be $1/2$, so $\hat\theta$ will be $\pi/2$ as desired. More generally, if all the cylinder axes form an angle $\theta$ with $\vec u$, then $\hat\theta$ will be $\theta$. If the cylinders form different angles with $\vec u$, then $\hat\theta$ will be an average angle in the sense that $\sin^2\theta$ (or equivalently $\cos^2\theta$) is averaged.
Best Answer
If you consider vectors $x_1,...,x_{80}\in\mathbb{R}^3\backslash\{0\}$, you consider the following two average vectors which are not necessarily equal:$$\frac{1}{80}\sum_{j=1}^{80}\frac{x_j}{||x_j||}\neq \frac{\frac{1}{80}\sum_{j=1}^{80}x_j}{||\frac{1}{80}\sum_{j=1}^{80}x_j||}$$ and the second might not even exist since it is very well possible that $\frac{1}{80}\sum_{j=1}^{80}x_j=0$. The first expression ( and thus Your second method) is certainly what You are looking for.