This is very much an applied maths question. I'm having trouble with Euler angles in the context of smartphone apps. I've been working with Android, but I would guess that the same problem arises whatever operating system one uses.
In the context of it's 3D sensors, the Android operating system is set up to calculate a rotation matrix $\mathbf{R}$ which is defined by
$$
\mathbf{R}=\left[
\begin{array}
[c]{ccc}%
E_{x} & E_{y} & E_{z}\\
N_{x} & N_{y} & N_{z}\\
G_{x} & G_{y} & G_{z}%
\end{array}
\right]
$$
where $x$, $y$ and $z$ are axes relative to the smartphone,
and where
$$
\begin{align*}
\mathbf{E} & = \left( E_{x},E_{y},E_{z}\right) =\text{a unit vector
which points East}\\
\mathbf{N} & = \left( N_{x},N_{y},N_{z}\right) =\text{a unit vector
which points North}\\
\mathbf{G} & = \left( G_{x},G_{y},G_{z}\right) =
\begin{array}
[c]{l}%
\text{a unit vector which points away from}\\
\text{the centre of the earth (Gravity vector)}
\end{array}
\text{ .}
\end{align*}
$$
Once $\mathbf{R}$ has been calculated, the Android operating system will then calculate for you the Euler angles $\phi$, $\theta$ and $\psi$ where
$$
\begin{array}
[c]{lll}%
\text{azimuth} & =\phi & =\text{rotation about }\mathbf{G}\\
\text{pitch} & =\theta & =\text{rotation about }\mathbf{E}\\
\text{roll} & =\psi & =\text{rotation about }\mathbf{N}
\end{array}
\text{ .}
$$
The relationship between $\mathbf{R}$ and $(\phi$, $\theta$, $\psi)$ is given by
$$
\mathbf{R}=\left[
\begin{array}
[c]{ccc}%
\cos\phi\cos\psi-\sin\phi\sin\psi\sin\theta & \sin\phi\cos\theta & \cos
\phi\sin\psi+\sin\phi\cos\psi\sin\theta\\
-\sin\phi\cos\psi-\cos\phi\sin\psi\sin\theta & \cos\phi\cos\theta & -\sin
\phi\sin\psi+\cos\phi\cos\psi\sin\theta\\
-\sin\psi\cos\theta & -\sin\theta & \cos\psi\cos\theta
\end{array}
\right] \text{ .}
$$
The azimuth $\phi$ is a particularly important result, because $\phi=0$ corresponds to the direction of north. However, there is a well known problem with these Euler angles, which is known as Gimbal lock. When $\theta=\pm\frac{\pi}{2}$ the co-ordinate system is denerate in the sense that the combination $\phi\pm\psi$ is defined, but the individual quantities $\phi$ and $\psi$ are not defined.
In the context of smartphone apps, this means that determining the direction of north is tricky when the device isn't lying flat on a table. The closer the device gets to being upright, the less reliable to co-ordinate system becomes. On stackoverflow.com some people recommend switching axes when $\theta$ moves away from $0$ so as to avoid Gimbal lock, but I find that that solution introduces other problems, such as jumps in $\phi$, $\theta$ and $\psi$ when the coordinate system is switched.
So my question is, instead of $\phi$, $\theta$ and $\psi$ as defined above, is there a better way of defining an angular co-ordinate system for expressing the orientation of a smartfone relative to the vectors $\mathbf{E}$, $\mathbf{N}$ and $\mathbf{G}$? In particular, is there a coordinate system that's better for defining the direction of north, that will work whether the device is upright or lying flat on a table.
Clarification Whatever the best methodology is, one result must be some kind of azimuth angle which can be explicitly calculated from the rotation matrix $R$, which represents the angle through which the device must be rotated so that it's pointing north.
Best Answer
Posting the question yesterday has focussed my thoughts on this problem, and I think that I've now come up an answer. In terms of the rotation matrix $\mathbf{R}$ that I defined in the question, I think that the best definition of the azimuth $\phi$ (i.e. the angle through which the phone must be rotated so that it's pointing north) is given by
$$ \phi=\tan^{-1}\left( \frac{E_{y}-N_{x}}{E_{x}+N_{y}}\right) \text{ .} $$
I give a full explanation below, but one of my main concerns is to address what people expect when using a smartphone. If a smartphone is lying flat on a table and there's a compass arrow pointing north in the direction of the longest edge of the phone as shown in the following image
then if the phone is rotated from its flat horizontal position into a vertical position by rotating along the short edge of the phone, the compass directions and hence the azimuth $\phi$ shouldn't change. Of course, with the phone upright the arrow that was pointing North when the phone was flat is now actually pointing away from the earth and straight into space, but people interpret that upward pointing arrow as pointing straight ahead in a horizontal direction. I've tested the above formula for $\phi$ in my android devices and as far as I can tell it works well. (NB: I've posted the relevant piece of the code that I'm using here on stackoverflow.com.)
In devising my answer, I use three angles which I call azimuth $\phi$, pitch $\theta$ and pitch axis $\psi$ $$ \begin{array} [c]{lll}% \text{azimuth} & =\phi & =\text{rotation about }\mathbf{G}\\ \text{pitch} & =\theta & =\text{rotation about axis at angle }\psi\text{ in the horizontal }\mathbf{E}\text{-}\mathbf{N}\text{ plane}\\ \text{pitch axis} & =\psi & =\text{angle in horizontal plane for pitch rotation} \end{array} $$ Whereas the three Euler angles correspond to three rotations, the $\phi$, $\theta$ and $\psi$ here only correspond to two rotations, namely
$$ \mathbf{R}=\mathbf{R}_{\mathbf{G}}\left( \phi\right) \mathbf{R} _{\mathbf{EN}}\left( \theta|\psi\right) $$
where
$$ \begin{align*} \mathbf{R}_{\mathbf{G}}\left( \phi\right) & \mathbf{=}\left[ \begin{array} [c]{ccc}% \cos\phi & \sin\phi & 0\\ -\sin\phi & \cos\phi & 0\\ 0 & 0 & 1 \end{array} \right] \\ \mathbf{R}_{\mathbf{EN}}\left( \theta|\psi\right) & \mathbf{=}\left[ \begin{array} [c]{ccc} \cos^{2}\psi+\sin^{2}\psi\cos\theta & -\sin\psi\cos\psi\left( 1-\cos \theta\right) & \sin\psi\sin\theta\\ -\sin\psi\cos\psi\left( 1-\cos\theta\right) & \sin^{2}\psi+\cos^{2}\psi \cos\theta & \cos\psi\sin\theta\\ -\sin\psi\sin\theta & -\cos\psi\sin\theta & \cos\theta \end{array} \right] \end{align*} $$ so that doing the matrix multiplication results in a matrix $\mathbf{R}$ which is given by $$ {\scriptsize \left[ \begin{array} [c]{ccc}% \cos\psi\cos\left( \phi+\psi\right) +\sin\psi\sin\left( \phi+\psi\right) \cos\theta & -\sin\psi\cos\left( \phi+\psi\right) +\cos\psi\sin\left( \phi+\psi\right) \cos\theta & \sin\left( \phi+\psi\right) \sin\theta\\ -\cos\psi\sin\left( \phi+\psi\right) +\sin\psi\cos\left( \phi+\psi\right) \cos\theta & \sin\psi\sin\left( \phi+\psi\right) +\cos\psi\cos\left( \phi+\psi\right) \cos\theta & \cos\left( \phi+\psi\right) \sin\theta\\ -\sin\psi\sin\theta & -\cos\psi\sin\theta & \cos\theta \end{array} \right]} $$ The idea of these coordinates is that there's always an Euler angle type of co-ordinate system where the roll angle is zero. If the roll angle is always zero it can be discarded, but then one needs an additional angle to specify the axis for the pitch rotation. Using Euler angles to describe the rotation of a smartphone out of the horizontal $\mathbf{E}$- $\mathbf{N}$ plane, the pitch corresponds to a rotation out of the plane along one particular axis, the roll corresponds to a rotation out of the plane along a second perpendicular axis, and rotations along other axes are described by a combination of pitch and roll. With the $\left( \phi,\theta,\psi\right) $ coordinates presented here there's just one angle $\theta$ to describe rotation out of the plane, but then the angle $\psi$ is required to describe which axis to rotate about. In the context of smartphones, the pitch angle $\theta$ just represents the phone's tilt out of the horizontal plane, irrespective of which axis it has been tilted along.
Thinking like this, any 3D rotation can be constructed from a rotation $\mathbf{R}_{\mathbf{G}}\left( \phi\right) $ about the vertical axis $\mathbf{G}$, and a second rotation $\mathbf{R}_{\mathbf{EN}}\left( \theta|\psi\right) $ to take the device of the plane perpendicular to $\mathbf{G}$. With $\mathbf{R}$ specified above as a function of $\phi$, $\theta$ and $\psi$, it is straightforward to show that $$ \begin{align*} \cos\phi\left( 1+\cos\theta\right) & =E_{x}+N_{y}\\ \sin\phi\left( 1+\cos\theta\right) & =E_{y}-N_{x} \end{align*} $$ and hence the $\tan^{-1}$ formula for $\phi$ given above. It is worth noting that $$ \begin{align*} \sin\left( \phi+2\psi\right) \left( 1-\cos\theta\right) & =-E_{y}-N_{x}\\ \cos\left( \phi+2\psi\right) \left( 1-\cos\theta\right) & =E_{x}-N_{y}\text{ .} \end{align*} $$ Also note that if $\psi\rightarrow\psi+\pi$ and $\theta\rightarrow-\theta$ then $\mathbf{R}$ is unchanged. Whereas the Euler angle system is degenerate when its $\theta=\pm\frac{\pi}{2}$, this $\left( \phi,\theta,\psi\right) $ co-ordinate system is degenerate at $\theta=0$ and $\theta=\pi$. The $\theta=0$ degeneracy occurs when the device is lying flat on the table facing upwards, and in that situation no rotation out of the horizontal $\mathbf{E}$-$\mathbf{N}$ plane is required so $\psi$ is undefined. However, $\phi$ is well defined, so if $\phi$ is the goal then the fact that $\psi$ is undefined does not matter.
The $\theta=\pi$ degeneracy occurs when the device is lying face down on the table, when $E_x+N_y=E_y-N_x=0$, and this is a more interesting case. In this situation, $\phi$ is undefined, although $\phi+2\psi$ is defined. The purpose of this $\left(\phi,\theta,\psi\right) $ co-ordinate system is to capture the idea that people expect the compass direction (i.e. the azimuth $\phi$) to be unchanged when rotating a phone from a horizontal $\theta=0$ position to a vertical $\theta=\frac{\pi}{2}$ position. However, if the rotation continues and $\theta$ increases beyond $\frac{\pi}{2}$, then as the phone approaches the horizontal $\theta=\pi$ upside down position, a compass arrow that had been pointing north when the device was horizontal will now be pointing south if it's direction remains fixed on the smartphone display. So as the $\theta=\pi$ degeneracy is approached, the smartphone is no longer behaving in an appropriate way. One way of dealing with this would be to switch co-ordinate systems at some point between $\frac{\pi}{2}$ and $\pi$ which could alter the direction of the arrow by $\pi$ and make the smartphone behave in an appropriate way again. However, it is worth noting that almost all the time when people are using their phones $\left\vert \theta\right\vert\le\frac{\pi}{2}$, so the inappropriate behaviour near $\theta=\pi$ when the phone is upside down should not be a problem in practice.
I've searched the internet and had a look at the article Shuster, M., "A Survey of attitude representations", Journal of the Astronautical Sciences 41(4):1993 that was suggested in another answer, but I haven't been able to find any references to the $\left( \phi,\theta,\psi\right) $ co-ordinate system defined here. In such a mature area as 3D rotations, it seems unlikely that no one has defined rotations in this way before? However, the emergence of smartphones has provided a new application for rotation matrices, so perhaps there haven't previously been applications where this type of co-ordinate system has been appropriate?