Consider the geometric interpretation of an orthogonal matrix, a projection matrix, a (Householder) reflector, or even just matrix-vector multiply in general.
A matrix takes a vector from a vector space (after a basis has been fixed) and performs a scaling, rotation, reflection, shear, projection, or a combination of these. This can include affine transformations as well. An orthogonal matrix represents a rotation. A projection matrix represents the projection of a vector onto a subspace. A Householder reflector reflects a vector onto an axis where a coordinate can become zero. A Givens matrix is a rotation with the same effect.
My question is, what is a geometric interpretation of the exponential operator?
For example, if $P$ is a projection matrix, then it represents the shadow of a vector onto a subspace. The property $P^2=P$ becomes obvious. But what is $e^P$? What does this represent geometrically? Similarly, thinking of an orthogonal matrix as a rotation, some of its properties become very obvious. It is always invertible because rotations can always be undone. It always preserves the Euclidean norm of a vector because a rotation cannot change the length of a vector, and so on. But what does the exponential of an orthogonal matrix represent?
Addendum
After looking at some comments and counter-comments posted below, let me clarify. After the initial setup (fields, building a vector space on top of it, selecting a basis, etc.) a matrix represents a linear transformation. If you imagine all possible linear transformations in $\mathbb{R}^3$ or $\mathbb{R}^2$, it turns out that they represent only a handful of geometric transformations such as dilation, reflection, rotation, shear, and so on. All linear transformations are some combination of these. In fact, the singular value decomposition's existence for any matrix tells us that every single linear transformation can be thought of as a rotation, dilation, followed by a rotation.
For some matrices, the geometric transformation is easy to see and easy to explain. For example, an orthogonal matrix represents a rotation. Thinking about orthogonal matrices like this, it is "obvious" that it must always be invertible, it must always preserve the Euclidean norm, the determinant must be $\pm1$ because rotations will preserve volume, etc. A projection matrix collapses a vector onto a subspace and it becomes apparent that $P^2$ must always be equal to $P$ for any projection matrix $P$.
My question is that for a given matrix $A$, what does $e^A$ represent? Does it represent something? Is there a general statement that can be made, like "the exponential always maps a matrix to an orthogonal matrix" or "the exponential of a matrix is always a projection followed by a dilation" or something? If not, then when can we say something and what can we say? For example, the exponential of any matrix is always invertible, but why? This is easy to see algebraically by what happens geometrically? Another example, the exponential of a dilation is another dilation where the new dilation factor is the exponential of the old dilation factor. Well, what if I take the exponential of a rotation? It must be a dilation/rotation/shear/reflection or some combination, but which is it in general? What about the exponential of a projection? Why and how does the exponential turn a projection into an invertible transformation even when the projection itself is not invertible? And why does the determinant of the exponential equal the exponential of the trace?
Best Answer
For a nice geometric interpretation, instead of just exponentiating a matrix $A$, we would first multiply $A$ by a real scalar $t$ and then exponentiate, i.e. we look at $e^{tA}$.
Now if you read on elementary linear algebra carefully, you would notice that there is no concept of "gradual" linear transformation. For instance, while we would have the concept of "gradually rotating a plane from $0$ to $90$ degree" way back in secondary school, we don't have such a concept in linear algebra. We instead have a matrix that rotates the plane by $90$ degree in an instant step. The map $f:\Bbb R\to GL(n,\Bbb R), f(t)= e^{tA}$, provides us a formal way to discuss gradual linear transformations.
As an example, consider the matrix
$$A=\begin{bmatrix} 0 & -1\\ 1 & 0\\ \end{bmatrix},$$ which has exponential $$e^{tA}= \begin{bmatrix} \cos t & -\sin t\\ \sin t & \cos t\\ \end{bmatrix}.$$ You should recognise this as the rotation matrix acting on the plane. As you let $t$ increases continuously, you can see that $e^{tA}$ is a gradual rotation acting on the plane.
When you define such a gradual linear transformation using a matrix $A$, $A$ would be known as the infinitesimal generator of the gradual linear transformation. By generator, we mean that given any differentiable group homomorphism $g:\Bbb R\to GL(n,\Bbb R)$ (which you can interpret as a gradual linear transformation on $\Bbb R^n$ like above), there is a unique square matrix $B$ such that $g(t)=e^{tB}$, and $B$ would generate the gradual linear transformation $g$. By infinitesimal, we mean that $B$ is obtained by calculating the derivative of $g$ at $0$, and $g'(0)=B$.
The precise geometric meaning of $e^A$ depends on what properties $A$ has of course. I can't tell what $e^A$ is if you give me an arbitrary $A$, but there are some famous examples where a property of $A$ makes $e^A$ belong to some well-known class of matrices:
if $A$ is skew-symmetric, then $e^A$ is an orthogonal matrix;
if $A$ has zero trace, then $e^A$ has determinant $1$;
if $A$ is partitioned into blocks, $$A=\begin{bmatrix} B & C\\ D & E\\ \end{bmatrix},$$ with $B$ being a $p$ by $p$ matrix, $C$ being a $p$ by $q$ matrix, $D$ being a $q$ by $p$ matrix, $E$ being a $q$ by $q$ matrix, and $B$, $E$ are skew-symmetric, and transpose of $C$, $C^T$, equals $D$, then $e^A$ satisfies the equation $$(e^A)^T\begin{bmatrix} I_p & 0_{p\times q}\\ 0_{q\times p} & -I_q\end{bmatrix}e^A=\begin{bmatrix} I_p & 0_{p\times q}\\ 0_{q\times p} & -I_q\end{bmatrix},$$ where $0_{m\times n}$ is the $m$ by $n$ matrix, $I_n$ is the $n$ by $n$ identity matrix.
You can see more of these relations between $A$ and $e^A$ if you study matrix lie groups.