This quickly got longer than I thought it would be, and I'm sure I inadvertently skipped something. If you have any questions, feel free to ask.
Definition: (Lie Algebra) A Lie algebra is a real vector space $V$ equipped with an operation $$[\cdot,\cdot]:V\times V\to V,$$ called the bracket of the Lie algebra, such that
- $[\cdot,\cdot]$ is bilinear. That is, for $u,v,w\in V$ and $r\in\mathbb{R}$, $[u+v,w]=[u,w]+[v,w]$, $[u,v+w]=[u,v]+[u,w]$, and $[ru,v]=[u,rv]=r[u,v]$.
- $[\cdot,\cdot]$ is alternating. That is, for $v\in V$, $[v,v]=0$.
- $[\cdot,\cdot]$ satisfies the Jacobi identity. That is, for $u,v,w\in V$, $$[u,[v,w]]+[v,[w,u]]+[w,[u,v]]=0.$$
This is an abstract definition of a Lie algebra. As we can see, a Lie algebra is a vector space by definition.$^*$ But, what about the Lie algebra associated to a Lie group?
The product in the Lie algebra of a Lie group can be defined by using the typical Lie bracket $[X,Y]=XY-YX$ on vector fields, where we are treating vector fields as derivations mapping smooth functions to smooth functions. We simply extend $X_\mathrm{e}\in T_\mathrm{e}G$ to the vector field $X:g\mapsto X_g=(g)^L_*X_\mathrm{e}$, where $(g)^L_*$ is the pushforward of left multiplying by $g\in G$ (Acting on the left or right doesn't actually matter). Then, we can simply define $[X_\mathrm{e},Y_\mathrm{e}]=[X,Y]_\mathrm{e}$.
Proposition 1: The tangent space $T_\mathrm{e}G$ to the identity $\mathrm{e}$ of the Lie group $G$, equipped with the bracket operation defined in the previous paragraph, is a Lie algebra.
For the sake of helping you practice (Practice is the only way to truly understand Lie theory), I will leave this proof as an exercise. It is just a matter of checking the axioms. If you have trouble, I will be happy to help you below, in the comments.
There are several definitions of the exponential map. However, my primary choice when I go to prove something is the one I consider most useful. To give that definition, we need one more bit of terminology.
Definition: (1-parameter Subgroup) A 1-parameter subgroup on a Lie group $G$ is a Lie group homomorphism (a smooth homomorphism between Lie groups) between $\mathbb{R}$ and $G$.
Note, first and foremost, that this is NOT a subgroup. It is a homomorphism. Sometimes, it is useful to think of these as copies of $\mathbb{R}$ in $G$, and that is why the term comes up, but they are not themselves subgroups.
The set of all 1-parameter subgroups, on the other hand, does form a vector space. In fact, this vector space is isomorphic to $T_\mathrm{e}G$. The proof I know of this is rather tedious, but the intuition is pretty straightforward: there is a bijection between the tangent vectors (thinking as we did in high school of "magnitude and direction") in $T_\mathrm{e}G$ and the paths through $\mathrm{e}\in G$ with particular velocity ("magnitude and direction") at $\mathrm{e}$. With this intuition in hand, we define our exponential map.
Definition: (Exponential Map) Let $\mathfrak{g}$ be the Lie algebra of a Lie group $G$. Then, the exponential map $\exp:\mathfrak{g}\to G$ maps $X_\mathrm{e}\in\mathfrak{g}$ to $\theta(1)\in G$, where $\theta:\mathbb{R}\to G$ is the unique 1-parameter subgroup such that $\theta_*(\left.\frac{\partial}{\partial t}\right|_0)=X_\mathrm{e}$. (That is, $\theta$ is the unique 1-parameter subgroup of $G$ such that its tangent vector at $\mathrm{e}$ is $X_\mathrm{e}$.)
Thus, by definition, $\exp$ maps into $G$. But, what about the matrix exponential?
Proposition 2: The map $$\exp:\mathfrak{gl}_n(\mathbb{R})\to GL_n(\mathbb{R}),~X\mapsto\sum_{k=0}^\infty\frac{1}{k!}X^k$$ satisfies the definition of the exponential map.
Proof: Consider the map $\gamma_X:\mathbb{R}\to GL_n(\mathbb{R}),~t\mapsto\exp(tX)$, where $X\in\mathfrak{gl}_n(\mathbb{R})$ (so, $X$ can be any real $n\times n$ matrix). Note that $\gamma_X$ does, indeed, map into $GL_n(\mathbb{R})$, since $\exp(tX)^{-1}=\exp(-tX)$, so $\gamma_X(t)$ is invertible. Clearly, $\gamma_X$ is smooth, and $(\gamma_X)_*(\left.\frac{\partial}{\partial t}\right|_0)=X$. It remains to show that $\gamma_X(t+s)=\gamma_X(t)\gamma_X(s)$, to show that $\gamma_X$ is a homomorphism. But, a classic property of the matrix exponential is that $\exp((t+s)X)=\exp(tX)\exp(sX)$, so we are done. Thus, $\gamma_X$ is a 1-parameter subgroup, and $\gamma_X(1)=\exp(X)$ is the image of $X$ under the exponential map. Tombstone.
Thus, the matrix exponential is the exponential map of $GL_n(\mathbb{R})$ (and $GL_n(\mathbb{C})$, but I am sticking to reals for this explanation). Extending this, we get that the matrix exponential is the exponential map for all matrix Lie groups (matrix Lie groups are Lie subgroups of $GL_n(\mathbb{R})$ for some $n$).
Now, rather than prove to you that the matrix Lie algebra of each matrix Lie group is isomorphic to the abstract Lie algebra of that Lie group, I will show how we use the latter idea to pseudo-rigorously obtain the former, as I think this is more instructive. As before, if more rigor is requested, more will be given.
When I think about Lie algebras of Lie groups, I think of the elements of the Lie algebra as tangent vectors, which leads to the thought of "infinitesimal" group elements. For example, given the orthogonal group $O_n(\mathbb{R})$, we can think of its Lie algebra elements as imperceptibly small rotations. In other words, $X\in\mathfrak{o}_n(\mathbb{R})$ implies that, for some $\varepsilon > 0$ such that $\varepsilon^2=0$ (note that such an $\varepsilon$ doesn't exist), $(I+\varepsilon X)\,``\in"O_n(\mathbb{R})$, where $I$ is the identity matrix. But, since $A\in O_n(\mathbb{R})$ if and only if $AA^T=I$, $X\in\mathfrak{o}_n(\mathbb{R})$ "if and only if" $$(I+\varepsilon X)(I+\varepsilon X)^T=I+\varepsilon(X+X^T)+\varepsilon^2XX^T=I+\varepsilon(X+X^T)=I,$$ so $X+X^T=0$, which is the definition for $\mathfrak{o}_n(\mathbb{R})$. Thus, thinking of Lie algebra elements as tangent vectors gives us the matrix Lie algebra interpretation.
Since you have a reference-request tag, I will suggest J. Frank Adams's Lectures on Lie Groups, which provides an easily comprehendible introduction to abstract Lie groups. Also, Chapter 10 of Spivak's A Comprehensive Introduction to Differential Geometry, Volume 1 (I use the 3rd edition) is, as intended, fairly comprehensive.
$^*$For you fans of category theory, I mean that there is clearly a forgetful functor from each Lie algebra to its underlying vector space.
Best Answer
The orthogonal group is a set of matrices satisfying a particular condition.
\begin{equation} O^TO = \mathbb{I}. \end{equation}
Using matrix multiplication as the group multiplication this set satisfies the axioms for a Lie group.
All Lie groups have an associated space called a Lie algebra. A Lie algebra is a vector space with an extra structure which is the Lie bracket. The exponential map provides a map from the Lie algebra to the Lie group. This map is some times surjective, sometimes injective and sometimes neither depending on the properties of the group.
In this case the Lie algebra consists of the set of all skew-symmetric matrices. It is the Lie algebra shared by both the orthogonal group and the special orthogonal group. The exponential map maps every skew-symmetric matrix to a special orthogonal matrix (which form a subgroup of the orthogonal group). Just FYI the reason for this lack of surjectivity comes from the fact that the orthogonal group is not connected.
The Lie bracket is an operator that can act between any two matrices in the Lie algebra. Here it is just the commutator.