Yes, the first order approximation using the adjoint is correct.
It is easyer to see if one interpret members of the Lie algebra as minimal vectors $\mathbf{x}$ instead of square matrices $\widehat{\mathbf{x}}$. Thus, we define the Lie bracket as $[\mathbf{a},\mathbf{b}]:=\widehat{\mathbf{a}}\cdot\widehat{\mathbf{b}}-\widehat{\mathbf{b}}\cdot\widehat{\mathbf{a}}$.
In case of SO3, it is simply the cross product: $[\mathbf{a},\mathbf{b}]=\mathbf{a}\times\mathbf{b}$. However, a Lie bracket in such a vector from exists for all other matrix Lie groups too.
Accordingly, the BCH-Formular is now defined as $\text{bch}(\mathbf{a},\mathbf{b}):=\log(\exp(\widehat{\mathbf{a}})\exp(\widehat{\mathbf{b}}))^\vee$
For instance, as a third order approximation, we get:
$$\left. \frac{\partial}{\partial \mathbf{x}} \log(\mathtt{A}\exp(\mathbf{x})\mathtt{B})^\vee\right|_{\mathbf{x}=\mathbf{0}} $$
$$ =\left.\frac{\partial}{\partial \mathbf{x}} \log\left(\exp(\widehat{\mathtt{Ad}_\mathtt{A}\mathbf{x}})\mathtt{A}\mathtt{B}\right)^\vee\right|_{\mathbf{x}=\mathbf{0}}$$
$$=\left.\frac{\partial}{\partial \mathbf{x}} \log\left(\exp(\widehat{\mathtt{Ad}_\mathtt{A}\mathbf{x}})\exp(\widehat{\mathbf{c}})\right)^\vee\right|_{\mathbf{x}=\mathbf{0}}$$
$$= \left.\frac{\partial}{\partial \mathbf{x}} \text{bch}(\mathtt{Ad}_\mathtt{A}\mathbf{x},\mathbf{c})\right|_{\mathbf{x}=\mathbf{0}}$$
$$\approx \frac{\partial}{\partial \mathbf{x}}\left( \mathtt{Ad}_\mathtt{A}\mathbf{x}+\mathbf{c}
+ \frac{1}{2}[\mathtt{Ad}_\mathtt{A}\mathbf{x},\mathbf{c}]+\frac{1}{12}([\mathtt{Ad}_\mathtt{A}\mathbf{x},[\mathtt{Ad}_\mathtt{A}\mathbf{x},\mathbf{c}]]+ [\mathbf{c},[\mathbf{c},\mathtt{Ad}_\mathtt{A}\mathbf{x}]])\right)_{\mathbf{x}=\mathbf{0}}$$
$$ =\left.\left(\frac{\partial \mathbf{y}}{\partial\mathbf{y}} + \frac{1}{2}\cdot\frac{\partial [\mathbf{y},\mathbf{c}]}{\partial \mathbf{y}}\right|_{\mathbf{y}=\mathtt{Ad}_\mathtt{A}\mathbf{0}}
+\frac{1}{12}\left(\frac{\partial[\mathbf{y},[\mathbf{y},\mathbf{c}]}{\partial\mathbf{y}}-\frac{\partial[\mathbf{c},[\mathbf{y},\mathbf{c}]}{\partial \mathbf{y}}\right)_{\mathbf{y}=\mathtt{Ad}_\mathtt{A}\mathbf{0}}\right)
\left.\frac{\partial \mathtt{Ad}_\mathtt{A}\mathbf{x}}{\partial \mathbf{x}}\right|_{\mathbf{x}=\mathbf{0}}$$
$$= \left(\mathtt{I}
+ \left.\frac{1}{2}\cdot\frac{\partial [\mathbf{y},\mathbf{c}]}{\partial \mathbf{y}}\right|_{\mathbf{y}=\mathbf{0}} + \frac{1}{12}
\left(\frac{\partial [\mathbf{y},[\mathbf{0},\mathbf{c}]]}{\partial \mathbf{y}}+
\frac{\partial [\mathbf{0},[\mathbf{y},\mathbf{c}]]}{\partial \mathbf{y}}
-\left.\frac{\partial [\mathbf{c},\mathbf{w}]}{\partial \mathbf{w}}\right|_{\mathbf{w}=[\mathbf{0},\mathbf{c}]}
\left.\frac{\partial [\mathbf{y},\mathbf{c}]}{\partial \mathbf{y}}
\right|_{\mathbf{y}=\mathbf{0}}
\right)\right)
\mathtt{Ad}_\mathtt{A}\nonumber$$
$$= \left(\mathtt{I}
+ \left.\frac{1}{2}\cdot\frac{\partial [\mathbf{y},\mathbf{c}]}{\partial \mathbf{y}}\right|_{\mathbf{y}=\mathbf{0}} + \frac{1}{12}\left.\frac{\partial [\mathbf{w},\mathbf{c}]}{\partial \mathbf{w}}\right|_{\mathbf{w}=\mathbf{0}}
\left.\frac{\partial [\mathbf{y},\mathbf{c}]}{\partial \mathbf{c}}\right|_{\mathbf{y}=\mathbf{0}}
\right)\mathtt{Ad}_\mathtt{A}$$
$$= \left(\mathtt{I}
+ \left.\frac{1}{2}\cdot\frac{\partial [\mathbf{y},\mathbf{c}]}{\partial \mathbf{y}}\right|_{\mathbf{y}=\mathbf{0}} + \frac{1}{12}\left(
\left.\frac{\partial [\mathbf{y},\mathbf{c}]}{\partial \mathbf{c}}\right|_{\mathbf{y}=\mathbf{0}} \right)^2
\right)\mathtt{Ad}_\mathtt{A}$$
As to question 1, I'm not sure I'd qualify it as a transform, it seems purely notational to me. Consider the vector space $\mathbb{R}^n$, then the vector $\vec{x} = (x_1\ x_2\ \ldots \ x_n)^T \in \mathbb{R}^n$ is shorthand for
$$\vec{x} = \sum_i x_i \hat{e}_i$$
where $\{\hat{e}_i\}$ is a basis of $\mathbb{R}^n$. Your function, $\hat{\cdot}$, operates in the same manner. In other words, what you think of as your vector is just shorthand.
As to question 2, most likely yes, but it will be basis and Lie algebra dependent.
The confusion you mentioned is really a restatement that unless $m = n^2$, the basis does not span $\mathcal{M}^{n\times n}(\mathbb{R})$ which is okay as $SO(3) \subset \mathcal{M}^{3\times 3}(\mathbb{R})$. Although, if I remember correctly, the basis for $SU(2)$ spans $\mathcal{M}^{2\times 2}(\mathbb{C})$.
Best Answer
Let $\gamma$ be the integral curve in $G$ of the left invariant vector field associated to $X$, i.e. $V(Y) = YX$. This means $\gamma(0) = e$ and $\gamma'(t) = \gamma(t) X$ for $Y \in G$. Then $\gamma$ is unique (even as considered a curve in $GL(n)$) by theorems of ODE but it is easy to see that the curve $\exp(tX)$ satisfies the defining properties of $\gamma$ so must be $\gamma$. In particular, $\exp X \in G$.