I don't think you need the general Jacobian $\frac{\partial}{\partial \mathbf p}\exp(\hat{\mathbf p})$, but only the much simpler Jacobian $\left.\frac{\partial}{\partial \mathbf p}\exp(\hat{\mathbf p})\right|_{\mathbf p=\mathbf 0}$ with $\mathbf p$ being at the identity.
Background
The group of 3d rotations (SO3) is a matrix lie group. Thus, in general we have the matrix exponential:
$$\exp(\mathtt M) := \sum_{k\ge 0} \frac{1}{k!} \mathtt M^k$$
which maps an element of the matrix Lie algebra onto the set of the matrix Lie group elemets.
Furthermore we have a function
$$\hat \cdot: \mathbb R^n \rightarrow \mathbb R^{m\times m}, \quad \hat{\mathbf a} = \sum_{k=0}^n a_i\mathtt G_i$$
which maps an $n$-vector onto the set of matrix Lie algebra elements (=matrices).
Thus, for SO3 the generators are:
$$ \mathtt G_1 = \begin{bmatrix}
0 & 0 & 0 \\
0 & 0 & -1 \\
0 & 1 & 0
\end{bmatrix},\quad
\mathtt G_2 =\begin{bmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
-1 & 0 & 0
\end{bmatrix},\quad
\mathtt G_3 =\begin{bmatrix}
0 & -1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0
\end{bmatrix}
$$
Derivative at the identity
For matrix Lie groups, it can be shown that
$$ \left.\frac{\partial}{\partial a_k}\exp(\hat {\mathbf a})\right|_{\mathbf a=\mathbf 0} = \mathtt G_k.$$
Thus for SO3
$$ \left.\frac{\partial}{\partial \mathbf p}\exp(\hat {\mathbf p})\right|_{\mathbf p=\mathbf 0} =
\begin{bmatrix}\mathtt G_1 & \mathtt G_2 & \mathtt G_3\end{bmatrix}$$
we receive a $3$d row vector of $3\times 3$ matrices, hence a $3\times 3 \times 3$ Jacobian tensor. (Alternatively, one could stack the columns of $\mathtt G_i$ into a 9-vectors and one would receive a $9\times 3$ Jacobian matrix.)
Background: Least square optimization
Let us assume we would like to minimize the function $F:\mathbb R^n \rightarrow \mathbb R$ with respect to a Euclidean vector $\mathbf x$. Furthermore let use assume we have a least square problem with
$$F = f(\mathbf x)^\top f(\mathbf x)$$
Following Gauss-Newton we repeatedly solve for an update $\delta$
$$ \frac{\partial f}{\partial \mathbf x^{(m)}}^\top\frac{\partial f}{\partial \mathbf x^{(m)}} \delta = -\frac{\partial f}{\partial \mathbf x^{(m)}}^\top f(\mathbf x^{(m)})$$
and update our estimate
$$ \mathbf x^{(m+1)}=\delta + \mathbf x^{(m)}$$
Least square optimization on matrix Lie groups
This scheme is only valid for Euclidean vector spaces and need to be adapted for matrix Lie groups. Especially, we calculate the derivative in the tangent space around the identity:
$$\mathtt J :=\left.\frac{\partial f(\exp(\hat{\mathbf p})\cdot\mathtt R^{(m)})}{\partial \mathbf p} \right|_{\mathbf p =\mathbf 0}$$
(Can be calculated from the generators and using the chain rule.)
Then we solve for $\delta$:
$$ \mathtt J^\top\mathtt J \delta = -\mathtt J^\top f(\mathtt R^{(m)})$$
Finally we adapt the update rule:
$$ \mathtt R^{(m+1)}= \exp(\hat\delta)\cdot \mathtt R^{(m)}.$$
$\def\vec#1{\overrightarrow{#1}}$
Let
$$\vec{u}=\frac{1}{\Vert\vec{AB}\Vert}\vec{AB}=\frac{1}{3\sqrt{5}}\left[\matrix{-5\cr 4\cr 2}\right].$$
Consider any unit vector that is orthogonal to $\vec{u}$, for example
$$
\vec{v}=\frac{1}{\sqrt{5}}\left[\matrix{0\cr -1\cr 2}\right]
$$
Finally, define $\vec{w}=\vec{u}\wedge \vec{v}$,
$$
\vec{w}=\frac{1}{3}\left[\matrix{2\cr 2\cr 1}\right]
$$
So that $(\vec{u}, \vec{v},\vec{w})$ is a direct orthonormal basis of $\mathbb{R}$. The rotation $R$ around $\vec{u}$ of angle $\theta=3\pi/2$ acts on this basis as follows
$$R(\vec{u})=\vec{u},\quad R(\vec{v})=-\vec{w},\quad R(\vec{w})=\vec{v}.$$
Now, if $(x,y,z)$ are the coordinates of a point $M$ in the canonical basis of $\mathbb{R}^3$ then then
$$\eqalign{\vec{OM}&=(\vec{OM}\cdot \vec{u})\vec{u}+(\vec{OM}\cdot \vec{v})\vec{v}+(\vec{OM}\cdot \vec{w})\vec{w}\cr
&=\frac{-5x+4y+2z}{3\sqrt{5}}\vec{u}+
\frac{-y+2z}{\sqrt{5}}\vec{v}+\frac{2x+2y+z}{3}\vec{w}
}
$$
and consequently, its image $M'$ under the the skew rotation $R$ followed by the translation of vector $\vec{p}=5\vec{u}$, is given by
$$\eqalign{\vec{OM'}&=5\vec{u}+
\frac{-5x+4y+2z}{3\sqrt{5}}\vec{u}-
\frac{-y+2z}{\sqrt{5}}\vec{w}+\frac{2x+2y+z}{3}\vec{v}
}
$$
The final step is just to replace the coordinates of $\vec{u}, \vec{v},\vec{w}$ in the previous expression to obtain the coordinates $(x',y',z')$ of $M'$ in terms of those of $M$, this is simple algebra.$\qquad\square$
Best Answer
Let $\mathbf{T}$ be the translation matrix, and $\mathbf{R}$ the rotation matrix: $$\mathbf{T} = \left[\begin{matrix} 1 & 0 & 0 & T_x \\ 0 & 1 & 0 & T_y \\ 0 & 0 & 1 & T_z \\ 0 & 0 & 0 & 1 \end{matrix}\right], \quad \mathbf{R} = \left[ \begin{matrix} \cos\alpha & \sin\beta \sin\alpha & \cos\beta \sin\alpha & 0 \\ 0 & \cos\beta & -\sin\beta & 0 \\ -\sin\alpha & \sin\beta \cos\alpha & \cos\beta \cos\alpha & 0 \\ 0 & 0 & 0 & 1 \end{matrix}\right]$$ Because rotations are combined by multiplying the matrices with the applied transform on the left (i.e., oldest rightmost, newest leftmost), the combined transformation matrix $\mathbf{M}_{T,R}$ if the translation is done first and rotation second is $$\mathbf{M}_{T,R} = \mathbf{R} \mathbf{T} = \left[\begin{matrix} \cos\alpha & \sin\beta \sin\alpha & \cos\beta \sin\alpha & T_x \cos\alpha + T_y \sin\beta \sin\alpha + T_z \cos\beta \sin\alpha \\ 0 & \cos\beta & -\sin\beta & T_y \cos\beta - T_z \sin\beta \\ -\sin\alpha & \sin\beta \cos\alpha & \cos\beta \cos\alpha & -T_x \sin\alpha + T_y \sin\beta \cos\alpha + T_z \cos\beta \cos\alpha \\ 0 & 0 & 0 & 1 \end{matrix}\right]$$ On the other hand, if the rotation is done first and translation second, the transformation matrix is $\mathbf{M}_{R,T}$, $$\mathbf{M}_{R,T} = \mathbf{T} \mathbf{R} = \left[\begin{matrix} \cos\alpha & \sin\beta \sin\alpha & \cos\beta \sin\alpha & T_x \\ 0 & \cos\beta & -\sin\beta & T_y \\ -\sin\alpha & \sin\beta \cos\alpha & \cos\beta \cos\alpha & T_z \\ 0 & 0 & 0 & 1 \end{matrix}\right]$$ Any combination (product) of pure rotation and pure translation matrices is invertible. The key thing to note is that inverting the result is the same as inverting each matrix, and the order; see e.g. the Wikipedia article on matrix inverse. That is, $$\mathbf{M}_{T,R}^{-1} = \left( \mathbf{R} \mathbf{T} \right)^{-1} = \mathbf{T}^{-1} \mathbf{R}^{-1}$$ and $$\mathbf{M}_{R,T}^{-1} = \left( \mathbf{T} \mathbf{R} \right)^{-1} = \mathbf{R}^{-1} \mathbf{T}^{-1}$$ Pure rotation matrices are orthonormal, and their inverse is their transpose, $$\mathbf{R}^{-1} = \mathbf{R}^{T} = \left[\begin{matrix} \cos\alpha & 0 & -\sin\alpha & 0 \\ \sin\beta \sin\alpha & \cos\beta & \sin\beta \cos\alpha & 0 \\ \cos\beta \sin\alpha & -\sin\beta & \cos\beta \cos\alpha & 0 \\ 0 & 0 & 0 & 1 \end{matrix}\right]$$ see e.g. Wikipedia orthogonal matrix article. For translation matrices, you negate the translation vector (rightmost column, except for the fourth element in the lower right corner of the matrix), i.e. $$\mathbf{T}^{-1} = \left [ \begin{matrix} 0 & 0 & 0 & -T_x \\ 0 & 0 & 0 & -T_y \\ 0 & 0 & 0 & -T_z \\ 0 & 0 & 0 & 1 \end{matrix} \right ]$$ You can verify this by inverting $\mathbf{T}$ by hand.
Combining all of the above, we see that $$\mathbf{M}_{T,R}^{-1} = \left( \mathbf{R} \mathbf{T} \right)^{-1} = \mathbf{T}^{-1} \mathbf{R}^{-1} = \left[\begin{matrix} \cos\alpha & 0 & -\sin\alpha & -T_x \\ \sin\beta \sin\alpha & \cos\beta & \sin\beta \cos\alpha & -T_y \\ \cos\beta \sin\alpha & -\sin\beta & \cos\beta \cos\alpha & -T_z \\ 0 & 0 & 0 & 1 \end{matrix}\right]$$ and $$\mathbf{M}_{R,T}^{-1} = \left( \mathbf{T} \mathbf{R} \right)^{-1} = \mathbf{R}^{-1} \mathbf{T}^{-1} = \left[\begin{matrix} \cos\alpha & 0 & -\sin\alpha & -T_x \cos\alpha + T_z \sin\alpha \\ \sin\beta \sin\alpha & \cos\beta & \sin\beta \cos\alpha & -T_x \sin\beta \sin\alpha - T_y \cos\beta - T_z \sin\beta \cos\alpha \\ \cos\beta \sin\alpha & -\sin\beta & \cos\beta \cos\alpha & -T_x \cos\beta \sin\alpha + T_y \sin\beta - T_z \cos\beta \cos\alpha \\ 0 & 0 & 0 & 1 \end{matrix}\right]$$
Thus, we can easily see the error in OP's inversion: they forgot to also invert the order of the matrices in the product.