I have $x= \exp(At)$ where $A$ is a matrix. I would like to find derivative of $x$ with respect to each element of $A$. Could anyone help with this problem?
Matrix Calculus – Derivative of Matrix Exponential with Respect to Each Element
derivativesmatricesmatrix exponentialmatrix-calculus
Related Solutions
Let $f : \mathbb R^{n \times n} \to \mathbb R^{n \times n}$ be defined by $f (X) = \exp(X)$. Hence,
$\begin{array}{rl} f (X + h V) &= \exp(X + h V)\\ &= I_n + (X + h V) + \frac{1}{2!} (X + h V)^2 + \frac{1}{3!} (X + h V)^3 + \cdots\\ &= I_n + X + h V + \frac{1}{2!} X^2 + \frac{h}{2!} (X V + V X) + \frac{h^2}{2!} V^2 + \frac{1}{3!} X^3 + \\ &\,\,\,\,\,+ \frac{h}{3!} (X^2 V + X V X + V X^2) + \frac{h^2}{3!} (X V^2 + V X V + V^2 X) + \frac{h^3}{3!} V^3 + \cdots\\ &= f (X) + h \left( V + \frac{1}{2!} (X V + V X) + \frac{1}{3!} (X^2 V + X V X + V X^2) + \cdots\right) + \cdots\end{array}$
Thus, the directional derivative of $f$ in the direction of $V$ at $X$ is given by
$$\begin{array}{rl} D_V f (X) &= \displaystyle\lim_{h \to 0} \frac{1}{h} \left( f (X + h V) - f (X) \right)\\ &= V + \frac{1}{2!} (X V + V X) + \frac{1}{3!} (X^2 V + X V X + V X^2) + \cdots\end{array}$$
We then write
$$D_V f (X) = M_0 + \frac{1}{2!} M_1 + \frac{1}{3!} M_2 + \frac{1}{4!} M_3 + \cdots$$
where
$$\begin{array}{rl} M_0 &= V\\ M_1 &= X V + V X =: \{X,V\}\\ M_2 &= X^2 V + X V X + V X^2\\ &\vdots\\ M_k &= \displaystyle\sum_{i=0}^k X^{k-i} V X^i\end{array}$$
Define some auxiliary matrices and calculate their differentials $$\eqalign{ B &= \tfrac{1}{\gamma}(XY-A) &\implies dB=\tfrac{1}{\gamma}(X\,dY+dX\,Y) \cr C &= -B\odot B &\implies dC=-2B\odot dB \cr E &= \exp(C) &\implies dE=E\odot dC \cr F &= -\tfrac{2}{\gamma}B\odot E \cr }$$ where the symbol $(\odot)$ represents the elementwise/Hadamard product, and the exp() function is understood to be applied elementwise.
Write the objective function in terms of these new variables.
Then calculate its differential and gradients.
$$\eqalign{
\Phi &= J:E \cr
d\Phi
&= J:dE \cr
&= J:(E\odot dC) \cr
&= E:dC \cr
&= E:(-2B\odot dB) \cr
&= -2(B\odot E):dB \cr
&= \gamma F:\tfrac{1}{\gamma}(X\,dY+dX\,Y) \cr
&= X^TF:dY + FY^T:dX \cr
\frac{\partial\Phi}{\partial Y} &= X^TF, \quad
\frac{\partial\Phi}{\partial X} = FY^T \cr
}$$
where $J\in{\mathbb R}^{m\times n}$ is a matrix of all ones, and (:) represents the trace/Frobenius product, i.e.
$$\eqalign{ A:B = {\rm Tr}(A^TB)}$$
The cyclic property of the trace allows the terms to be rearranged in various ways.
$$\eqalign{
A:BC = AC^T:B = B^TA:C
}$$
Finally, the Hadamard and Frobenius products commute with themselves and each other
$$\eqalign{
A:B &= B:A \cr
B\odot C &= C\odot B \cr
A:(B\odot C) &= (A\odot B):C \cr
}$$
Best Answer
Considering the expression $x = \exp(tA)$ I can think of two derivatives.
First, the derivative with respect to the real variable $t$ of the matrix-valued function $t \mapsto \exp(tA)$. Here the result is easily derived from direct calculation of the series definition of the matrix exponential: \begin{align} \frac{d}{dt} \exp(tA) &= \frac{d}{dt} \left[ I+tA+\frac{1}{2}t^2A^2+\frac{1}{3!}t^3A^3+ \cdots\right] \\ &= A+tA^2+\frac{1}{2}t^2A^3+ \cdots \\ &= A\exp(tA) \end{align} Thus, $\frac{d}{dt} \exp(tA) = A\exp(tA)$. (Edited to fix typo)
Second, we can differentiate with respect to the component $A_{ij}$ of $A = \sum A_{ij} E_{ij}$ where $E_{ij}=e_ie_j^T$ is the matrix which is zero except in the $ij$-th spot where there is a $1$. In other words, $(E_{ij})_{kl} = \delta_{ik}\delta_{jl}$. I'll look at the derivative as a directional derivative essentially: calculate the difference along the $E_{ij}$ direction: considering $f(t,A)=\exp(tA)$ $$ \frac{\partial f}{\partial A_{ij}}= \lim_{h \rightarrow 0}\frac{1}{h} \left[\exp(t(A+hE_{ij}))-\exp(tA)\right]$$ I expect this can be simplified.
Ok, the matrix exponential satisfies the Baker-Campbell-Hausdorf relation: $$ \exp(A)\exp(B) = \exp\left(A+B+ \frac{1}{2}[A,B] + \cdots\right)$$ From this we derive the Zassenhaus formula, $$ \exp(A+B) = \exp(A)\exp(B)\exp\left(-\frac{1}{2}[A,B] + \cdots\right) $$ I'll use this to simplify $\exp( t(A+hE_{ij})) = \exp\left(tA+ thE_{ij}\right)$ $$ \exp\left(tA+ thE_{ij}\right) = \exp(tA)\exp\left( thE_{ij}\right) \exp\left( -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right)$$ hence $$ \exp\left(tA+ thE_{ij}\right) = \exp(tA)\exp\left( thE_{ij} -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right)$$ where I am omitting terms with $h^2,h^3,\dots$ as those vanish in the limit and I am also omitting terms with nested commutators of $A$ so the answer below is just the first couple terms in an infinite series flowing from the BCH relation. \begin{align} \frac{\partial f}{\partial A_{ij}}&= -\lim_{h \rightarrow 0}\frac{1}{h} \left[\exp(tA)-\exp(t(A+hE_{ij}))\right] \\ &=-\lim_{h \rightarrow 0}\frac{1}{h} \left[\exp(tA)-\exp(tA)\exp\left( thE_{ij} -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right) \right] \\ &=-\exp(tA)\lim_{h \rightarrow 0}\frac{1}{h} \left[I-\exp\left( thE_{ij} -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right) \right] \\ &=-\exp(tA)\lim_{h \rightarrow 0}\frac{1}{h} \left[I-I-thE_{ij} +\frac{1}{2}[tA,thE_{ij}]+ \cdots \right] \\ &=-\exp(tA)\left[-tE_{ij} +\frac{t^2}{2}[A,E_{ij}]+ \cdots \right]. \end{align} Note the terms linear in $h$ do survive the limit and there are such terms (indicated by the $+ \cdots$) stemming from $[tA,[tA,hE_{ij}]]$ and $[tA,[tA,[tA,hE_{ij}]]]$ etc. Now, you can calculate: $[A,E_{ij}] = \sum_{k=1}^n \left(A_{ki}E_{kj}-A_{jk}E_{ik} \right)$ so, $$ \frac{\partial f}{\partial A_{ij}} = -\exp(tA) \left[-tE_{ij}+ \frac{t^2}{2}\left(A_{ki}E_{kj}-A_{jk}E_{ik} +\cdots \right)\right]$$ For what it's worth, you can simplify the nested commutator: $$ [A,[A,E_{ij}]] = \sum_{k,l=1}^n \left( A_{lk}A_{ki}E_{lj}-2A_{ki}A_{jl}E_{kl}+A_{jl}A_{lk}E_{ik} \right)$$ Or, in Einstein notation, $$ [A,[A,E_{ij}]] = (A^2)_{li}E_{lj}-2A_{ki}A_{jl}E_{kl}+(A^2)_{jk}E_{ik}.$$ Anyway, I hope this helps. Notice if $i=j$ and $A$ is diagonal or if simply $[A,E_{ij}]=0$ then we obtain: $$ \frac{\partial }{\partial A_{ij}} \exp(tA) = t\exp(tA)E_{ij}.$$ (I fixed the sign-errors, sorry for all weridly placed minus signs: 9-18-21).