[Math] the dot product between a vector of matrices

inner-productslinear algebranotationprobability

There is a notation used in many sources (e.g. Wikipedia: http://en.wikipedia.org/wiki/Exponential_family) for the natural parameters of exponential family distributions which I do not understand, and I cannot find a description of.

With vector parameters and variables, the exponential family form has the dot product between the vector natural parameter, ${\boldsymbol\eta}({\boldsymbol\theta})$ and the vector sufficient statistic, ${\mathbf{T}}({\mathbf{x}})$, in the exponent. i.e. $e^{{\boldsymbol\eta}({\boldsymbol\theta})^{\top}{\mathbf{T}}({\mathbf{x}})}$.

However, many examples of these parameters for different distributions are vectors composed of matrices & vectors. E.g. the multivariate Normal distribution has parameter $[\Sigma^{-1}\mu\space\space-\frac{1}{2}\Sigma^{-1}]$ and sufficient statistic $[\mathbf{x}\space\space\mathbf{xx^{\top}}]$.

So what are these "vectors" and moreover, how is the dot product between them defined? Does this notation have a name?

Best Answer

I believe you're supposed to "vectorize" the matrix, i.e. rearrange into a $n^2 \times 1$ vector.

Equivalently, you can take $A\cdot B = \textrm{tr}(A^TB)$ as the definition.

EDIT - Example:

If $A = \left(\begin{array}{cc}a & b \\ c & d \end{array}\right)$, $B = \left(\begin{array}{cc}e & f \\ g & h \end{array}\right)$, then

$$ A^TB = \left(\begin{array}{cc}ae + cg & af + ch \\ be + dg & bf + dh \end{array}\right) $$

and the trace is $ae + bf + cg + dh$. Likewise, if we first vectorize the matrices $$ \widetilde{A} = \left(\begin{array}{c}a & b & c & d \end{array}\right)^T\\ \widetilde{B} = \left(\begin{array}{c}e & f & g & h \end{array}\right)^T\\ $$

it's straightforward to see $\widetilde{A}\cdot\widetilde{B} = \textrm{tr}(A^TB)$.