In maximum likelihood estimation for the parameter, mean $\mu$ of multivariate normal distribution, I end up with the following result:
$\mu_{MLE} = \dfrac{1}{n}\sum_{i=1}^n x_i$
The way I understand the above equation is the following way below, but I am wondering if the above equation means the following equation:
$\mu_{MLE} = \begin{bmatrix}\mu_1\\\mu_2\\\vdots\\\mu_n\end{bmatrix} = \begin{bmatrix}\dfrac{1}{n}\sum_{i=1}^n x_1\\\dfrac{1}{n}\sum_{i=1}^n x_2\\\vdots\\\dfrac{1}{n}\sum_{i=1}^n x_i\end{bmatrix} = \begin{bmatrix}E(X_1)\\E(X_2)\\\vdots\\E(X_n)\end{bmatrix} = \begin{bmatrix}\bar{X_1} \\ \bar{X_2} \\ \vdots \\ \bar{X_n}\end{bmatrix}$
And also Maximum Likelhood Estimation for $\Sigma$, I have ended up with the following equation:
$\Sigma_{MLE} = \dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T = \begin{bmatrix}Var(\sigma_{11}) & Cov(\sigma_{12}) & \dots & Cov(\sigma_{1n}) \\ Cov(\sigma_{21}) & Var(\sigma_{22}) & \dots & Cov(\sigma_{2n}) \\ {}& \vdots& \\ Cov(\sigma_{n1}) & Cov(\sigma_{n2}) & \dots & Var(\sigma_{nn})
\end{bmatrix}$
Somehow, I got the equation following some algebraic steps, but I can't see how this part, $\dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T$, works and produces the covariance matrix. Hope to hear some explanations.
Best Answer
It's just linear algebra. Let's take a concrete example where $n=3$. Then ${\bf \mu} \in \mathbb{R}^{3 \times 1}$.
$$\Sigma_{MLE} = \dfrac{1}{3}\sum_{i=1}^3(x_i-{\bf \mu})(x_i-{\bf \mu})^T = \frac13 \left( \begin{array}{c} x_{1} - \mu \\ x_{2} - \mu \\ x_{3} - \mu\end{array} \right)\left( \begin{array}{ccc} x_{1} - \mu & x_{2} - \mu & x_{3} - \mu\end{array} \right) $$ $$ = \frac13 \left( \begin{array}{ccc} (x_{1} - \mu)^2 & (x_{1} - \mu)(x_{2} - \mu) & (x_{1} - \mu)(x_{3} - \mu) \\ (x_{2} - \mu)(x_{1} - \mu) & (x_{2} - \mu)^2 & (x_{2} - \mu)(x_{3} - \mu) \\ (x_{3} - \mu)(x_{1} - \mu) & (x_{3} - \mu)(x_{2} - \mu) & (x_{3} - \mu)^2 \end{array} \right) $$ $$ = \left( \begin{array}{ccc} Var(x_1) & Cov(x_1,x_2) & Cov(x_1,x_3) \\ Cov(x_2,x_1) & Var(x_2) & Cov(x_2,x_3)\\ Cov(x_3,x_1) & Cov(x_3,x_2) & Var(x_3) \end{array} \right) $$
Also note that $Var(\sigma_{11})$ is odd notation. You might be thinking $\sigma^2_{x_1}$ or $Var(x_1)$.