Quadratic Form and Chi-Squared Distribution – How They Relate and Applications

chi-squared-distributioneigenvaluesmultivariate normal distributionquadratic form

It's about the demostration of the quadratic forms and chi-squared distribution.

Let's split the problem:

  1. We have a $n$ vector with n standardized normal distribution called $z={[z_1,z_2…z_n]}$.
    Obviously $z'z$ is a $\chi^2_n$
  2. If we have a symmetric, idempotent matrix (i.e. called $A$) then $z'Az
    \thicksim \chi^2_p $ where $p$ is the rank of $A$. I got this splitting the $A$ matrix with the spectral theorem into: $Q'\Lambda Q$ hence:
    $z'Az = z'Q'\Lambda Qz$ we call $w= Qz$ hence $z'Az =w'\Lambda w$. Because it's a idempotent matrix we have only 1s or 0s as eigenvalues and the number of 1's is exactly the rank of $A$. Then we have $w'w$ with with different length now as the number of the rank of $A$. We backward the process and we get: $z'Q'Qz$ but $Q'Q= I_{rank(A)}$ (they are orthogonal since we consider the eigenvectors) so we have the summation of $p$ $Z^2$.
  3. Now the question comes: if we have a symmetric, positive definite matrix $\Sigma_{nxn}$ my teacher told me that $z'\Sigma z$ is a $\chi^2_n $ distribution. I was trying to find a proof, but I can't find one.
  4. My efforts are: use the spectral theorem again, hence we have: $z'Q' \Lambda Qz$ but now $\Lambda$ is a diagonal matrix with real positive numbers (cause it's positive definite).

    $\Lambda$= \begin{bmatrix}
    \lambda_{1} & 0 & 0 & \dots & 0 \\
    0 & \lambda_{2} & 0 & \dots & 0 \\
    \vdots & \vdots & \vdots & \ddots & \vdots \\
    0 & 0 & 0 & \dots & \lambda_{n}
    \end{bmatrix}

We have called, as before, $w=Qz$ hence we have: $w'\Lambda w$. Let's suppose, that w is 3-dimensional vector with 3 elements called a,b and c.
So $w=$ \begin{bmatrix}
a \\
b \\
c \\
\end{bmatrix}

so $w'\Lambda w= a^2 \lambda_1+b^2 \lambda_2+c^2 \lambda_3$
And here I'm stuck.
Maybe this last point is wrong. Therefore I'm asking for clarification and a clear proof of what my teacher said.

[EDIT]

I found this:

Let $X$ be a $ K \times 1$ standard multivariate normal random vector, i.e., $X \thicksim N(0,I)$. Let $A$ be an orthogonal $K \times K$ real matrix. Define
$Y=AX$
Then also $Y$ has a standard multivariate normal distribution, i.e., $Y \thicksim N(0,I)$.

I'm okay with this, the proof is here: https://www.statlect.com/probability-distributions/normal-distribution-quadratic-forms
So my problem would come something like this: $w'\Lambda w$ where $w=QZ$ and by this theorem $w \thicksim N(0,I)$ since Q is an orthogonal matrix.
So, if w is a 3 dimensional vector I'll have something like this:

$\lambda_1 \chi^2_1 + \lambda_2 \chi^2_1 + \lambda_3 \chi^2_1 $ A lineare combination of chi-squared distribution with weights the value of the eigenvalues. Am I right? If I am, does this sum has a particular distribution?

Best Answer

In general, the quadratic form is a weighted sum of $\chi_1^2$

It is not true in general that $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi^2_p$ for any symmetric positive-definite (variance) matrix $\mathbf{\Sigma}$. Breaking this quadratic form down using the spectral theorem you get:

$$\begin{equation} \begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} = \mathbf{z}^\text{T} \mathbf{Q} \mathbf{\Lambda} \mathbf{Q}^\text{T} \mathbf{z} &= (\mathbf{Q}^\text{T} \mathbf{z})^\text{T} \mathbf{\Lambda} (\mathbf{Q}^\text{T} \mathbf{z}) \\[6pt] &= \sum_{i=1}^p \lambda_i ( \mathbf{q}_i \cdot \mathbf{z} )^2, \\[6pt] \end{aligned} \end{equation}$$

where $\mathbf{q}_1,...,\mathbf{q}_p$ are the eigenvectors of $\mathbf{\Sigma}$ (i.e., the columns of $\mathbf{Q}$). Define the random variables $y_i = \mathbf{q}_i \cdot \mathbf{z}$. Since $\mathbf{\Sigma}$ is a real symmetric matrix, the eigenvectors $\mathbf{q}_1,...,\mathbf{q}_p$ are orthonormal, which means that $y_1,...,y_p \sim \text{IID N}(0,1)$. Thus, we have:

$$\begin{equation} \begin{aligned} \mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} &= \sum_{i=1}^p \lambda_i ( \mathbf{q}_i \cdot \mathbf{z} )^2 \\[6pt] &= \sum_{i=1}^p \lambda_i \cdot y_i^2 \\[6pt] &\sim \sum_{i=1}^p \lambda_i \cdot \chi_1^2. \\[6pt] \end{aligned} \end{equation}$$

We can see that the distribution of the quadratic form is a weighted sum of $\chi_1^2$ random variables, where the weights are the eigenvalues of the variance matrix. In the special case where these eigenvalues are all one we do indeed obtain $\mathbf{z}^\text{T} \mathbf{\Sigma} \mathbf{z} \sim \chi_n^2$, but in general this result does not hold. In fact, we can see that in general, the quadratic form is distributed as a weighted sum of chi-squared random variables each with one degree-of-freedom. The general distribution for this form is complicated and its density function does not have a closed form representation. Bodenham and Adams (2015) examine some approximations to this distribution, and provide comparisons with simulations.