I was watching a lecture on Gaussian Process and when the covariance matrix was introduced, the tutor explained that the matrix is $(n \times n)$ because every point is covered twice – we include the information about the covariance of $(x_1, x_2)$ and about the covariance of $(x_2, x_1)$. She then said: that's because your covariance can vary in different directions.
How is it possible that the covariances can vary in different directions inside the GP covariance matrix? Could you give me an example of when that could be the case?
Update:
After giving it some thought, I realized that it is not $cov(x_1, x_2)$ or $cov(x_2, x_1)$ (as computed from the definition of covariance) that go into the GP covariance matrix, but instead (as was shown in the lecture as well), the covariance matrix is populated by a covariance kernel $k(x, y)$ that acts/is interpreted as a covariance, but it is some function of the distance between $x$ and $y$.
I could therefore imagine, that we might have a covariance kernel that is a function of $(x – y)^p$ where $p$ is an odd power. In such instance, it would indeed make $k(x_1, x_2) \neq k(x_2, x_1)$. But would this be a valid kernel?
Could you clarify if my thinking about the covariance kernel is reasonable? Could you explain if the covariance matrix in Gaussian Process can be non-symmetric? If yes, could you give an example of a dataset where it would make sense to make covariance different in different directions, i.e. where we would like $k(x_1, x_2)$ to be different from $k(x_2, x_1)$?
Best Answer
Every valid covariance matrix is a real symmetric non-negative definite matrix. This holds regardless of the underlying distribution. So no, it can't be non-symmetric. If the lecturers are making an argument for using some non-symmetric matrix (e.g., using a non-symmetric kernel) in a way that "acts/is interpreted as a covariance" somehow, then the onus is on them to explain how far this analogy holds, given that the matrix is not a valid covariance matrix.