I am reading a textbook Gaussian Process for Machine Learning by C.E. Rasmussen and C.K.I. Williams and I am having some trouble understanding what does distribution over functions mean. In the textbook, an example is given, that one should imagine a function as a very long vector (in fact, it should be infinitely long?). So I imagine a distribution over functions to be a probability distribution drawn "above" such vector values. Would it then be a probability that a function will take this particular value? Or would it be a probability that a function will take a value that is in a given range? Or is distribution over functions a probability assigned to a whole function?
Quotes from the textbook:
Chapter 1: Introduction, page 2
A Gaussian process is a generalization of the Gaussian probability
distribution. Whereas a probability distribution describes random
variables which are scalars or vectors (for multivariate
distributions), a stochastic process governs the properties of
functions. Leaving mathematical sophistication aside, one can loosely
think of a function as a very long vector, each entry in the vector
specifying the function value f(x) at a particular input x. It turns
out, that although this idea is a little naive, it is surprisingly
close what we need. Indeed, the question of how we deal
computationally with these infinite dimensional objects has the most
pleasant resolution imaginable: if you ask only for the properties of
the function at a finite number of points, then inference in the
Gaussian process will give you the same answer if you ignore the
infinitely many other points, as if you would have taken them all into
account!
Chapter 2: Regression, page 7
There are several ways to interpret Gaussian process (GP) regression
models. One can think of a Gaussian process as defining a distribution
over functions, and inference taking place directly in the space of
functions, the function-space view.
From the initial question:
I made this conceptual picture to try to visualize this for myself. I am not sure if such explanation that I made for myself is correct.
After the update:
After the answer of Gijs I updated the picture to be conceptually more something like this:
Best Answer
The concept is a bit more abstract than a usual distribution. The problem is that we are used to the concept of a distribution over $\mathbb{R}$, typically shown as a line, and then expand it to a surface $\mathbb{R}^2$, and so on to distributions over $\mathbb{R}^n$. But the space of functions cannot be represented as a square or a line or a vector. It's not a crime to think of it that way, like you do, but theory that works in $\mathbb{R}^n$, having to do with distance, neighborhoods and such (this is known as the topology of the space), are not the same in the space of functions. So drawing it as a square can give you wrong intuitions about that space.
You can simply think of the space of functions as a big collection of functions, perhaps a bag of things if you will. The distribution here then gives you the probabilities of drawing a subset of those things. The distribution will say: the probability that your next draw (of a function) is in this subset, is, for example, 10%. In the case of a Gaussian process on functions in two dimensions, you might ask, given an
x-coordinate and an interval ofy-values, this is a small vertical line segment, what is the probability that a (random) function will pass through this small line? That's going to be a positive probability. So the Gaussian process specifies a distribution (of probability) over a space of functions. In this example, the subset of the space of functions is the subset that passes through the line segment.Another confusing naming conventention here is that a distribution is commonly specified by a density function, such as the bell shape with the normal distribution. There, the area under the distribution function tells you how probable an interval is. This doesn't work for all distributions however, and in particular, in the case of functions (not $\mathbb{R}$ as with the normal distributions), this doesn't work at all. That means you won't be able to write this distribution (as specified by the Gaussian process) as a density function.