Yes, in scalar field theory, $\langle 0 | T\{\phi(y) \phi(x)\} | 0 \rangle$ is the amplitude for a particle to propagate from $x$ to $y$. There are caveats to this, because not all QFTs admit particle interpretations, but for massive scalar fields with at most moderately strong interactions, it's correct. Applying the operator $\phi({\bf x},t)$ to the vacuum $|0\rangle$ puts the QFT into the state $|\delta_{\bf x},t \rangle$, where there's a single particle whose wave function at time $t$ is the delta-function supported at ${\bf x}$. If $x$ comes later than $y$, the number $\langle 0 | \phi({\bf x},t)\phi({\bf y},t') | 0 \rangle$ is just the inner product of $| \delta_{\bf x},t \rangle$ with $| \delta_{\bf y},t' \rangle$.
However, the function $f(x,y) = \langle 0 | T\{\phi(y) \phi(x)\} | 0 \rangle$ is not actually a correlation function in the standard statistical sense. It can't be; it's not even real-valued. However, it is a close cousin of an honest-to-goodness correlation function.
If make the substitution $t=-i\tau$, you'll turn the action
$$iS = i\int dtd{\bf x} \{\phi(x)\Box\phi(x) - V(\phi(x))\}$$
of scalar field theory on $\mathbb{R}^{d,1}$ into an energy function
$$-E(\phi) = -\int d\tau d{\bf x} \{\phi(x)\Delta\phi(x) + V(\phi(x))\}$$
which is defined on scalar fields living on $\mathbb{R}^{d+1}$. Likewise, the oscillating Feynman integral $\int \mathcal{D}\phi e^{iS(\phi)}$ becomes a Gibbs measure $\int \mathcal{D}\phi e^{-E(\phi)}$.
The Gibbs measure is a probability measure on the set of classical scalar fields on $\mathbb{R}^{d+1}$. It has correlation functions $g(({\bf x}, \tau),({\bf y},\tau')) = E[\phi({\bf x}, \tau)\phi({\bf y},\tau')]$. These correlation functions have the property that they may be analytically continued to complex values of $\tau$ having the form $\tau = e^{i\theta}t$ with $\theta \in [0,\pi/2]$. If we take $\tau$ as far as we can, setting it equal to $i t$, we obtain the Minkowski-signature "correlation functions" $f(x,y) = g(({\bf x},it),({\bf y},it'))$.
So $f$ isn't really a correlation function, but it's the boundary value of the analytic continuation of a correlation function. But that takes a long time to say, so the terminology gets abused.
Saying that $\delta(0) = 0$ is completely non-sensical since the Dirac delta function is not a function to begin with. When we physicists write
$$ \int \delta(x)f(x) \mathrm{d}x = f(0) \tag{1}$$
when that's all the "definition" of the delta "function" you actually need. Formally, the $\delta$ function is a tempered distribution, something that assigns numbers to test functions. The "integral notation" eq. (1) is just a mnemonic because in many respects this assignment "behaves like an integral", e.g. it obeys a variant of integration by parts. The formal definition of the delta "function" is just $f\mapsto \delta[f] = f(0)$ where you are already notationally prohibited from trying to feed a position like $x=0$ as $\delta(0)$ to it.
While it is true that one can represent the $\delta$ function as the limit of certain other functions (these are called nascent delta functions), this limit is not taken in the space of functions, but in the space of distributions, so the result is not a function. I do not have access to the specific article you are reading, but in general, manipulating the value of the delta function at specific points does not make any rigorous sense. As so often in physics, this does not necessarily mean the result obtained is wrong, but it should be proven by other means in order to trust it.
Best Answer
The correlation function you wrote is a completely general correlation of two quantities, $$\langle f(X) g(Y)\rangle$$ You just use the symbol $x'$ for $Y$ and the symbol $x+x'$ for $X$.
If the environment - the vacuum or the material - is translationally invariant, it means that its properties don't depend on overall translations. So if you change $X$ and $Y$ by the same amount, e.g. by $z$, the correlation function will not change.
Consequently, you may shift by $z=-Y=-x'$ which means that the new $Y$ will be zero. So $$\langle f(X) g(Y)\rangle = \langle f(X-Y)g(0)\rangle = \langle f(x)g(0) \rangle$$ As you can see, for translationally symmetric systems, the correlation function only depends on the difference of the coordinates i.e. separation of the arguments of $f$ and $g$, which is equal to $x$ in your case.
So this should have explained the dependence on $x$ and $x'$.
Now, what is a correlator? Classically, it is some average over the probabilistic distribution $$\langle S \rangle = \int D\phi\,\rho(\phi) S(\phi)$$ This holds for $S$ being the product of several quantities, too. The integral goes over all possible configurations of the physical system and $\rho(\phi)$ is the probability density of the particular configuration $\phi$.
In quantum mechanics, the correlation function is the expectation value in the actual state of the system - usually the ground state and/or a thermal state. For a ground state which is pure, we have $$\langle \hat{S} \rangle = \langle 0 | \hat{S} | 0 \rangle$$ where the 0-ket-vector is the ground state, while for a thermal state expressed by a density matrix $\rho$, the correlation function is defined as $$\langle \hat{S} \rangle = \mbox{Tr}\, (\hat{S}\hat{\rho})$$ Well, correlation functions are functions that know about the correlation of the physical quantities $f$ and $g$ at two points. If the correlation is zero, it looks like the two quantities are independent of each other. If the correlation is positive, it looks like the two quantities are likely to have the same sign; the more positive it is, the more they're correlated. They're correlated with the opposite signs if the correlation function is negative.
In quantum field theory, correlation functions of two operators - just like you wrote - is known as the propagator and it is the mathematical expression that replaces all internal lines of Feynman diagrams. It tells you what is the probability amplitude that the corresponding particle propagates from the point $x+x'$ to the point $x'$. It is usually nonzero inside the light cone only and depends on the difference of the coordinates only. An exception to this is the Feynman Propagator in QED. It is nonzero outside the light cone as well, but invokes anti-particles, which cancel this nonzero contribution outside the light cone, and preserve causality.
Correlation functions involving an arbitrary positive number of operators are known as the Green's functions or $n$-point functions if a product of $n$ quantities is in between the brackets. In some sense, the $n$-point functions know everything about the calculable dynamical quantities describing the physical system. The fact that everything can be expanded into correlation functions is a generalization of the Taylor expansions to the case of infinitely many variables.
In particular, the scattering amplitude for $n$ external particles (the total number, including incoming and outgoing ones) may be calculated from the $n$-point functions. The Feynman diagrams mentioned previously are a method to do this calculation systematically: a complicated correlator may be rewritten into a function of the 2-point functions, the propagators, contracted with the interaction vertices.
There are many words to physically describe a correlation function in various contexts - such as the response functions etc. The idea is that you insert an impurity or a signal into $x'$, that's your $g(x')$, and you study how much the field $f(x+x')$ at point $x+x'$ is affected by the impurity $g(x')$.