The correlation function you wrote is a completely general correlation of two quantities,
$$\langle f(X) g(Y)\rangle$$
You just use the symbol $x'$ for $Y$ and the symbol $x+x'$ for $X$.
If the environment - the vacuum or the material - is translationally invariant, it means that its properties don't depend on overall translations. So if you change $X$ and $Y$ by the same amount, e.g. by $z$, the correlation function will not change.
Consequently, you may shift by $z=-Y=-x'$ which means that the new $Y$ will be zero. So
$$\langle f(X) g(Y)\rangle = \langle f(X-Y)g(0)\rangle = \langle f(x)g(0) \rangle$$
As you can see, for translationally symmetric systems, the correlation function only depends on the difference of the coordinates i.e. separation of the arguments of $f$ and $g$, which is equal to $x$ in your case.
So this should have explained the dependence on $x$ and $x'$.
Now, what is a correlator? Classically, it is some average over the probabilistic distribution
$$\langle S \rangle = \int D\phi\,\rho(\phi) S(\phi)$$
This holds for $S$ being the product of several quantities, too. The integral goes over all possible configurations of the physical system and $\rho(\phi)$ is the probability density of the particular configuration $\phi$.
In quantum mechanics, the correlation function is the expectation value in the actual state of the system - usually the ground state and/or a thermal state. For a ground state which is pure, we have
$$\langle \hat{S} \rangle = \langle 0 | \hat{S} | 0 \rangle$$
where the 0-ket-vector is the ground state, while for a thermal state expressed by a density matrix $\rho$, the correlation function is defined as
$$\langle \hat{S} \rangle = \mbox{Tr}\, (\hat{S}\hat{\rho})$$
Well, correlation functions are functions that know about the correlation of the physical quantities $f$ and $g$ at two points. If the correlation is zero, it looks like the two quantities are independent of each other. If the correlation is positive, it looks like the two quantities are likely to have the same sign; the more positive it is, the more they're correlated. They're correlated with the opposite signs if the correlation function is negative.
In quantum field theory, correlation functions of two operators - just like you wrote - is known as the propagator and it is the mathematical expression that replaces all internal lines of Feynman diagrams. It tells you what is the probability amplitude that the corresponding particle propagates from the point $x+x'$ to the point $x'$. It is usually nonzero inside the light cone only and depends on the difference of the coordinates only.
An exception to this is the Feynman Propagator in QED. It is nonzero outside the light cone as well, but invokes anti-particles, which cancel this nonzero contribution outside the light cone, and preserve causality.
Correlation functions involving an arbitrary positive number of operators are known as the Green's functions or $n$-point functions if a product of $n$ quantities is in between the brackets. In some sense, the $n$-point functions know everything about the calculable dynamical quantities describing the physical system. The fact that everything can be expanded into correlation functions is a generalization of the Taylor expansions to the case of infinitely many variables.
In particular, the scattering amplitude for $n$ external particles (the total number, including incoming and outgoing ones) may be calculated from the $n$-point functions. The Feynman diagrams mentioned previously are a method to do this calculation systematically: a complicated correlator may be rewritten into a function of the 2-point functions, the propagators, contracted with the interaction vertices.
There are many words to physically describe a correlation function in various contexts - such as the response functions etc. The idea is that you insert an impurity or a signal into $x'$, that's your $g(x')$, and you study how much the field $f(x+x')$ at point $x+x'$ is affected by the impurity $g(x')$.
Density function (normalied) of a $N$-particle classical system with cordinates $\{\vec{r}_{i}^{}\}$ can be defined as :
$$\rho(\vec{r})=\frac{1}{N}\sum_{i=1}^{N}\delta_{}^{(3)}(\vec{r}-\vec{r}_{i}^{}).$$
And the density-density correlation function (in canonical ensemble) is defined as :
$$\langle\rho(\vec{r}) \rho(\vec{r}')\rangle = \frac{\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}\rho(\vec{r}) \rho(\vec{r}')e^{-\beta\mathcal{H}(\{\vec{r}_{i}^{}\},\{\vec{p}_{i}^{}\})}}{\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{-\beta\mathcal{H}(\{\vec{r}_{i}^{}\},\{\vec{p}_{i}^{}\})}}.$$
$\textbf{Addendum (Field theoretic generating function technology):}$
To recast the problem in field theoretic terms (to obtain a statistical field theory) for simplicity consider :
$$\mathcal{H}(\{\vec{r}_{i}^{}\},\{\vec{p}_{i}^{}\})=\sum_{i=1}^{N}\frac{\vec{p}_{i}^{}\cdot\vec{p}_{i}^{}}{2m}+\frac{1}{2}\sum_{i\neq j=1}^{N}\mathcal{V}(\vec{r}_{i}^{},\vec{r}_{j}^{}).$$
Now define a density moment generating function as :
$$\mathcal{Z}[\{\phi(\vec{r})\}] = \frac{\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{i\int d^{3}_{}\vec{r}\phi(\vec{r})\rho(\vec{r})}_{}e^{-\beta\mathcal{H}(\{\vec{r}_{i}^{}\},\{\vec{p}_{i}^{}\})}}{\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{-\beta\mathcal{H}(\{\vec{r}_{i}^{}\},\{\vec{p}_{i}^{}\})}}.$$
Using which we can calculated density-density correlation functions to all order as :
$$\langle\rho(\vec{r}_{1}^{})\cdots\
\rho(\vec{r}_{n}^{})\rangle=(-i)^{n}_{}\frac{\delta}{\delta \phi(\vec{r}_{1}^{})}\cdots\frac{\delta}{\delta \phi(\vec{r}_{n}^{})}\mathcal{Z}[\{\phi(\vec{r})\}]\Big|_{\{\phi(\vec{r})=0\}}^{}.$$
Recast the moment generating function as (self interaction of particles need to be regularized somehow (denoted as $\Delta$ here)) :
$$\mathcal{Z}[\{\phi(\vec{r})\}] = \frac{\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{i\int d^{3}_{}\vec{r}\phi(\vec{r})\rho(\vec{r})}_{}e^{-\beta\big[\sum_{i=1}^{N}\frac{\vec{p}_{i}^{}\cdot\vec{p}_{i}^{}}{2m}+\frac{N^{2}_{}}{2}\int\int d^{3}_{}\vec{r} d^{3}_{}\vec{r}'\rho(\vec{r})\mathcal{V}(\vec{r},\vec{r}')\rho(\vec{r}')+\Delta[\rho(\vec{r})]\big]}}{\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{-\beta\mathcal{H}(\{\vec{r}_{i}^{}\},\{\vec{p}_{i}^{}\})}}.$$
Now we can introduce the functional delta identity :
$$\int\mathcal{D}[\sigma(\vec{r})]\delta[\sigma(\vec{r})-\rho(\vec{r})]=1$$
inside the generating function to get :
$$\mathcal{Z}[\{\phi(\vec{r})\}] = \frac{\int\mathcal{D}[\sigma(\vec{r})]e^{i\int d^{3}_{}\vec{r}\phi(\vec{r})\sigma(\vec{r})}_{}e^{-\beta\big[\frac{N^{2}_{}}{2}\int\int d^{3}_{}\vec{r} d^{3}_{}\vec{r}'\sigma(\vec{r})\mathcal{V}(\vec{r},\vec{r}')\sigma(\vec{r}')+\Delta[\sigma(\vec{r})]\big]}e^{-\beta\Phi[\sigma(\vec{r})]}}{\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{-\beta\mathcal{H}(\{\vec{r}_{i}^{}\},\{\vec{p}_{i}^{}\})}}.$$
Where :
$$e^{-\beta\Phi[\sigma(\vec{r})]}=\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}\delta[\sigma(\vec{r})-\rho(\vec{r})]e^{-\beta\big[\sum_{i=1}^{N}\frac{\vec{p}_{i}^{}\cdot\vec{p}_{i}^{}}{2m}\big]}$$
which can be rewritten using functional Fourier representation of functional delta functional as :
$$e^{-\beta\Phi[\sigma(\vec{r})]}=\int\mathcal{D}[\omega(\vec{r})]\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{-i\int d_{}^{3}\vec{r}\omega(\vec{r})[\sigma(\vec{r})-\rho(\vec{r})]}e^{-\beta\big[\sum_{i=1}^{N}\frac{\vec{p}_{i}^{}\cdot\vec{p}_{i}^{}}{2m}\big]}.$$
Finally everything can be recast as :
$$\mathcal{Z}[\{\phi(\vec{r})\}] = \frac{\int\mathcal{D}[\omega(\vec{r})]\int\mathcal{D}[\sigma(\vec{r})]e^{i\int d^{3}_{}\vec{r}[\phi(\vec{r})-\omega(\vec{r})]\sigma(\vec{r})}_{}e^{-\beta\big[\frac{N^{2}_{}}{2}\int\int d^{3}_{}\vec{r} d^{3}_{}\vec{r}'\sigma(\vec{r})\mathcal{V}(\vec{r},\vec{r}')\sigma(\vec{r}')+\Delta[\sigma(\vec{r})]\big]}e^{-\beta F[\omega(\vec{r})]}}{\mathcal{Z}[\{0\}]}.$$
With :
$$e^{-\beta F[\omega(\vec{r})]}=\int\dots\int\prod_{i=1}^{N}d_{}^{3}\vec{r}_{i}^{}d_{}^{3}\vec{p}_{i}^{}e^{-\beta\sum_{i=1}^{N}\big[\frac{\vec{p}_{i}^{}\cdot\vec{p}_{i}^{}}{2m}+i\beta_{}^{-1}\omega(\vec{r}_{i}^{})\big]},$$
is the partition function of non-interacting particles in a external imaginary potential.
$\textbf{Probability functional for densities:}$ Finally, on inverse functional Fourier transforming $\mathcal{Z}[\{\phi(\vec{r})\}]$ we get probability functional for densities as :
$$P[\{\sigma(\vec{r})\}]=\frac{e^{-\beta\big[\frac{N^{2}_{}}{2}\int\int d^{3}_{}\vec{r} d^{3}_{}\vec{r}'\sigma(\vec{r})\mathcal{V}(\vec{r},\vec{r}')\sigma(\vec{r}')+\Delta[\sigma(\vec{r})]+\mathbb{V}_{}^{}[\sigma(\vec{r})]\big]}}{\int\mathcal{D}[\sigma(\vec{r})]e^{-\beta\big[\frac{N^{2}_{}}{2}\int\int d^{3}_{}\vec{r} d^{3}_{}\vec{r}'\sigma(\vec{r})\mathcal{V}(\vec{r},\vec{r}')\sigma(\vec{r}')+\Delta[\sigma(\vec{r})]+\mathbb{V}_{}^{}[\sigma(\vec{r})]\big]}}$$
with
$$e^{-\beta\mathbb{V}_{}^{}[\sigma(\vec{r})]}=\int\mathcal{D}[\omega(\vec{r})]e^{-i\int d^{3}_{}\vec{r}\omega(\vec{r})\sigma(\vec{r})}_{}e^{-\beta F[\omega(\vec{r})]}.$$
Best Answer
Saying that $\delta(0) = 0$ is completely non-sensical since the Dirac delta function is not a function to begin with. When we physicists write $$ \int \delta(x)f(x) \mathrm{d}x = f(0) \tag{1}$$ when that's all the "definition" of the delta "function" you actually need. Formally, the $\delta$ function is a tempered distribution, something that assigns numbers to test functions. The "integral notation" eq. (1) is just a mnemonic because in many respects this assignment "behaves like an integral", e.g. it obeys a variant of integration by parts. The formal definition of the delta "function" is just $f\mapsto \delta[f] = f(0)$ where you are already notationally prohibited from trying to feed a position like $x=0$ as $\delta(0)$ to it.
While it is true that one can represent the $\delta$ function as the limit of certain other functions (these are called nascent delta functions), this limit is not taken in the space of functions, but in the space of distributions, so the result is not a function. I do not have access to the specific article you are reading, but in general, manipulating the value of the delta function at specific points does not make any rigorous sense. As so often in physics, this does not necessarily mean the result obtained is wrong, but it should be proven by other means in order to trust it.