Confusion about the definition of the Fisher information for discrete random variables

Question

Suppose that $X$ has a absolutely continuous and positive probability density function $f(x)$, I know its Fisher information is defined through $$I(X) := \int_{\mathbb R} \frac{|f'|^2}{f} \mathrm{d}x.$$ However, I am very confused about the definition of $I(X)$ when $X$ is a non-negative integer-valued random variable with probability mass function $(p_0,p_1,\ldots,p_n,\ldots)$. In most references I have found so far, it seems that the definition of the Fisher information contains some unknown parameter $\theta$ (for example, if $X$ is Bernoulli distributed, then the parameter is the success probability in each trial). Since I am not familiar with statistics, I am very confused as to how should we define Fisher information $I(X)$ when $X$ is a non-negative integer-valued random variable with (unknown) probability mass function $(p_0,p_1,\ldots,p_n,\ldots)$. Basically, my motivation is that I have a Fokker-Planck equation for the evolution of the probability mass function of a non-negative integer-valued stochastic process, and I want to know how its Fisher information evolves in time. Thanks for any help or reference!

Answer 1

For a discrete known probability mass function, there is no parameter $\theta$—you know the full distribution. If however you know just the type or form distribution (such as a Gaussian, Bernoulli, etc.), you need to know the parameters (such as the sufficient statistics) in order calculate the Fisher Information (and other measures). After all, the Fisher Information (and the mean, and the variance, and...) of a Gaussian distribution depends upon the mean and the standard deviation, which in your terminology is $\theta$.

In the discrete case, every textbook on information theory will give the discrete version of the definition, in which an integral is replaced by a sum, for instance,

$$I[{\bf P}] = \sum\limits_{i=1}^n \frac{(p_{i} - p_{i-1})^2}{p_i}$$

as shown here.