Interpret that a distribution does not have defined entropy (or has infinite entropy)

entropyinformation theorystatistics

An entropy (is Shannon sense) can be interpreted as uncertainty or missing knowledge. When the knowledge is added, the entropy decreases. Hence it can also be interpreted as information content.

However, there are discrete distributions which do not have an entropy because $-\sum_{i=1}^{\infty}p_i\log p_i$ tends to infinity (see here).

How to interpret this situation? Does non-existent or infinite entropy means that it is not possible to remove uncertainty and get complete knowledge about values of variable described by such distribution?

Best Answer

A discrete distribution (with infinite outcomes) can have an infinite entropy. I don't see much to interpret here. Actually, one might argue that, on the contrary, the surprising thing is that some (most) discrete distributions with infinite outcomes have finite entropy, i.e. they produce a finite amount of information, so they can be described (in average) with a finite number of bits.

I think that the most natural example of infinite entropy is a continuous distribution (say, uniform on $[0,1]$), which can be regarded as the limit of a discrete uniform distribution for increasingly large number of values. It's not surprising that a real number in $[0,1]$ has infinite entropy, because to describe it you need an infinite number of bits (think of its binary fractional representation). Put in other way, in a real number in $[0,1]$ you can code an arbitrarily big amount of information.

Related Question