Solved – the rationale behind the exponential family of distributions

exponential-family

From elementary probability course, the probability distributions such as Gaussian, Poisson or exponential all have a good motivation. After staring at the formula of the exponential family distributions for a long time, I still do not get any intuition.

$$f_{X}(x\mid {\boldsymbol {\theta }})=h(x)\exp {\Big (}{\boldsymbol {\eta }}({\boldsymbol {\theta }})\cdot \mathbf {T} (x)-A({\boldsymbol {\theta }}){\Big )}$$

Can anyone help me understand Why we need it in the first place?
What are some advantages of modeling a response variable to be exponential family vs normal?

EDIT:
By the exponential family, I meant the general class of distributions described here.

Best Answer

What are some advantages of modeling a response variable to be exponential family vs normal?

  1. The exponential family is much broader than the normal. For example, what's the advantage of using a Poisson or a binomial instead of a normal? A normal's not much use if you have counts with a low mean. What about if your data are continuous but very right skew -- perhaps times or monetary amounts? The exponential family includes the normal, the binomial, the Poisson and the Gamma as special cases (among many others)

  2. It incorporates a wide variety of variance-mean relationships.

  3. It derives from trying to answer a question along the lines of "what distributions are functions of a sufficient statistic", and so has models can be estimated via ML using very simple sufficient statistics; this includes the usual models available in programs that fit generalized linear models. Indeed the sufficient statistic ($T(x)$) is explicit in the exponential-family density function.

  4. It makes it easy to decouple the relationship between the response and predictor from the conditional distribution of the response (via link functions). For example you could fit a straight-line relationship to a model which specifies the conditional response has a gamma distribution, or an exponential relationship with a conditionally Gaussian response in a GLM framework.

For Bayesians the exponential family is quite interesting because all members of the exponential family have conjugate priors.

Related Question