Likelihood – Difference Between Partial, Profile, and Marginal Likelihood

Question

I see these terms being used and I keep getting them mixed up. Is there a simple explanation of the differences between them?

Answer 1

The likelihood function usually depends on many parameters. Depending on the application, we are usually interested in only a subset of these parameters. For example, in linear regression, interest typically lies in the slope coefficients and not on the error variance.

Denote the parameters we are interested in as $\beta$ and the parameters that are not of primary interest as $\theta$. The standard way to approach the estimation problem is to maximize the likelihood function so that we obtain estimates of $\beta$ and $\theta$. However, since the primary interest lies in $\beta$ partial, profile and marginal likelihood offer alternative ways to estimate $\beta$ without estimating $\theta$.

In order to see the difference denote the standard likelihood by $L(\beta, \theta|\mathrm{data})$.

Maximum Likelihood

Find $\beta$ and $\theta$ that maximizes $L(\beta, \theta|\mathrm{data})$.

Partial Likelihood

If we can write the likelihood function as:

$$L(\beta, \theta|\mathrm{data}) = L_1(\beta|\mathrm{data}) L_2(\theta|\mathrm{data})$$

Then we simply maximize $L_1(\beta|\mathrm{data})$.

Profile Likelihood

If we can express $\theta$ as a function of $\beta$ then we replace $\theta$ with the corresponding function.

Say, $\theta = g(\beta)$. Then, we maximize:

$$L(\beta, g(\beta)|\mathrm{data})$$

Marginal Likelihood

We integrate out $\theta$ from the likelihood equation by exploiting the fact that we can identify the probability distribution of $\theta$ conditional on $\beta$.