Optimization – Maximum a Posteriori Estimation (MAP) for Gaussian and Cauchy Models

estimationmaximum likelihoodoptimizationparameter estimationproximal-operators

How to solve:

$$ \operatorname{Prox}_{\lambda \phi \left( \cdot \right)} \left( x \right) = \arg \min_{y} \frac{1}{2} {\left( x – y \right)}^{2} + \lambda \phi \left( y \right) $$

Where $ \phi \left( y \right) $ is the log likelihood of the Cauchy Distribution:

$$ \phi \left( y \right) = \frac{\gamma}{ \pi \left( {\gamma}^{2} + {y}^{2} \right) } $$

The model above is the Maximum a Posteriori Estimation (MAP) for the case:

$$ x = y + n, \; n \sim N \left( 0, \lambda \right), \; y \sim \text{Cauchy} \left( {y}_{0} = 0, \gamma \right) $$

Best Answer

The problem is given by (Adding factor for simplicity):

$$\begin{aligned} \arg \min_{y} \frac{1}{2} {\left( x - y \right)}^{2} + \frac{\lambda}{2} \phi \left( y \right) = \arg \min_{y} \frac{1}{2} {\left( x - y \right)}^{2} - \frac{\lambda}{2} \log \left( \frac{\gamma}{ \pi \left( {y}^{2} + {\gamma}^{2} \right) } \right) \\ \end{aligned}$$

Looking at the derivative (Stationary point):

$$\begin{aligned} 0 & = \frac{d}{d y} \left( \frac{1}{2} {\left( x - y \right)}^{2} - \frac{\lambda}{2} \log \left( \frac{\gamma}{ \pi \left( {y}^{2} + {\gamma}^{2} \right) } \right) \right) && \text{Definition of stationary point} \\ & = \frac{1}{2} \frac{d}{d y} {\left( y - x \right)}^{2} - \frac{\lambda}{2} \frac{d}{d y} \left( \log \left( \frac{\gamma}{ \pi \left( {y}^{2} + {\gamma}^{2} \right) } \right) \right) && \text{} \\ & = y - x - \frac{\lambda}{2} \frac{d}{d y} \left( \log \left( \frac{\gamma}{ \pi \left( {y}^{2} + {\gamma}^{2} \right) } \right) \right) && \text{} \\ & = y - x + \frac{\lambda}{2} \left( \frac{d}{d y} \log \left( {y}^{2} + {\gamma}^{2} \right) \right) && \text{} \\ & = y - x + \frac{\lambda}{2} \frac{1}{{y}^{2} + {\gamma}^{2}} \frac{d}{d y} \left( {y}^{2} + {\gamma}^{2} \right) && \text{} \\ & = y - x + \lambda \frac{y}{{y}^{2} + {\gamma}^{2}} && \text{} \\ & = {y}^{3} - x {y}^{2} + \left( \lambda + {\gamma}^{2} \right) y - x {\gamma}^{2} \end{aligned}$$

The above is a cubic polynomial which has a closed form solution.

Related Question