Solved – Hessian matrix and initial guess in logistic regression

logisticregression

The log-likelihood function for logistic function is $$l(\theta) = \sum_{i=1}^m(y^{(i)}\log h(x^{(i)}) + (1-y^{(i)})\log(1 – h(x^{(i)})))$$, where $$h(x^{(i)}) = \frac{1}{1 + e^{-\theta^Tx^{(i)}}}\,.$$

In order to obtain maximum likelihood estimation, I implemented fitting the logistic regression model using Newton's method. I encountered 2 problems:

  1. I try to fit the model to my data, but during the iterations, a singular Hessian matrix is encountered, what do I do with this kind of problem?

  2. With different initial guess $\theta$, will the model converge to different results?

Best Answer

One trick that often helps for logistic regression type problems is to realize that:

$1 - h(x^{(i)}) = h(-x^{(i)})$

and that $h(-x^{(i)})$ is more numerically stable than $1 - h(x^{(i)})$.

You can find a discussion of that here. This is an article in the Stata Journal so the examples are in Stata/Mata, but the problem has to do with the way computers store numbers and is thus more general. For example, I have been able to reproduce the first anomalous example exactly in R, i.e. not just the general pattern but the exact values.

Related Question