Solved – get NAN for p-values while using statsmodels logit function

generalized linear modellogisticrstatsmodels

My data

I used statsmodels to build a logistic regression as follows:

X = np.copy(train_data)
X = sm_.add_constant(X)

model = sm.Logit(train_y, X)

result = model.fit(method='bfgs', maxiter=10000)

p_values[i-1, j-1, :] = result.pvalues
logistic_Coefficients[i-1, j-1, :] = result.params

But I get the following error and my p-values are all NAN:

C:\Users\maryamr\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\statsmodels\base\model.py:488: HessianInversionWarning: Inverting hessian failed, no bse or cov_params available
  'available', HessianInversionWarning)
C:\Users\maryamr\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in greater
  return (self.a < x) & (x < self.b)
C:\Users\maryamr\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in less
  return (self.a < x) & (x < self.b)
C:\Users\maryamr\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\scipy\stats\_distn_infrastructure.py:1821: RuntimeWarning: invalid value encountered in less_equal
  cond2 = cond0 & (x <= self.a)

I also tried glm in r and I get the following error but just one of the features has NAN coefficient and p-value.

Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred

Best Answer

R/GLM and statsmodels.GLM have different ways of handling "perfect separation" (which is what is happening when fitted probabilities are 0 or 1). In Statsmodels, a fitted probability of 0 or 1 creates Inf values on the logit scale, which propagates through all the other calculations, generally giving NaN values for everything.

There are many ways of dealing with perfect separation. One option is to manually drop variables until the situation resolves. There are also some automated approaches. Statsmodels has elastic net penalized logistic regression (using fit_regularized instead of fit). But this will give you point estimates without standard errors. The statsmodels master has conditional logistic regression. I don't think Statsmodels has Firth's method.

Edited: reading your question again, it looks like the optimization converged but the Hessian was not invertible. This is a related but less severe version of what I described above. But the same advice still applies.

Related Question