SAS Nonlinear Regression – What Does ‘Hessian Is Singular’ Mean in PROC NLIN?

nonlinear regressionregressionsas

Im trying to fit a power addictive nonlinear model, and keep getting a message in the results that convergence criterion met but hessian is singular. What does this mean? also in the correlation matrix I have no values for 1 or more coefficients. My guess is that it means I converged to a local minimum, but I'm not sure.

Best Answer

The Hessian is the matrix of second derivatives of the objective function you are attempting to minimize (resp. maximize, depending on how SAS set this up). The Hessian is a square $k \times k$ matrix, where $k$ is the number of parameters in your model. In your case, the Hessian is singular, which means that your parameters are linear functions of each other (or almost collinear) ... either absolutely, or with respect to the data that you have.

A young colleague of mine once got the exact error you did in SAS. Turns out he had included an intercept term twice. So the first thing you want to do is to check the specification of the model in SAS.

Another source of difficulty will occur if one of the parameters is basically useless at the data points you have. With a large sample and a good model, the log likelihood should look like a paraboloid near the minimum. However, a useless parameter can produce a long, narrow valley. Any solution in the trough of the valley will produce much the same minimum in the objective function. Your algorithm will terminate normally (it will no longer be able to improve itself by taking additional steps), but the Hessian will be singular.

If you don't have loads of parameters, (and you shouldn't with a non-linear model) you could try plotting the likelihood in the vicinity of the "solution" that SAS came up with, to see where the problem lies.

The Hessian is used to estimate the standard errors of the maximum likelihood estimates, so if the Hessian is broken, you won't get complete entries in the correlation matrix. These are obtained from the inverse of the Hessian, which you can't have, since the Hessian is singular.

I'm not sure that you converged to a minimum. With a singular Hessian, you could have converged to a saddlepoint in the parameter space, or the optimizer may simply have become disorientated and quit in the naïve belief that it had accomplished its mission. Something about your model is not working: either the way you specified it to SAS, or the data you have, or the model you are attempting to fit. Trust nothing in the results you got and start over. Perhaps try a simpler, linear model and then add more complicated features to improve the fit.

Related Solutions

Solved – Maximum likelihood estimation of dlmModReg

I think your setup is not correct. Try this:

set.seed(1234)
r <- rnorm(100)
X <- r
u <- -1*X + 0.5*rnorm(100)
MyModel <- function(x)  dlmModReg(X, FALSE, dV = x[1]^2)
fit <- dlmMLE(u, parm = c(0.3), build = MyModel)
mod <- MyModel(fit$par)
dlmFilter(u,mod)$a

You recover the estimate of the observation variance from the only element of fit$par:

> fit
$par
[1] 0.4431803

$value
[1] -20.69313

$counts
function gradient 
      17       17 

$convergence
[1] 0

$message
[1] "CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH"

while your estimate of the coefficient (should be around -1 in your case) can be obtained as the last element of dlmFilter(u,mod)$a, which gives the values of the state as new observations are processed:

    > dlmFilter(u,mod)$m
  [1]  0.0000000 -1.1486921 -1.2123431 -1.1172783 -1.1231454 -1.1170222
  [7] -1.0974931 -1.1377114 -1.0378758 -1.0927136 -1.0955372 -1.0120210
 [13] -0.9874791 -1.0036429 -1.0765513 -1.0678725 -1.0795124 -1.1568597
 [19] -1.2044821 -1.2056687 -1.2102896 -1.2938958 -1.2922945 -1.2670604
 [25] -1.1789594 -1.1570172 -1.1601590 -1.1417200 -1.1585501 -1.1608675
 [31] -1.1616278 -1.1744861 -1.1717561 -1.1715025 -1.1568086 -1.1451311
 [37] -1.1520867 -1.1379211 -1.1270897 -1.1048035 -1.1015793 -1.1054597
 [43] -1.0621750 -1.0621218 -1.0696813 -1.0807651 -1.0816893 -1.0647963
 [49] -1.0643440 -1.0667282 -1.0626404 -1.0623697 -1.0586265 -1.0571205
 [55] -1.0569135 -1.0579224 -1.0607623 -1.0582257 -1.0495232 -1.0494288
 [61] -1.0539632 -1.0555427 -1.0553468 -1.0491239 -1.0488604 -1.0491036
 [67] -1.0510551 -1.0576294 -1.0611296 -1.0628612 -1.0626451 -1.0573650
 [73] -1.0629577 -1.0647724 -1.0658052 -1.0823839 -1.0753808 -1.0747229
 [79] -1.0747762 -1.0615243 -1.0630352 -1.0697431 -1.0666448 -1.0617227
 [85] -1.0585460 -1.0583981 -1.0563544 -1.0567715 -1.0544349 -1.0573228
 [91] -1.0588404 -1.0639155 -1.0625845 -1.0578004 -1.0571034 -1.0602645
 [97] -1.0604838 -1.0586019 -1.0580891 -1.0587096 -1.0577559

Hope this helps.

Solved – Hessian matrix and initial guess in logistic regression

One trick that often helps for logistic regression type problems is to realize that:

$1 - h(x^{(i)}) = h(-x^{(i)})$

and that $h(-x^{(i)})$ is more numerically stable than $1 - h(x^{(i)})$.

You can find a discussion of that here. This is an article in the Stata Journal so the examples are in Stata/Mata, but the problem has to do with the way computers store numbers and is thus more general. For example, I have been able to reproduce the first anomalous example exactly in R, i.e. not just the general pattern but the exact values.

Best Answer

Related Solutions

Solved – Maximum likelihood estimation of dlmModReg

Solved – Hessian matrix and initial guess in logistic regression

Related Question