I need to invert a Hessian matrix to calculate the covariance matrix. The matrices are fairly large, typical sizes are (300×300), or values of that order. In general, the Hessian is very ill-conditioned. The covariance matrix (in this case, the inverse of the Hessian) will have a blocky structure (blocks of elements around the main diagonal). I have tried to do an SVD of the Hessian, and invert it, but I am at a loss of where to cut the singular values. If I cut very early, I have what looks like a smoothed version of the covariances. If I cut too lates, it's just noise.
All my "clever" 🙂 processing is to truncate the svd at some level, reconstruct the truncated Hessian and invert it. I don't know whether there are any more robust ways of going about this, and that do not involve trial and error in terms of truncating the matrix.
Thanks
Best Answer
You're really asking the wrong question here...
Let's back up a bit. You're attempting to estimate some parameters here, either by a maximum likelihood method or more likely by $\chi^2$ minimization, right? Normally, after you've completed the $\chi^2$ minimization, you take the Hessian of $\chi^2$ at the optimal parameters, and then use its inverse as a covariance matrix for the fitted parameters, right? However, you've found that your Hessian matrix is essentially singular, right?
So what does this mean?
First, the fact that the Hessian is essentially singular tells you that the $\chi^2$ minimization problem is ill-conditioned in the sense that many different sets of parameters fit the data equally well. This presents problems for whatever algorithm you used to perform the $\chi^2$ minimization, and it's likely in practice that you haven't got a very good minimum.
More importantly, the fact that the $\chi^2$ minimization problem is ill-conditioned tells you that the experiment you conducted really doesn't pin down the values of the parameters that you're trying to estimate.
It's important to understand that the problem you have here is not one of numerical computing, but that it is in fact a significant statistical issue. What you really need to do here is redesign your experiment so that it will provide you with useful information about the parameters that you're trying to estimate.
Trying to kludge together some sort of approximate covariance matrix by using the truncated SVD is simply the wrong way to approach this problem. What will go wrong? By truncating the SVD, you'll end up with confidence intervals for the fitted parameters that are overly tight.