[Math] Eigenvalue decomposition of huge matrices

eigenvalues-eigenvectorspython

I have a huge matrix (1000,000 x 1000,000) and I am trying to find the first twenty something eigenvectors of this problem. Is there a way to maybe break this huge matrix into many smaller ones to speed up the process.

I have tried the Lanczos method in python (scipy), but it is still computationally very expensive. At the moment I am not even able to load the huge matrix into memory, let alone compute anything.

I have checked the matrix, and it is not sparse.

Best Answer

Too long for a comment

python is probably not the best way to go. Even loading the whole matrix in memory is probably not possible at all in your case.

Most numerical methods rely on multiplication of the matrix times an input vector, so instead of generating the matrix and then multiplying it times a given vector you can create the matrix elements on-the-fly

I've used scalapack in the past and it is very powerfull

http://www.netlib.org/scalapack/

But it does requiere to know C/Fortran. However, if you really are out of your confort zone with this languages you can always write a wraper for python

Related Question