MATLAB: What’s the Largest Recorded Data Set Ever Used with MATLAB

Question

Does The Mathworks have any information concerning the largest known data sets to have ever been used with MATLAB?

I'm now working with extremely large matricies and am trying to determine what kind of hardware would be needed in order to continue my work. The trial-and-error approach is proving expensive, so it would help to understand realistic expectations.

I once believed that as long as I have a computer with enough RAM to hold my matricies, I'd be set. But now I know that allowable data flow between memory and processors, is a critical factor, along with my number of cores. Rather than dump more money into the latest GPU's, and see what happens, I'd like to know what others have already done. Quite simply put, if they're using greater hardware than I could ever afford, and are still struggling to meet similar goals, I'd like to know.

If The Mathworks, or anyone, is aware of huge MATLAB projects, and their outcomes, then I'd like to hear about it. What are the sizes of their data sets, and what hardware are they using in order to reach their goals?

I've been unable to find hard numbers on matrix sizes, hardware being used, and corresponding run times. Any known cases, with specific numbers, would be greatly appreciated.

Answer 1

Just how big is your data? The biggest I've worked on personally is CT data of about 20 GB per image, though I've only worked on a slice at a time since it was unnecessary to have the whole thing in memory at one time. I've heard of people with mass spectrometry images that are about 100 GB. I think some people in geology (oil, gas, earthquakes, etc.) have 3-D images that are terabytes. NASA has dozens of probes and receives hundreds of terabytes every hour!

For really large images, you can use memmapfile() to get around having to have the whole image in memory at one time, though I personally have not used the function.