MATLAB: How to query the size of the largest contiguous block of memory available to MATLAB on Linux

linuxMATLABmemory

I am looking for a command that can tell me the size of the largest matrix that can be stored in MATLAB. I would like something similar to 'feature memstats' on Windows.

Best Answer

There is no built-in MATLAB function for querying the largest contiguous block on Linux. However, the following location on MATLAB File Exchange has a similar function available:

 <http://www.mathworks.com/matlabcentral/fileexchange/25725-largest-memory-block-available-all-platforms>

Note that MathWorks does not guarantee or warrant the use or content of these submissions. Any questions, issues, or complaints should be directed to the contributing author.

Related Solutions

MATLAB: Does MATLAB use the same memory address for variables that have different names but the same value

When MATLAB creates a variable, it actually creates two elements of data. One part is the data itself, the other part is a header that contains information about the data (such as where in memory the data lives, how big it is, etc). For more information see the External Interfaces/API Manual which may be found at:

https://www.mathworks.com/help/releases/R13sp2/techdoc/matlab_external/matlab_external.html

When executing an assignment such as 'b = a', MATLAB creates a new header for 'b', but just tells the data element for 'b' to point to the data element for 'a'. This is done for efficiency. Until there is a definite need to create a separate copy of data for 'b', 'a' and 'b' will share the same data. When the value of the data for 'a' or 'b' changes, MATLAB will make a separate copy of the data for each variable. This is called "copy on write".

The output of WHOS may make it seem like MATLAB is allocating memory for each variable, but this is not always the case. WHOS will show you the memory usage assuming each variable has its own data set.

For example, after entering in 'b=a', any of the commands below will cause MATLAB to allocate new storage space for 'b'.

b(1)=7;
b=2*a;
a(1)=2;
b = b+0;

This feature allows MATLAB to be more efficient, allocating memory only when needed.

MATLAB: How to maximize total available memory and largest contiguous memory block

Here are some resources to try if you see memory allocation errors, or you suspect that memory allocation is behind a problem. Most of these resources are specific to 32-bit Windows.

1. Consult product documentation for memory allocation.

https://www.mathworks.com/help/matlab/ref/memory.html

2. Consult Tech Notes "1806 - Simulink Performance & Memory Management Guide":

https://www.mathworks.com/company/newsletters/articles/improving-simulation-performance-in-simulink.html

3. Use the Windows Task Manager or Process Explorer to see how much memory MATLAB is using.

https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer

4. Use MATLAB commands to see what is in MATLAB's memory.

a. Run "feature memstats" to get a summary of memory usage.

b. Run "feature dumpmem" for a long list of all the allocated memory blocks. MATLAB.exe, DLLs, and MEX files appear by name. Everything else appears as "<anonymous>", including storage for MATLAB variables, MATLAB code, Simulink blocks, and JAVA classes. See below for more about interpreting dumpmem output.

c. Run "whos" to see MATLAB variables and how many bytes they require.

d. Run "inmem" to see what MATLAB functions, MEX files, and JAVA classes are loaded.

e. Run "find_system('type', 'block_diagram') to what Simulink models and libraries are loaded.

5. Use "clear" to remove variables, functions, and class definitions from memory. Use "close_system" to remove Simulink block diagrams from memory.

6. Saving your work and restarting MATLAB is the most effective way to clear everything from memory. It is the *only* way to defragment the MATLAB workspace on 32-bit Windows system. If an operation sometimes runs out of memory, try it immediately afteer restarting MATLAB.

Run "matlab -h" for terse documentation of MATLAB command line switches.

7. The “-nojvm” command line switch disables many capabilites, but saves a lot of memory. Here are some results. Memory usage is in Kilobytes.

R2007b:

Physical memory: (155436 with jvm - 53296 without jvm) = 102140 difference.

Virtual memory: (524040 with jvm - 172824 without jvm) = 351216 difference.

R2008a:

Physical memory: (142464 with jvm - 57576 without jvm) = 84886 difference.

Virtual memory: (468052 with jvm - 142464 without jvm) = 325588 difference.

8. The "-shield <level>" command line switch controls how aggressively MATLAB tries to maximize large blocks of free memory by forcing DLLs to load elsewhere. There are four levels:

"-shield none" disbles shielding.  MATLAB does nothing to protect a large free block of memory.
"-shield minimum" protects 0x5000000-0x7000000 address range until before matlabrc.m is processed.
"-shield medium" protects 0x5000000-0x7000000 address range until *after* matlabrc.m is processed.
"-shield maximum" protects a calculated maximum address range until after matlabrc.m is processed.

Levels medium and maximum are experimental.

9. The "-shieldload < list > " command line switch loads dlls in < list > while protecting part of the memory address space as controlled by the "-shield" switch. Doing so forces those dlls to relocate to memory addresses that do not break up the large free block.

To find useful arguments for shieldload, start MATLAB and use ‘feature dumpmem’ at the MATLAB command line to get names and addresses of all the loaded dlls, as described in the following example. Output from 'feature dumpmem' included the following:

                        Module                                                                Base            In Use           Free  
 ============================================================
...
 <anonymous>                                                   06f70000    00800000*   08890000**
 C:\Program Files\Lenovo\HOTKEY\hkvolkey.dll 10000000    00011000    0ffef000**
 C:\WINNT\system32\xpsp2res.dll                     20000000    002c5000*   256cb000***
 C:\WINNT\System32\spool\DRIVERS\W32X86\3\PS5UI.DLL          45990000    000b5000    1ab8b000***
 C:\WINNT\system32\mslbui.dll                           605d0000    00009000    114c7000***
 C:\WINNT\system32\WS2HELP.dll                     71aa0000    00008000    00008000
... 
 Largest available memory block is 627879936 bytes (598.79 MB) located at address 202c5000

Columns Base, In Use, and Free are hexadecimal numbers. The last three digits are 000, because memory is allocated in pages of 0x1000 (or 4096) bytes. The named or anonymous "Module" occupies "In Use" bytes, starting at address "Base". There are "Free" unused bytes between the end of that module and the begining of the next. Base + In Use + Free = next(Base)

A '+' after the module name means the module was relocated because the addresses at which it usually loads were in use.

Asterisks after In Use or Free flag blocks of at least 1, 16, or 256 Megabytes.

Modules hkvolkey.dll, xpsp2res.dll, PS5UI.DLL, and mslbui.dll occupy about 3.6 MB, but they are placed in such a way that they reduce the largest free block of memory from about 1600 MB to about 599 MB.

Can -shieldload force those modules to load somewhere else in memory? After starting MATLAB with the command line:

matlab -nojvm -shieldload "C:\Program Files\Lenovo\HOTKEY\hkvolkey.dll,C:\WINNT\system32\xpsp2res.dll,C:\WINNT\System32\spool\DRIVERS\W32X86\3\PS5UI.DLL,C:\WINNT\system32\mslbui.dll"

the relevant part of ‘feature dumpmem’ output was:

                       Module                                                                Base            In Use           Free  
 ============================================================
...
 <anonymous>                                                 07140000    00080000    08e40000** 
 C:\Program Files\Lenovo\HOTKEY\hkvolkey.dll                 10000000    00011000    0ffef000** 
 C:\WINNT\system32\xpsp2res.dll                              20000000    002c5000*   256cb000***
 C:\WINNT\System32\spool\DRIVERS\W32X86\3\PS5UI.DLL          45990000    000b5000    2c05b000***
 C:\WINNT\system32\WS2HELP.dll                               71aa0000    00008000    00008000   
...
 Largest available memory block is 738570240 bytes (704.36 MB) located at address 45a45000

Apparently hkvolkey.dll, xpsp2res.dll, and PS5UI.DLL are locked down.

How well does “-shield maximum” work? After starting MATLAB with the command line:

“matlab -nojvm -shield maximum”

the relevant part of ‘feature dumpmem’ output was:

                       Module                                                                Base            In Use           Free  
 ============================================================
...
 <anonymous>                                                 07a10000    00080000    08570000** 
 C:\Program Files\Lenovo\HOTKEY\hkvolkey.dll                 10000000    00011000    0ffef000** 
 C:\WINNT\system32\xpsp2res.dll                              20000000    002c5000*   4030b000***
 C:\WINNT\system32\mslbui.dll                                605d0000    00009000    17b57000***
...
 Largest available memory block is 1076932608 bytes (1027.04 MB) located at address 202c5000

That was better.

Finally, do –shield and –shieldload together work better than either alone? After starting MATLAB with the command line:

matlab -nojvm -shield maximum -shieldload C:\WINNT\system32\mslbui.dll

the relevant part of ‘feature dumpmem’ output was:

                       Module                                                                Base            In Use           Free  
 ============================================================
...
 <anonymous>                                                 07990000    00080000    085f0000** 
 C:\Program Files\Lenovo\HOTKEY\hkvolkey.dll                 10000000    00011000    0ffef000** 
 C:\WINNT\system32\xpsp2res.dll                              20000000    002c5000*   57e6b000***
 C:\WINNT\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.762_x-ww_6b128700\MSVCR80.dll78130000    0009b000    00005000   
...
Largest available memory block is 1474736128 bytes (1406.42 MB) located at address 202c5000

Still no luck moving hkvolkey.dll, xpsp2res.dll, but we've increased the largest block from 599 MB to 1406 MB.

Best Answer

Related Solutions

MATLAB: Does MATLAB use the same memory address for variables that have different names but the same value

MATLAB: How to maximize total available memory and largest contiguous memory block

Related Question