Id like to be able to memory map a set of actual memory addresses and access data from those addresses from multiple instances of Matlab. Is this possible?
Note: This differs from standard memory mapping in the sense that I do NOT want to map some data on disk to virtual memory addresses, I want to map actual memory addresses and access the data stored in them from multiple instances of Matlab.
Thanks!
.
.
Note on Intended usage:
Without getting into too many details, I envision using this in a way such that one instance reads data into the memory-mapped memory addresses (perhaps with some zero padding) (some data headers will also get stripped out and delt with separately in this step), one or more instances processes the data (each instance would use the output aof the previous instance as an input), and one final instance takes data that has finished processing and writes it to disk in a new variable.
This same set of memory addresses would be cyclically reused multiple times to process the entire dataset (A bit over 2TB of data in my immediate use case). Each processing step only requires a local subset of the data that is saved sequentially in the data (no more than ~3000 uint32 values, and as little as a single uint32 value). Each instance would memory-map a "progress indicator" to ensure that other instances dont access data before it is ready for them as well as to ensure that data isnt overwritten until after the last instance has finished saving the processed data to disk.
This is conceptually a bit like "piping" the data in the sense that a given operation would start working on data before the previous operation has completely finished, though is different in the sense that the data is returned to main memory in between each step and each step is performed by different CPU cores (this makes it so that single threaded operations like fread and fwrite dont force you to not use the other CPU cores). It might make sense to lock each Matlab instance to the CPU core(s) you want it to use (to prevent cache thrashing and other potential problems).
.
.
Side Note: if this isnt directly possible, has anyone tried creating a ramdisk and then memory mapping that? Any chance Matlab is smart enough to realize that the data is already in memory and not try to remap the physical memory adresses to virtual ones?
(I dont know if this would affect performance much, but I have A LOT of data to run through and am concerned with overusing virtual memory addresses resulting in out of memory issues).
(The machine this will run on also has a rather unique NUMA related problem, where if I limit the Matlab instance to using 1 NUMA node and run out of memory (virtual+physical) on that node, the process of moving stuff (such as cached data) to another NUMA node can be very slow in some situations. This can slow things down by a factor of 20x during parts of my code where Matlab is requesting additional memory).
Best Answer