MATLAB: Is it possible to create a sparse binary (.bin) file on disk

Question

I have a project where I would like to save my results to a binary (.bin) file that is stored on disk. Results need to be saved as they are generated (so that memory can be cleared), but the order in which these results are added to the binary file is not necessarily sequential (e.g., first I write to bytes 1-100, then 1001-1100, then 301-400, etc.).

In order to write non-sequentially to a binary file, I believe that file needs to be pre-allocated on the disk in some form or another. Is it possible to create a "sparse" binary file that has an area on disk set aside but which does not require writing zeros to every bit in the .bin file? I know how many bytes the file will take up when I am done saving to it, so this isnt a problem. Alternately, is there a way for me to write non-sequentially to a binary file without pre-allocating it first?

Thanks.

Answer 1

In case anyone comes across this question looking for the same thing...at some point in the last year I figured out a much better way to do this. Make a system call to

fallocate (Linux/UNIX - create or extend file)
fsutil file createnew (Windows - create file)
fsutil file seteof (Windows - extend file)
mkfile -n (MacOS - create file)

I haven't figured out extending a file on MacOS, but since this is a very unusual use case for me I have it setup to either zero-write to the end of the file or to read the data, delete, allocate a larger file, and re-write the data when a file of MacOS needs to be sparse-extended.

This is effectively instant, since it is true write-less allocation. For example, as a test I just allocated a 4 GB file in 0.05 seconds.

That said, writing non-sequentially to a file like this can be very slow, so you might be better off adding in zeros and writing data to the end of the file on the fly as needed, but write less allocation is possible to implement from within MATLAB.