MATLAB: Mex mxCreateStructMatrix and mxSetFieldByNumber massive Matlab workspace byte use.

mex compiler

I am trying to write a mex c file which creates a structure in matlab. 59mB of structure in C of 6 uint32's becomes 1.5GBs structure in matlab.

I don't know where I'm going wrong with this code.

       plhs[13] = mxCreateStructMatrix(1, gyro_time_mark.size(), gps_time_field_count, &gps_time_names_pointers[0]);
       for (int i = 0; i < gyro_time_mark.size(); i++) {
         uint32_t* offset = (uint32_t*)&(gyro_time_mark[i]);
        for (int j = 0; j < gps_time_field_count; j++){
          mxArray *field_value;
          field_value = mxCreateNumericMatrix(1, 1, mxUINT32_CLASS, mxREAL);
          uint32_t*  copy = (uint32_t*)mxCalloc(1, sizeof(uint32_t));
          std::memcpy(copy, offset, sizeof(int32_t));
          offset++;
          mxSetData(field_value, copy);
          mxSetFieldByNumber(plhs[13], i, j, field_value);
        }
       }

The data in Matlab is correct, its just that the structure in matlab is 1.5GB in size when it should be 50MB. The number of elements is correct and the structure field count and inside types are correct. However the data use is way way above what I would expect.

Instead of a creating a structure , I tried creating an mx6 array and that did come out to ~50mB which is exactly what I would expect. How am I using mxCreateStructMatrix and mxSetFieldByNumber incorrectly?

Best Answer

I will look at this some more tomorrow, but first, this line creates a permanent memory leak:

mxSetData(field_value, copy);

You already have a data pointer that is part of field_value since it was created as a 1x1 matrix, and that data pointer is NOT on the garbage collection list since it is already part of an mxArray. So when you call mxSetData you overwrite this data pointer and lose that memory permanently. The proper way to attach a data pointer to an mxArray is:

mxFree(mxGetData(field_value));
mxSetData(field_value, copy);

That being said, you are doing this the hard way. Instead of creating a separate memory block and attaching it to field_value you should be modifying the data area of field_value directly.

What are the values of the following?

gyro_time_mark.size()
gps_time_field_count

Also, realize that each of those mxArray field_value variables takes approx 100+ bytes of memory for overhead. I don't recall the exact number, but suppose it is 120 bytes. Then for each of your uint32 values of 4 bytes, you are adding 120 bytes for the mxArray header stuff. That is an increase of 30x memory. 50MB x 30 = 1500MB = 1.5GB. So this seems to match the memory increase you are seeing if each one of those uint32 values is inside an mxArray field value in a struct. Of course, this points out why you should not store these values in a struct like you are doing ... you should store them in a simple numeric array.

Related Solutions

MATLAB: What’s wrong with this trivial C engine program

Do not destroy the fields of the struct manually. This works:

...
engPutVariable(ep, "array", array);
/* deleted: for (j=0 ... mxDestroyArray(field_value) ... */
mxDestroyArray(array);
...

Destroying the fields does not clear the corresponding pointer in the struct. But destroying the struct clears the fields automatically. Your program seems to worked for small array sizes just by accident. But the memory management is corrupted immediately.

MATLAB: Issue with Mex Function

You can't re-use mxArray variables like you are doing when you are building a cell or struct array in a mex routine. There is an official way to fix this, and an unofficial way to fix this. The official way, which has the downside of requiring a deep data copy, is:

  mxSetField(subMexMsg, 0, "field3", inputMxArray);
  mxSetField(mexMsg, 0, "field1", subMexMsg);
  mxSetField(mexMsg, 0, "field2", mxDuplicateArray(inputMxArray));

That is, every time you re-use a variable as a cell or struct element, you must actually use a deep data copy of it. Not very memory efficient if you have large variables, but that is the way it is. The reason your code is seg faulting is because when the 1st reference to inputMxArray gets destroyed, the memory for it is actually free'd. Then when the 2nd reference to inputMxArray gets destroyed, the memory for it has already been free'd so invalid memory is accessed. MATLAB didn't know there was another reference to the variable in memory when the 1st reference was destroyed.

MATLAB can, and does at the m-file level, re-use variables the way you are doing, but it increments a "reference count" for the variable to keep track of how many copies are around in cell and struct variables. It does this automatically in the background. That way, when variables get cleared, memory doesn't actually get free'd until the last reference copy gets cleared. I.e., when the 1st reference to inputMxArray gets destroyed, the only thing that happens is the reference count gets decremented by 1. No memory is free'd. When the 2nd reference to inputMxArray gets destroyed, the memory actually gets free'd since it is the last copy in memory.

There is no official way to duplicate this behavior in a mex routine using only official API routines. There is an unofficial mex routine called mxCreateReference, that will do this for you (i.e., increment the reference count by 1 instead of creating a deep data copy). To use it, simply include the prototype and call it (but you will be departing from official library code). The mxCreateReference function is intended specifically for re-using variables in cell or struct variables. E.g.,

mxArray *mxCreateReference( mxArray * );
    :
        mxSetField(subMexMsg, 0, "field3", inputMxArray);
  mxSetField(mexMsg, 0, "field1", subMexMsg);
  mxSetField(mexMsg, 0, "field2", mxCreateReference(inputMxArray));

The return value (mxArray address) is the same as the input value.

Unfortunately, TMW has removed mxCreateReference in later versions of MATLAB. The only recourse left to the programmer at this point would be to hack into the mxArray to bump up the reference count manually. Let me know if you need this.

For comparison, mxSetField, mxSetFieldByNumber, and mxSetCell do the following to the input variable:

1) Changes the variable type to 3 (SUB_ELEMENT)
2) Removes the variable address from the garbage collection

Best Answer

Related Solutions

MATLAB: What’s wrong with this trivial C engine program

MATLAB: Issue with Mex Function

Related Question