MATLAB: Do I get a runtime error concerning the “ESP” register when attempting to call a MEX-file from a C++-based test harness

Question

I've created a test harness for MEX-files, written in C++. It includes the following lines of code:

 #include <windows.h>
 // some lines
 typedef void (CALLBACK* PFUN)(int, mxArray**, int, mxArray**);
 HINSTANCE hDLL = LoadLibrary("MyMexFunction.mexw32");
 PFUN f;
 f = (PFUN)GetProcAddress(hDLL,"mexFunction");

Later, "f" is used to call the MEX-function.

While I can call my MEX-file from MATLAB without problem, I receive a runtime error such as:

 Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call.  This is usually a result of calling a function declared with one calling convention with a function pointer declared with a different calling convention.

when I attempt to run my test harness.

Answer 1

This issue stems from your use of the CALLBACK macro. This is a macro for the STDCALL calling convention, but MEX-files always use the CDECL convention. This conflict leads to the runtime error.

You should be able to resolve this issue by using:

 typedef void (*LPFNDLLFUNC1)(int, mxArray**, int, mxArray**);

instead of

 typedef void (CALLBACK* LPFNDLLFUNC1)(int, mxArray**, int, mxArray**);