Your tests are with symbolic x, but fmincon runs with numeric x. So f = 2*x(1)+3*x(2)^2+exp(2*x(1)^2+x(2)^2) is going to produce a numeric scalar because x(1) and x(2) will have particular numeric values. You then apply the numeric gradient function to that scalar, giving the x(1) and x(2) as the step sizes (which is obviously wrong), but since f is only a scalar the step sizes are irrelevant and the numeric gradient is going to return just the scalar.
What you should probably do is
function [f,grad,hess] = exam_sylvi(x)
syms x1 x2
F = 2*x+3*x2^2+exp(2*x1^2+x2^2);
f = double( subs(F, [x1 x2], x) );
if nargout > 1
grad = double( subs( gradient(F,[x1, x2]), [x1, x2], x) );
end
if nargout > 2
hess = double( subs( hessian(F,[x1, x2]), [x1, x2], x) );
end
end
Except, of course, that would be inefficient and you would be better to pre-calculate those before calling fmincon:
syms x1 x2
F = 2*x+3*x2^2+exp(2*x1^2+x2^2);
fh = matlabFunction(F, 'vars', {x1, x2});
gradh = matlabFunction( gradient(F, [x1, x2]), 'vars', {x1, x2});
hessh = matlabFunction( hessian(F, [x1, x2]), 'vars', {x1, x2});
[x,fval,exitflag,output] = fminunc(@(x) exam_sylvi(x, fh, gradh, hessh), [1,0], options)
Together with
function [f, grad, hess] = exam_slvi(x, fh, gradh, hessh)
f = fh(x);
if nargout > 1
grad = gradh(x);
end
if nargout > 2
hess = hessh(x);
end
end
Best Answer