The input of every function includes 'x', but there's no x in the function body. Actually, you should write your function as follows:(and it's a bit weird without x(2) in your target function...)
function f=mae342hw5p2(x)
f=x(1).^3-6.*x(1).^2+11.*x(1)+x(3);
For the constraints, it's better to separate the linear and non-linear constraints, which makes the non-linear constraints function to be:
function [c,ceq]=confun(x)
c=[ x(1)^2+x(2)^2-x(3)^2; 4-x(1)^2-x(2)^2-x(3)^2];
ceq=[];
the linear constraints can be set by upper and lower boundary when you call fmincon: lb=[-Inf,-Inf,0];ub=[0,0,5];
Hope you understand!
Best Answer