Let us consider the following two variable function $$f(x,y) = x^2y^3$$ with domain as the first quadrant. To find if the function is convex/concave/quasi-convex/quasi-concave.
$f_x = 2xy^3, f_y = 3x^2y^2$
Then we have $f_{xx} = 2y^3, f_{yy} = 6x^2y$ and $f_{xy} = 6xy^2$.
Hence the Hessian matrix is given by $$\begin{bmatrix}
2y^3 & 6xy^2\\
6xy^2 & 6x^2y
\end{bmatrix}.$$
Note that determinant of the Hessian matrix is $12x^2y^4 – 36 x^2y^4 = – 24x^2y^4<0$.
Hence the function is neither convex or concave.
Next, how can I verify if the function is quasi-convex/quasi-concave?
Best Answer
A quasiconvex function is a real-valued function defined on $\,\Bbb R^n$ or on a convex subset of $\,\Bbb R^n$ such that the inverse image of any set of the form $\,(-\infty,a]\,$ is a convex set.
A function $\,f\,$ is said to be quasiconcave if its negative $\,g=-f\,$ is a quasiconvex function.
The function $\,f(x,y)=x^2y^3:\Bbb R^2\to\Bbb R\,$ is not quasiconvex, indeed the inverse image of $\,(-\infty,1]\,$ is not a convex set.
Look at this picture :
Moreover, the function $\,f(x,y)=x^2y^3:\Bbb R^2\to\Bbb R\,$ is not quasiconcave, indeed the inverse image of $\,(-\infty,1]\,$ of the function $\,g(x,y)=-f(x,y)=-x^2y^3\,$ is not a convex set.
Look at this picture :
But if the domain of the function $\,f(x,y)=x^2y^3\,$ is the first quadrant, then the function $\,f(x,y)\,$ is quasiconcave, indeed the inverse image of $\,(-\infty,a]\,$ of $\,g(x,y)=-f(x,y)=-x^2y^3\,$ is the following set $A:$
$A=\begin{cases}[0,+\infty)^2&\text{ if }\;a\geqslant0\\[3pt]\left\{(x,y)\;\big|\; x>0\;\land\;y\geqslant\sqrt[3]{\frac{-a}{x^2}}\right\}&\text{ if }\;a<0\end{cases}$
which, in any case, is a convex set.
In the following picture there is the set $\,A\,$ for $\,a=-1:$