Given the following composition of functions:
$h:\Bbb R^k\rightarrow\Bbb R$
$g:\Bbb R^n\rightarrow\Bbb R$
$f(x)=h(g_1(x),g_2(x),…,g_k(x))$
There are known rules which guarantee convexity/concavity of $f$, for example:
$f$ is convex if: $h$ is convex, $h$ is nondecreasing in each argument, and $g_i$ are convex.
It seems all such rules require $h$ to be monotonic. Are there any such convexity rules that do not require monotonicity of $h$? In other words, given a convex/concave nonmonotonic function $h$ is there any way to choose $g_i$ such that $f$ is convex/concave?
To clarify what my goal is: I have a convex function $h$ which I know, and I want to let an external "user" supply $g$. I do not know what $g$ will be but I want to be able to provide guidelines on what types of $g$ would allow $f$ to be convex function.
Best Answer
Having your clarification in mind, I'll give (one case of) an example to show that every $h$ has a bad $g$. More exactly: for any convex nonmonotonic $h\colon\mathbb{R}\to\mathbb{R}$, there exists a convex $g\colon\mathbb{R}\to\mathbb{R}$ such that $h\circ g$ is neither concave nor convex.
I'll treat just the case where there exist $a,b,c$ such that $a<b<c$ and $h(b)<h(a)<h(c)$. (Other cases are similar, or can be reduced to this one with some fiddling.) Define $g(x) = a+|x|$. Then $$ h(g(\tfrac12(b-a)+\tfrac12(a-b))) = h(a) > h(b) = \tfrac12 h(g(b-a)) + \tfrac12 h(g(a-b)) $$ which shows that $h\circ g$ is not convex, and $$ h(g(\tfrac12(c-a)+\tfrac12(a-c))) = h(a) < h(c) = \tfrac12 h(g(c-a)) + \tfrac12 h(g(a-c)) $$ which shows that $h\circ g$ is not concave.