I have taken two elements $(x_1,y_1)$ and $(x_2,y_2)$ in the set.
I want to check that $(x_3,y_3) = \theta(x_1,y_1) + (1-\theta)(x_2,y_2) \le 1$
I referenced Tom's answer on this question: Prove that the set $A := \left\{ (x,y) \in \Bbb R_{> 0}^2 \mid xy \geq 1 \right\}$ is convex
So I need to check that $\theta^2x_1y_1 + (1-\theta)^2x_2y_2 + \theta(1-\theta)[x_1y_2+x_2y_1] \le 1$
So $x_1y_1 \le 1$, $x_2y_2\le1$
But $[x_1y_2+x_2y_1] \ge 2\sqrt{x_1y_1x_2y_2} \le 2$
I don't think I can use the same proof since the inequalities are in different directions.
Best Answer
To show that a set is nonconvex, you need the negation of the definition of convexity:
$$\exists(x_1,y_1),(x_2,y_2)\in A:\exists \theta \in (0,1):\theta(x_1,y_1)+(1-\theta)(x_2,y_2)\notin A$$
That is,
$$\exists(x_1,y_1),(x_2,y_2)\in A:\exists \theta \in (0,1):(\theta x_1+(1-\theta)x_2)(\theta y_1+(1-\theta)y_2)>1$$
and we see that $(0.5, 2), (2,0.5)$ with $\theta = 0.5$ does the trick:
$$(0.5\times 0.5+0.5\times 2)(0.5\times2+0.5\times 0.5)=1.25^2>1$$
Hence $A$ is nonconvex.
If the inequality in your question do hold, we would have proven that $A$ is convex.