I disagree with your second attempt, I don't understand it completely though, it's a bit ambiguous what you're saying, but the way I interpret it sounds wrong, but your first attempt is definitely wrong, you have to show that inequality holds for the coordinates of all vectors that lie in the line segment between each two vectors in the set.
Suppose that for some arbitrarily chosen $0\leq \alpha \leq 1$ we write:
$(z_1,z_2)=\alpha (x_1,x_2) + (1-\alpha) (y_1,y_2)$
Does $z_1 \cdot z_2 \geq k$ hold?
Remember that $x_1x_2 \geq k$ and $y_1y_2\geq k$.
Addendum:
This is how I can prove what you want, I don't use $\displaystyle \frac{a}{b} + \frac{b}{a} \geq 2$ in my proof, but since I'm using AM-GM inequality, I guess that with a suitable change of variables you might find a way to write what you want by using that particular inequality (which is implied by AM-GM):
As you said, we have:
$z_1 = \alpha x_1 + (1-\alpha)y_1$
$z_2 = \alpha x_2 + (1-\alpha)y_2$
Just by doing some simple algebraic manipulations, we get:
$z_1 z_2 = \alpha^2 x_1x_2 + \alpha(1-\alpha)x_1y_2 + \alpha(1-\alpha)x_2y_1 + (1-\alpha)^2y_1y_2$
Now use your hypotheses to get:
$z_1z_2 \geq \alpha^2 k + \alpha(1-\alpha)(x_1y_2+x_2y_1) +(1-\alpha)^2k$
But by using AM-GM inequality we have:
$x_1y_2+x_2y_1 \geq 2 \sqrt{x_1x_2}\sqrt{y_1y_2}\geq 2k$
Therefore, we get:
$$z_1z_2 \geq \alpha^2k + \alpha(1-\alpha)(2k)+(1-\alpha)^2k \geq k (\alpha^2+2\alpha(1-\alpha)+(1-\alpha)^2)$$
$$z_1z_2 \geq k (\alpha + (1-\alpha))^2=k$$
Say $f(x)=\frac{1}{x}$, so our set is $\{(x,y): y\geq f(x)\}$. We know that $f(x)$ is convex, that is for $0\leq\theta\leq 1$ $$f(\theta x+(1-\theta)y)\leq \theta f(x)+(1-\theta)f(y).$$
Now take $(x_1,y_1)$ and $(x_2,y_2)$ in the set. In other words $y_1\geq f(x_1)$ and $y_2\geq f(x_2)$. For $0\leq\theta\leq 1$
$$\theta y_1+(1-\theta)y_2\geq\theta f(x_1)+(1-\theta)f(x_2)\geq f(\theta x_1+(1-\theta)x_2)$$ so it is in the set.
Best Answer
Take two points in $M$, say, $(x_1,y_1)$ and $(x_2,y_2)$. You need to prove that $$(\lambda x_1 + (1-\lambda)x_2, \lambda y_1 + (1-\lambda)y_2) \in M \Leftrightarrow \lambda y_1 + (1-\lambda)y_2 \geq \left( \lambda x_1 + (1-\lambda)x_2\right)^2$$ For every $\lambda \in (0,1)$. The right side is $$\left( \lambda x_1 + (1-\lambda)x_2\right)^2 \leq \lambda x_1^2 + (1-\lambda)x_2^2 \leq \lambda y_1+(1-\lambda) y_2$$ Here i used the convexity of $x^2$ on the first inequality, since $(x^2)'' = 2$ and the fact that $\lambda \in (0,1)$.
One more general result is that the epigraph of a function is convex iff the function is convex.