I disagree with your second attempt, I don't understand it completely though, it's a bit ambiguous what you're saying, but the way I interpret it sounds wrong, but your first attempt is definitely wrong, you have to show that inequality holds for the coordinates of all vectors that lie in the line segment between each two vectors in the set.
Suppose that for some arbitrarily chosen $0\leq \alpha \leq 1$ we write:
$(z_1,z_2)=\alpha (x_1,x_2) + (1-\alpha) (y_1,y_2)$
Does $z_1 \cdot z_2 \geq k$ hold?
Remember that $x_1x_2 \geq k$ and $y_1y_2\geq k$.
Addendum:
This is how I can prove what you want, I don't use $\displaystyle \frac{a}{b} + \frac{b}{a} \geq 2$ in my proof, but since I'm using AM-GM inequality, I guess that with a suitable change of variables you might find a way to write what you want by using that particular inequality (which is implied by AM-GM):
As you said, we have:
$z_1 = \alpha x_1 + (1-\alpha)y_1$
$z_2 = \alpha x_2 + (1-\alpha)y_2$
Just by doing some simple algebraic manipulations, we get:
$z_1 z_2 = \alpha^2 x_1x_2 + \alpha(1-\alpha)x_1y_2 + \alpha(1-\alpha)x_2y_1 + (1-\alpha)^2y_1y_2$
Now use your hypotheses to get:
$z_1z_2 \geq \alpha^2 k + \alpha(1-\alpha)(x_1y_2+x_2y_1) +(1-\alpha)^2k$
But by using AM-GM inequality we have:
$x_1y_2+x_2y_1 \geq 2 \sqrt{x_1x_2}\sqrt{y_1y_2}\geq 2k$
Therefore, we get:
$$z_1z_2 \geq \alpha^2k + \alpha(1-\alpha)(2k)+(1-\alpha)^2k \geq k (\alpha^2+2\alpha(1-\alpha)+(1-\alpha)^2)$$
$$z_1z_2 \geq k (\alpha + (1-\alpha))^2=k$$
$$M = \cap_{y \in S} M_y$$
where,
$$M_y = \{x|\quad \|x - x_0\|_2 \leq \|x - y\|_2 \quad\}$$
We can try to show that $M_y$ is convex. Now,
$$\|x - x_0\|_2 \leq \|x - y\|_2 \\
\Rightarrow \|x - x_0\|_2^2 \leq \|x - y\|_2^2 \\
\Rightarrow \|x\|_2^2 + \|x_0\|_2^2 - 2x_0^{\top}x \leq \|x\|_2^2 + \|y\|_2^2 - 2y^{\top}x \\
\Rightarrow (y - x_0)^{\top}x \leq \frac{1}{2} \left( \|y\|_2^2 - \|x_0\|_2^2 \right)$$
So, $M_y$ is a Half Space, which is a convex set (which is fairly easy to show as well).
Finally, $M$ is the intersection of convex sets which is also convex.
Best Answer
We have $(\pm 2\sqrt2)^2=8 \ge 8(1)$ hence we can conclude that
$(1, 2\sqrt2)\in S$ and $(1, -2\sqrt2)\in S$.
However, their midpoint, $(1, 0)$ is not in $S$ since
$$0^2 < 8(1)$$
It is not convex.
We can make it more general, let $t>0$, then we have $(t, \pm 2\sqrt2 t) \in S$ but $(t,0)\notin S.$
More detailed reasoning in proving non-convexity.
If $S$ is convex, then for any $(a, b), (c,d) \in S$, then $\forall \alpha \in (0,1), \alpha (a, b) + (1-\alpha)(c,d) \in S. $
To exhibit that $S$ is not convex, it means that we can find $(a,b), (c,d) \in S$ such that there is such an $\alpha \in (0,1)$, $\alpha (a,b) + (1-\alpha) (c,d) \not\in S$.
We pick $(a,b)=(1, 2\sqrt2), (c,d) = (1, -2\sqrt2)$, we can check that $(a,b), (c,d) \in S$. Pikcing $\alpha =0.5$ illustrates that it is not convex.