You wrote:
We need to find an $A\times B$ open in $\mathbb{R}\times\mathbb{R}$ such that
This would be true if we were working with the product topology on $\mathbb R\times\mathbb R$. But in this example we are working with the order topology coming from the lexicographic order.
In this topology on $\mathbb R\times\mathbb R$ the set $\{1/2\}\times(1/2,3/2)$ is open, since it is precisely the set of all points which are between $(1/2,1/2)$ and $(1/2,3/2)$ (w.r.t. the linear order which we are working with).
The set $\{1/2\}\times(1/2,1]$ is not open in order topology (from the lexicographic order on $I\times I$) since every neighborhood of the point $(1/2,1)$ contains some points with $x$-coordinate greater than $1/2$. (Since the point $(1/2,1)$ does not have an immediate successor nor is it the greatest element in this order.)
It looks to me as if you have a fundamental misunderstanding of some kind. In your answer to (b) you write:
Also, the largest coordinate 1 x 1 does not have a neighborhood in Y, since the x-coordinate of 1 x 1 is not in Y.
This doesn’t make sense: $Y$ is a set of ordered pairs of real numbers, so individual coordinates of those ordered pairs can’t possibly be elements of $Y$. In any case the point $1\times 1$, or $\langle 1,1\rangle$ as I prefer to write it, isn’t in $Y=(0,1)\times[0,1]$ anyway, since its first coordinate isn’t in $(0,1)$, so whether $\langle 1,1\rangle$ has an open nbhd in $Y$ is irrelevant.
As it happens, $(0,1)\times[0,1]$ is open. For each $x\in\left(0,\frac12\right)$ try to sketch the open interval
$$\left(\left\langle x,\frac12\right\rangle,\left\langle 1-x,\frac12\right\rangle\right)\;,$$
or in your notation
$$\left(x\times\frac12,(1-x)\times\frac12\right)\;;$$
can you show that every point of $Y$ is in one of these open intervals, and that each interval is a subset of $Y$?
In (a) the set $Y=(0,1)\times(0,1)$ is indeed open, but it’s not equal to $(0\times 0,1\times 1)$: $(0\times 0,1\times 1)$ is all of $X$ except the two endpoints $\langle 0,0\rangle$ and $\langle 1,1\rangle$. HINT: Note that for each $x\in(0,1)$ the interval
$$\big(\langle x,0\rangle,\langle x,1\rangle\big)=(x\times 0,x\times 1)$$
is a subset of $Y$.
In (c) you’ve made the same error as in (b): it makes no sense to ask whether a coordinate of a point is in $Y$, and anyway the point $\langle 0,0\rangle$ isn’t in $Y$ in the first place. Do (a) first; once you’ve done it, this one should be pretty straightforward.
Best Answer
Let $(p,0)$ be in $[0,1]^2$ in this order topology, where $0 < p$. A basic open neighbourhood of $(p,0)$ is an open interval with endpoints in $[0,1]^2$, say $\langle (l_1,l_2), (r_1,r_2)\rangle$. It follows by the definition of the lexicographic order that $l_1 < p$. A moment's thought reveals that we can in fact only need to consider basic open sets of the form $\langle (l_1,1), (p,r_2)\rangle$ for $(p,0)$ (with $0< l_1 < p, r_2 >0$), by shrinking such an open interval.
For a point $(q,1), q < 1$ we likewise only need to consider open basic sets of the form $\langle (q,l_1), (r_1,0)\rangle$ (with $l_1 > 0, q < r_1 < 1$).
It's then easy to see that a point $(p,0) \in \overline{S}$ iff $p \in (a,b]$ and $(q,1) \in \overline{S}$ iff $q \in [a,b)$.