You wrote:
We need to find an $A\times B$ open in $\mathbb{R}\times\mathbb{R}$ such that
This would be true if we were working with the product topology on $\mathbb R\times\mathbb R$. But in this example we are working with the order topology coming from the lexicographic order.
In this topology on $\mathbb R\times\mathbb R$ the set $\{1/2\}\times(1/2,3/2)$ is open, since it is precisely the set of all points which are between $(1/2,1/2)$ and $(1/2,3/2)$ (w.r.t. the linear order which we are working with).
The set $\{1/2\}\times(1/2,1]$ is not open in order topology (from the lexicographic order on $I\times I$) since every neighborhood of the point $(1/2,1)$ contains some points with $x$-coordinate greater than $1/2$. (Since the point $(1/2,1)$ does not have an immediate successor nor is it the greatest element in this order.)
The solution is explained well in the comments, but I thought I would draw a few pictures to clarify things further!
Consider the dictionary order topology on $(0, \ 1) \ \times \ (0, \ 1)$, which consists of all sets of the form $(a \ \times \ b, \ c \ \times \ d)$, with $0 \ < \ x \ < \ 1$ for $x \ = \ a, \ b, \ c, \ d$ and $a \ < \ c$ or $a \ = \ c$ and $b \ < \ d$.
Now, the dictionary order topology on $\mathbb{R} \ \times \ \mathbb{R}$ behaves the same way, but consists of dictionary ordered open intervals of all real numbers. We then take the intersection of all these sets with $(0, \ 1) \ \times \ (0, \ 1)$ to get the subspace topology.
The best way to prove this is to simply visualize it. Consider the two forms of open set for the first topology we described:
Then consider sets of this same form in $\mathbb{R} \ \times \ \mathbb{R}$, intersecting $(0, \ 1) \ \times \ (0, \ 1)$:
Proving this is just a matter of taking these pictures, and writing down what is visually true (the topologies are the same), in more rigorous language, which is fairly straightforward once the intuition is gained with the pictures.
Best Answer
The base for the topology is all sets of the form $[0\times 0, a \times b)$ and $(a \times b, 1\times 1]$ and all open intervals of the form $(a \times b, c \times d)$ where both endpoints lie in the square. (we have to treat the minimum and maximum of the set a bit differently).
I think (C) is indeed the correct option.