I'd appreciate some feedback on whether or not I'm thinking about this problem correctly.
Show whether or not the subsets $Y \subset [0,1]\times[0,1] = X$ are open in $X$ when $X$ has the dictionary order topology. If $Y$ is open in $X$, express $Y$ as a union of basis elements. If $Y$ is not open in $X$, show that some point in $Y$ does not have a neighborhood in $Y$.
a) $(0,1)\times(0,1) \subset [0,1]\times[0,1]$
b) $(0,1)\times[0,1] \subset [0,1]\times[0,1]$
c) $[0,1]\times(0,1) \subset [0,1]\times[0,1]$
Since our set $X = [0,1]\times[0,1]$ has a least element $[0,0]$ and a greatest element $[1,1]$, then a basis $\mathcal B$ for the order topology on $X$ is $$\mathcal B = \{(x,y): x,y \in X, x < y\} \cup \{[0,y): y \in X\}\cup \{(x,1]: x \in X\}.$$
Thus,
a) $(0,1)\times(0,1)$ = $(0\times0, 1\times1)$
b) $(0,1)\times[0,1]$
not open since the smallest coordinate $0\times0$ does not have a neighborhood in $Y$, since the
x-coordinate of $0\times0$ is not in $Y$. Also, the largest coordinate $1\times1$ does not have a neighborhood in $Y$, since the $x$-coordinate of $1\times1$ is not in $Y$.
c) $[0,1]\times(0,1)$
not open since the smallest coordinate $0\times0$ does not have a neighborhood in $Y$, since the
$y$-coordinate of $0\times0$ is not in $Y$. Also, the largest coordinate $1\times1$ does not have a neighborhood in $Y$, since the $y$-coordinate of $1\times1$ is not in $Y$.
Best Answer
It looks to me as if you have a fundamental misunderstanding of some kind. In your answer to (b) you write:
This doesn’t make sense: $Y$ is a set of ordered pairs of real numbers, so individual coordinates of those ordered pairs can’t possibly be elements of $Y$. In any case the point $1\times 1$, or $\langle 1,1\rangle$ as I prefer to write it, isn’t in $Y=(0,1)\times[0,1]$ anyway, since its first coordinate isn’t in $(0,1)$, so whether $\langle 1,1\rangle$ has an open nbhd in $Y$ is irrelevant.
As it happens, $(0,1)\times[0,1]$ is open. For each $x\in\left(0,\frac12\right)$ try to sketch the open interval
$$\left(\left\langle x,\frac12\right\rangle,\left\langle 1-x,\frac12\right\rangle\right)\;,$$
or in your notation
$$\left(x\times\frac12,(1-x)\times\frac12\right)\;;$$
can you show that every point of $Y$ is in one of these open intervals, and that each interval is a subset of $Y$?
In (a) the set $Y=(0,1)\times(0,1)$ is indeed open, but it’s not equal to $(0\times 0,1\times 1)$: $(0\times 0,1\times 1)$ is all of $X$ except the two endpoints $\langle 0,0\rangle$ and $\langle 1,1\rangle$. HINT: Note that for each $x\in(0,1)$ the interval
$$\big(\langle x,0\rangle,\langle x,1\rangle\big)=(x\times 0,x\times 1)$$
is a subset of $Y$.
In (c) you’ve made the same error as in (b): it makes no sense to ask whether a coordinate of a point is in $Y$, and anyway the point $\langle 0,0\rangle$ isn’t in $Y$ in the first place. Do (a) first; once you’ve done it, this one should be pretty straightforward.