This is question 7N #2 from Willard's General Topology, on p. 51.
For any topological space $X$, let $H(X)$ denote the group of
homeomorphisms of $X$ onto itself, with composition as the group
operation. Let $X = [0,1]$ and $Y = (0,1)$ and define $\phi(h) = h\restriction_Y$, for each $h \in H(X)$. Then $\phi$ is an isomorphism
of $H(X)$ with $H(Y)$ but there is no homeomorphism of $X$ onto $Y$.
I have already seen that $[0,1]$ is not homeomorphic to $(0,1)$ (i.e. $X \nsim Y$), so I am just trying to show that $\phi: H(X) \to H(Y)$ is an isomorphism. I am hoping that someone can double check my proof for showing that $\phi$ preserves the composition operation and that $\phi$ is injective and hoping that someone can help me show that $\phi$ is also surjective, which I am struggling with.
First, let $h_1, h_2 \in H(X)$. Then:
$\phi(h_1 \circ h_2) = (h_1 \circ h_2)\restriction_Y = (h_1\restriction_Y) \circ (h_2\restriction_Y) = \phi(h_1) \circ \phi(h_2)$.
I'm not sure if the move from the first equality to the second is obvious. It seems like it should be but I'm not sure if I need to justify it more. If this argument works, it shows that $\phi$ preserves the group operation.
Second, suppose $h_1, h_2 \in H(X)$ and $\phi(h_1) = \phi(h_2)$. This implies that $h_1\restriction_Y = h_2\restriction_Y$, i.e. $h_1(x) = h_2(x) (\forall x \in (0,1) = Y)$. But since $h_1, h_2$ are continuous on $X = [0,1]$, then $h_1(x) \to h_1(0)$ and $h_2(x) \to h_2(0)$ as $x \to 0$. Since $h_1(x) = h_2(x) \forall x \in (0,1)$, then they must converge to the same point, by uniqueness of limits. So $h_1(0) = h_2(0)$. A similar argument holds when taking $x \to 1$, so we can conclude that $h_1(x) = h_2(x) \forall x \in [0,1]$, i.e. $h_1 = h_2$ on $X$, so $\phi$ is injective.
It remains to show that $\phi$ is surjective. This is where I'm less sure.
My idea is to take some $g \in H(Y)$ and a sequence like $(x_n) = (\frac{1}{n+1}), n \in \mathbb{N}$. Clearly $x_n \to 0$. Since $x_n \in Y$ for each $n \in \mathbb{N}$ then we can define the sequence $(g_n) = (g(\frac{1}{n+1})), n \in \mathbb{N}$. Since $g$ is continuous, we have that $(g_n) \to x_0$ for some $x_0 \in X$ (I think?). We can do something similar starting with a sequence in $Y$ that converges to $1$. So if we want to extend $g$ to a homeomorphism $h$ in $H(X)$ we would take $h(x) = g(x)$ for $x \in Y$, $h(0) = x_0$, and $h(1) = x_1$. Does this argument make sense? Intuitively, it seems like it should be correct but I'm worried I'm not being rigorous enough.
Best Answer
For surjectivity note that any continuous injection $\mathbb R\to\mathbb R$ is strictly monotonic. Since there is a strictly monotonic homeomorphism $(0,1)\cong \mathbb R$ (e.g. using tangent), the same is true for $(0,1)$.
Hence, a homeomorphism $g\colon (0,1)\to (0,1)$ must satisfy $$ \lim_{x\to 1} g(x)=1 \quad\text{and}\quad \lim_{x\to 0} g(x) = 0 $$ when it is monotonically increasing, or $$ \lim_{x\to 1} g(x)=0 \quad\text{and}\quad \lim_{x\to 0} g(x) = 1 $$ when it is monotonically decreasing.
In both cases, we obtain an extension of $g$ to a continuous bijection $[0,1]\to[0,1]$. Since the same is true for $g^{-1}$, the extension is a homeomorphism.