Let $U(\mathcal{H})$ be the group of unitary operators on a (complex) Hilbert space $\mathcal{H}$. Then $U(\mathcal{H})$ with the strong topology is a topological group with composition.
In particular, composition is continuous:
Consider a net $(U_i, V_i)_{i \in I} \subseteq U(\mathcal{H}) \times U(\mathcal{H})$ s.t. $(U_i, V_i) \rightarrow (U,V)$. Then since $U(\mathcal{H}) \times U(\mathcal{H})$ has the product topology, this is equivalent to $U_i \rightarrow U$ and $V_i \rightarrow V$, which, in turn, is equivalent to $\forall x \in \mathcal{H}: U_i(x) \rightarrow U(x), V_i(x) \rightarrow V(x)$ by definition of the strong operator topology.
Now, we want to show that $U_i V_i \rightarrow UV$. Again, by the strong operator topology, this is equivalent to point-wise convergence. So let $x \in \mathcal{H}$ be fixed, then we want to show that for any $\varepsilon > 0$ there exists a $i_0 \in I$ s.t. $\forall i \geq i_0: U_i V_i (x) \in B_{\varepsilon}(UVx)$. Indeed
\begin{align}
\Vert (U_i V_i – U V) x \Vert &= \Vert (U_i V_i – U_i V + U_i V – U V) x \Vert \\
&\leq \Vert U_i (V_i – V) x \Vert + \Vert (U_i – U) V x \Vert \\
&\leq \underbrace{\Vert U_i \Vert}_{=1} \cdot \Vert (V_i – V) x \Vert + \Vert (U_i – U) Vx \Vert\\
&= \Vert (V_i – V)x \Vert + \Vert (U_i – U) Vx \Vert\\
\end{align}
Now choose $i_0 \in I$ s.t. $(U_i,V_i) \in \{B \in \mathcal{B}(\mathcal{H}) \vert \Vert (B – U) Vx \Vert < \varepsilon/2 \} \times \{B \in \mathcal{B}(\mathcal{H}) \vert \Vert (B – V) x \Vert < \varepsilon/2 \} =: N(U,\{Vx\},\varepsilon/2) \times N(V, \{x \}, \varepsilon/2)$, which are neighborhoods of $U$ and $V$ respectively (by definition of the strong operator topology).
Then by definition we have $(\Vert (V_i – V)x \Vert + \Vert (U_i – U)x \Vert) \leq \varepsilon$ for $i \geq i_0$.
As far as I know, $GL(\mathcal{H})$ the group of invertible bounded linear operators on $\mathcal{H}$ is not a topological group, precisely because the composition is not continuous. However, I do not see how the above proof fails for $GL(\mathcal{H})$.
The only time one uses the boundedness of $U(\mathcal{H})$ is in $\underbrace{\Vert U_i \Vert}_{=1}$. However, since $U_i$ is pointwise convergent for each $i \in I$, this term would still be bounded above by the uniform boundedness principle.
Question: Where does the proof fail for $GL(\mathcal{H})$ instead of $U(\mathcal{H})$?
Ideally, I am also looking for a counter example.
Best Answer
As Daniel Fischer and Conifold pointed out, the proof fails for unbounded subsets of $GL(\mathcal{H})$, since one needs to bound $\Vert U_i \Vert$.
I thought one could deal with that via uniform boundedness as follows:
Since $U_i x \rightarrow Ux$ we have pointwise convergence and thus pointwise boundedness i.e. $\forall x \in X: \sup_{i \in I} \Vert U_i x \Vert < \infty$. And thus by uniform boundedness: $\sup_{i \in I} \Vert U_i \Vert < \infty$.
However, as opposed to a convergent sequence, a convergent net in a normed space need not be norm bounded. A simple example is given in the answer here.
Remark: note that for separable $\mathcal{H}$, bounded subsets of $GL(\mathcal{H})$ are metrizable. In this case, the argument works, since in order to show continuity it would have been enough to consider sequences.