I would like to understand what is the "outer-automorphism group" $Out$ of $SO(p,q)$ and $O(p,q)$, where $p+q >0$ and $pq \neq 0$. My working definition of $Out$ is as follows:
Let us denote by $Aut(G)$ the automorphism group of a Lie group $G$. I take the inner-automorphism group $Inn(G)$ of $G$ to be all elements $K\in Aut(G)$ for which there exists a $g\in G$ such that $K = Ad_{g}$, namely $K(h) = g h g^{-1}$ for all $h\in G$. $Inn(G)$ is a normal subgroup of $Aut(G)$ and then $Out(G) = Aut(G)/Inn(G)$ is a group which I define to be the outer-morphism group of $G$. I have not been able to find what $Out(G)$ is for $G = SO(p,q), O(p,q)$.
I have noticed that there are many references dealing with the outer-automorphism group of complex Lie algebras, which can be read off from their Dynkin diagram. However, $\mathfrak{so}(p,q)\simeq\mathfrak{o}(p,q)$ is not a complex Lie algebra but a real form. I don't know how the outer-automorphism group of a simple real Lie algebra can be computed in general. In fact, Wikipedia says that the characterization of the outer-automorphism group of a real simple Lie algebra in terms of a short exact sequence involving the full and inner autmorphisms groups (a result classical for complex Lie algebras) was only obtained as recently as in 2010! In any case, I expect the answer to my question to be even more involved since I am not interested in the outer-automorphism group of a real Lie algebra but of the full real Lie group, in my case $SO(p,q)$ and $O(p,q)$. If I am not mistaken, for $q=0$ and $p = even$ we have $O(p,0) = SO(p,0)\rtimes\mathbb{Z}_{2}$, where $\mathbb{Z}_{2}$ is the outer-automorphism group of $SO(p,0)$, so $Out(SO(p,0)) = \mathbb{Z}_{2}$.
Thanks.
Best Answer
Let's first address your comment in response to Igor Rivin's answer: why don't we find this topic addressed in textbooks on Lie groups? Beyond the definite (= compact) case, disconnectedness issues become more complicated and your question is thereby very much informed by the theory of linear algebraic groups $G$ over $\mathbf{R}$. That in turn involves two subtle aspects (see below) that are not easy to express solely in analytic terms and are therefore beyond the level of such books (which usually don't assume familiarity with algebraic geometry at the level needed to work with linear algebraic groups over a field such as $\mathbf{R}$ that is not algebraically closed). And books on linear algebraic groups tend to say little about Lie groups.
The first subtlety is that $G(\mathbf{R})^0$ can be smaller than $G^0(\mathbf{R})$ (i.e., connectedness for the analytic topology may be finer than for the Zariski topology), as we know already for indefinite orthogonal groups, and textbooks on Lie groups tend to focus on the connected case for structural theorems. It is a deep theorem of Elie Cartan that if a linear algebraic group $G$ over $\mathbf{R}$ is Zariski-connected semisimple and simply connected (in the sense of algebraic groups; e.g., ${\rm{SL}}_n$ and ${\rm{Sp}}_{2n}$ but not ${\rm{SO}}_n$) then $G(\mathbf{R})$ is connected, but that lies beyond the level of most textbooks. (Cartan expressed his result in analytic terms via anti-holomorphic involutions of complex semisimple Lie groups, since there was no robust theory of linear algebraic groups at that time.) The group $G(\mathbf{R})$ has finitely many connected components, but that is not elementary (especially if one assumes no knowledge of algebraic geometry), and the theorem on maximal compact subgroups of Lie groups $H$ in case $\pi_0(H)$ is finite but possibly not trivial appears to be treated in only one textbook (Hochschild's "Structure of Lie groups", which however does not address the structure of automorphism groups); e.g., Bourbaki's treatise on Lie groups assumes connectedness for much of its discussion of the structure of compact Lie groups.
The second subtlety is that when the purely analytic operation of "complexification" for Lie groups (developed in Hochschild's book too) is applied to the Lie group of $\mathbf{R}$-points of a (Zariski-connected) semisimple linear algebraic group, it doesn't generally "match" the easier algebro-geometric scalar extension operation on the given linear algebraic group (e.g., the complexification of the Lie group ${\rm{PGL}}_3(\mathbf{R})$ is ${\rm{SL}}_3(\mathbf{C})$, not ${\rm{PGL}}_3(\mathbf{C})$). Here too, things are better-behaved in the "simply connected" case, but that lies beyond the level of introductory textbooks on Lie groups.
Now let us turn to your question. Let $n = p+q$, and assume $n \ge 3$ (so the Lie algebra is semisimple; the cases $n \le 2$ can be analyzed directly anyway). I will only address ${\rm{SO}}(p,q)$ rather than ${\rm{O}}(p, q)$, since it is already enough of a headache to keep track of disconnected effects in the special orthogonal case. To be consistent with your notation, we'll write $\mathbf{O}(p,q) \subset {\rm{GL}}_n$ to denote the linear algebraic group over $\mathbf{R}$ "associated" to the standard quadratic form of signature $(p, q)$ (so its group of $\mathbf{R}$-points is what you have denoted as ${\rm{O}}(p,q)$), and likewise for ${\mathbf{SO}}(p,q)$.
We will show that ${\rm{SO}}(p, q)$ has only inner automorphisms for odd $n$, and only the expected outer automorphism group of order 2 (arising from reflection in any nonzero vector) for even $n$ in both the definite case and the case when $p$ and $q$ are each odd. I will leave it to someone else to figure out (or find a reference on?) the case with $p$ and $q$ both even and positive.
We begin with some preliminary comments concerning the definite (= compact) case for all $n \ge 3$, for which the Lie group ${\rm{SO}}(p,q) = {\rm{SO}}(n)$ is connected. The crucial (non-trivial) fact is that the theory of connected compact Lie groups is completely "algebraic'', and in particular if $G$ and $H$ are two connected semisimple $\mathbf{R}$-groups for which $G(\mathbf{R})$ and $H(\mathbf{R})$ are compact then every Lie group homomorphism $G(\mathbf{R}) \rightarrow H(\mathbf{R})$ arising from a (unique) algebraic homomorphism $G \rightarrow H$. In particular, the automorphism groups of $G$ and $G(\mathbf{R})$ coincide, so the automorphism group of ${\rm{SO}}(n)$ coincides with that of $\mathbf{SO}(n)$.
Note that any linear automorphism preserving a non-degenerate quadratic form up to a nonzero scaling factor preserves its orthogonal and special orthogonal group. It is a general fact (due to Dieudonne over general fields away from characteristic 2) that if $(V, Q)$ is a non-degenerate quadratic space of dimension $n \ge 3$ over any field $k$ and if ${\mathbf{GO}}(Q)$ denotes the linear algebraic $k$-group of conformal automorphisms then the action of the algebraic group ${\mathbf{PGO}}(Q) = {\mathbf{GO}}(Q)/{\rm{GL}}_1$ on ${\mathbf{SO}}(Q)$ through conjugation gives exactly the automorphisms as an algebraic group. More specifically, $${\mathbf{PGO}}(Q)(k) = {\rm{Aut}}_k({\mathbf{SO}}(Q)).$$ This is proved using a lot of the structure theory of connected semisimple groups over an extension field that splits the quadratic form, so it is hard to "see'' this fact working directly over the given ground field $k$ (such as $k = \mathbf{R}$); that is one of the great merits of the algebraic theory (allowing us to prove results over a field by making calculations with a geometric object over an extension field, and using techniques such as Galois theory to come back to where we began).
Inside the automorphism group of the Lie group ${\rm{SO}}(p,q)$, we have built the subgroup ${\rm{PGO}}(p,q) := {\mathbf{PGO}}(p,q)(\mathbf{R})$ of "algebraic'' automorphisms (and it gives all automorphisms when $p$ or $q$ vanish). This subgroup is $${\mathbf{GO}}(p,q)(\mathbf{R})/\mathbf{R}^{\times} = {\rm{GO}}(p,q)/\mathbf{R}^{\times}.$$ To analyze the group ${\rm{GO}}(p,q)$ of conformal automorphisms of the quadratic space, there are two possibilities: if $p \ne q$ (such as whenever $p$ or $q$ vanish) then any such automorphism must involve a positive conformal scaling factor due to the need to preserve the signature, and if $p=q$ (the "split'' case: orthogonal sum of $p$ hyperbolic planes) then signature-preservation imposes no condition and we see (upon choosing a decomposition as an orthogonal sum of $p$ hyperbolic planes) that there is an evident involution $\tau$ of the vector space whose effect is to negative the quadratic form. Thus, if $p \ne q$ then ${\rm{GO}}(p,q) = \mathbf{R}^{\times} \cdot {\rm{O}}(p,q)$ whereas ${\rm{GO}}(p,p) = \langle \tau \rangle \ltimes (\mathbf{R}^{\times} \cdot {\rm{O}}(p,p))$. Hence, ${\rm{PGO}}(p,q) = {\rm{O}}(p,q)/\langle -1 \rangle$ if $p \ne q$ and ${\rm{PGO}}(p,p) = \langle \tau \rangle \ltimes ({\rm{O}}(p,p)/\langle -1 \rangle)$ for an explicit involution $\tau$ as above.
We summarize the conclusions for outer automorphisms of the Lie group ${\rm{SO}}(p, q)$ arising from the algebraic theory. If $n$ is odd (so $p \ne q$) then ${\rm{O}}(p,q) = \langle -1 \rangle \times {\rm{SO}}(p,q)$ and so the algebraic automorphisms are inner (as is very well-known in the algebraic theory). Suppose $n$ is even, so $-1 \in {\rm{SO}}(p, q)$. If $p \ne q$ (with the same parity) then the group of algebraic automorphisms contributes a subgroup of order 2 to the outer automorphism group (arising from any reflection in a non-isotropic vector, for example). Finally, the contribution of algebraic automorphisms to the outer automorphism group of ${\rm{SO}}(p,p)$ has order 4 (generated by two elements of order 2: an involution $\tau$ as above and a reflection in a non-isotropic vector). This settles the definite case as promised (i.e., all automorphisms inner for odd $n$ and outer automorphism group of order 2 via a reflection for even $n$) since in such cases we know that all automorphisms are algebraic.
Now we may and do assume $p, q > 0$. Does ${\rm{SO}}(p, q)$ have any non-algebraic automorphisms? We will show that if $n \ge 3$ is odd (i.e., $p$ and $q$ have opposite parity) or if $p$ and $q$ are both odd then there are no non-algebraic automorphisms (so we would be done).
First, let's compute $\pi_0({\rm{SO}}(p,q))$ for any $n \ge 3$. By the spectral theorem, the maximal compact subgroups of ${\rm{O}}(p,q)$ are the conjugates of the evident subgroup ${\rm{O}}(q) \times {\rm{O}}(q)$ with 4 connected components, and one deduces in a similar way that the maximal compact subgroups of ${\rm{SO}}(p, q)$ are the conjugates of the evident subgroup $$\{(g,g') \in {\rm{O}}(p) \times {\rm{O}}(q)\,|\, \det(g) = \det(g')\}$$ with 2 connected components. For any Lie group $\mathscr{H}$ with finite component group (such as the group $G(\mathbf{R})$ for any linear algebraic group $G$ over $\mathbf{R}$), the maximal compact subgroups $K$ constitute a single conjugacy class (with every compact subgroup contained in one) and as a smooth manifold $\mathscr{H}$ is a direct product of such a subgroup against a Euclidean space (see Chapter XV, Theorem 3.1 of Hochschild's book "Structure of Lie groups'' for a proof). In particular, $\pi_0(\mathscr{H}) = \pi_0(K)$, so ${\rm{SO}}(p, q)$ has exactly 2 connected components for any $p, q > 0$.
Now assume $n$ is odd, and swap $p$ and $q$ if necessary (as we may) so that $p$ is odd and $q>0$ is even. For any $g \in {\rm{O}}(q) - {\rm{SO}}(q)$, the element $(-1, g) \in {\rm{SO}}(p, q)$ lies in the unique non-identity component. Since $n \ge 3$ is odd, so ${\rm{SO}}(p, q)^0$ is the quotient of the connected (!) Lie group ${\rm{Spin}}(p, q)$ modulo its order-2 center, the algebraic theory in characteristic 0 gives $${\rm{Aut}}({\mathfrak{so}}(p,q)) = {\rm{Aut}}({\rm{Spin}}(p, q)) = {\rm{SO}}(p, q).$$ Thus, to find nontrivial elements of the outer automorphism group of the disconnected Lie group ${\rm{SO}}(p, q)$ we can focus attention on automorphisms $f$ of ${\rm{SO}}(p, q)$ that induce the identity on ${\rm{SO}}(p, q)^0$.
We have arranged that $p$ is odd and $q>0$ is even (so $q \ge 2$). The elements $$(-1, g) \in {\rm{SO}}(p, q) \cap ({\rm{O}}(p) \times {\rm{O}}(q))$$ (intersection inside ${\rm{O}}(p, q)$, so $g \in {\rm{O}}(q) - {\rm{SO}}(q)$) have an intrinsic characterization in terms of the Lie group ${\rm{SO}}(p, q)$ and its evident subgroups ${\rm{SO}}(p)$ and ${\rm{SO}}(q)$: these are the elements outside ${\rm{SO}}(p, q)^0$ that centralize ${\rm{SO}}(p)$ and normalize ${\rm{SO}}(q)$. (To prove this, consider the standard representation of ${\rm{SO}}(p) \times {\rm{SO}}(q)$ on $\mathbf{R}^{p+q} = \mathbf{R}^n$, especially the isotypic subspaces for the action of ${\rm{SO}}(q)$ with $q \ge 2$.) Hence, for every $g \in {\rm{O}}(q) - {\rm{SO}}(q)$ we have $f(-1,g) = (-1, F(g))$ for a diffeomorphism $F$ of the connected manifold ${\rm{O}}(q) - {\rm{SO}}(q)$.
Since $f$ acts as the identity on ${\rm{SO}}(q)$, it follows that the elements $g, F(g) \in {\rm{O}}(q) - {\rm{SO}}(q)$ have the same conjugation action on ${\rm{SO}}(q)$. But ${\rm{PGO}}(q) \subset {\rm{Aut}}({\rm{SO}}(q))$, so $F(g)g^{-1} \in \mathbf{R}^{\times}$ inside ${\rm{GL}}_q(\mathbf{R})$ with $q>0$ even. Taking determinants, this forces $F(g) = \pm g$ for a sign that may depend on $g$. But $F$ is continuous on the connected space ${\rm{O}}(q) - {\rm{SO}}(q)$, so the sign is actually independent of $g$. The case $F(g) = g$ corresponds to the identity automorphism of ${\rm{SO}}(q)$, so for the study of non-algebraic contributions to the outer automorphism group of ${\rm{SO}}(p, q)$ (with $p$ odd and $q > 0$ even) we are reduced to showing that the case $F(g) = -g$ cannot occur.
We are seeking to rule out the existence of an automorphism $f$ of ${\rm{SO}}(p, q)$ that is the identity on ${\rm{SO}}(p, q)^0$ and satisfies $(-1, g) \mapsto (-1, -g)$ for $g \in {\rm{O}}(q) - {\rm{SO}}(q)$. For this to be a homomorphism, it is necessary (and sufficient) that the conjugation actions of $(-1, g)$ and $(-1, -g)$ on ${\rm{SO}}(p, q)^0$ coincide for all $g \in {\rm{O}}(q) - {\rm{SO}}(q)$. In other words, this requires that the element $(1, -1) \in {\rm{SO}}(p, q)$ centralizes ${\rm{SO}}(p, q)^0$. But the algebraic group ${\mathbf{SO}}(p, q)$ is connected (for the Zariski topology) with trivial center and the same Lie algebra as ${\rm{SO}}(p, q)^0$, so by consideration of the compatible algebraic and analytic adjoint representations we see that $(1, -1)$ cannot centralize ${\rm{SO}}(p, q)^0$. Thus, no non-algebraic automorphism of ${\rm{SO}}(p, q)$ exists in the indefinite case when $n \ge 3$ is odd.
Finally, suppose $p$ and $q$ are both odd, so ${\rm{SO}}(p,q)^0$ does not contain the element $-1 \in {\rm{SO}}(p,q)$ that generates the center of ${\rm{SO}}(p,q)$ (and even the center of ${\rm{O}}(p,q)$). Thus, we have ${\rm{SO}}(p,q) = {\rm{SO}}(p,q)^0 \times \langle -1 \rangle$ with ${\rm{SO}}(p,q)^0$ having trivial center. Any (analytic) automorphism of ${\rm{SO}}(p,q)$ clearly acts trivially on the order-2 center $\langle -1 \rangle$ and must preserve the identity component too, so such an automorphism is determined by its effect on the identity component. It suffices to show that every analytic automorphism $f$ of ${\rm{SO}}(p,q)^0$ arises from an algebraic automorphism of ${\rm{SO}}(p,q)$, as then all automorphisms of ${\rm{SO}}(p,q)$ would be algebraic (so the determination of the outer analytic automorphism group for $p, q$ odd follows as for the definite case with even $n \ge 4$).
By the theory of connected semisimple algebraic groups in characteristic 0, for any $p, q \ge 0$ with $p+q \ge 3$ every analytic automorphism of the connected (!) group ${\rm{Spin}}(p,q)$ is algebraic. Thus, it suffices to show that any automorphism $f$ of ${\rm{SO}}(p,q)^0$ lifts to an automorphism of the degree-2 cover $\pi:{\rm{Spin}}(p,q) \rightarrow {\rm{SO}}(p,q)^0$. (Beware that this degree-2 cover is not the universal cover if $p, q \ge 2$, as ${\rm{SO}}(p,q)^0$ has maximal compact subgroup ${\rm{SO}}(p) \times {\rm{SO}}(q)$ with fundamental group of order 4.) The Lie algebra automorphism ${\rm{Lie}}(f)$ of ${\mathfrak{so}}(p,q) = {\mathfrak{spin}}(p,q)$ arises from a unique algebraic automorphism of the group ${\mathbf{Spin}}(p,q)$ since this latter group is simply connected in the sense of algebraic groups. The induced automorphism of the group ${\rm{Spin}}(p,q)$ of $\mathbf{R}$-points does the job, since its compatibility with $f$ via $\pi$ can be checked on Lie algebras (as we are working with connected Lie groups).
This final argument also shows that the remaining problem for even $p, q \ge 2$ is to determine if any automorphism of ${\rm{SO}}(p,q)$ that is the identity map on ${\rm{SO}}(p,q)^0$ is itself the identity map. (If affirmative for such $p, q$ then the outer automorphism group of ${\rm{SO}}(p,q)$ is of order 2, and if negative then the outer automorphism group is bigger.)