Let's first address your comment in response to Igor Rivin's answer: why don't we find this topic addressed in textbooks on Lie groups? Beyond the definite (= compact) case, disconnectedness issues become more complicated and your question is thereby very much informed by the theory of linear algebraic groups $G$ over $\mathbf{R}$. That in turn involves two subtle aspects (see below) that are not easy to express solely in analytic terms and are therefore beyond the level of such books (which usually don't assume familiarity with algebraic geometry at the level needed to work with linear algebraic groups over a field such as $\mathbf{R}$ that is not algebraically closed). And books on linear algebraic groups tend to say little about Lie groups.
The first subtlety is that $G(\mathbf{R})^0$ can be smaller than $G^0(\mathbf{R})$ (i.e., connectedness for the analytic topology may be finer than for the Zariski topology), as we know already for indefinite orthogonal groups, and textbooks on Lie groups tend to focus on the connected case for structural theorems. It is a deep theorem of Elie Cartan that if a linear algebraic group $G$ over $\mathbf{R}$ is Zariski-connected semisimple and simply connected (in the sense of algebraic groups; e.g., ${\rm{SL}}_n$ and ${\rm{Sp}}_{2n}$ but not ${\rm{SO}}_n$) then $G(\mathbf{R})$ is connected, but that lies beyond the level of most textbooks. (Cartan expressed his result in analytic terms via anti-holomorphic involutions of complex semisimple Lie groups, since there was no robust theory of linear algebraic groups at that time.) The group $G(\mathbf{R})$ has finitely many connected components, but that is not elementary (especially if one assumes no knowledge of algebraic geometry), and the theorem on maximal compact subgroups of Lie groups $H$ in case $\pi_0(H)$ is finite but possibly not trivial appears to be treated in only one textbook (Hochschild's "Structure of Lie groups", which however does not address the structure of automorphism groups); e.g.,
Bourbaki's treatise on Lie groups assumes connectedness for much of its discussion of
the structure of compact Lie groups.
The second subtlety is that when the purely analytic operation of "complexification" for Lie groups (developed in Hochschild's book too) is applied to the Lie group of $\mathbf{R}$-points of a (Zariski-connected) semisimple linear algebraic group, it doesn't generally "match" the easier algebro-geometric scalar extension operation on the given linear algebraic group (e.g., the complexification of the Lie group ${\rm{PGL}}_3(\mathbf{R})$ is ${\rm{SL}}_3(\mathbf{C})$, not ${\rm{PGL}}_3(\mathbf{C})$). Here too, things are better-behaved in the "simply connected" case, but that lies beyond the level of introductory textbooks on Lie groups.
Now let us turn to your question. Let $n = p+q$, and assume $n \ge 3$ (so the Lie algebra is semisimple;
the cases $n \le 2$ can be analyzed directly anyway). I will only address ${\rm{SO}}(p,q)$ rather than ${\rm{O}}(p, q)$, since it is already enough of a headache to keep track of disconnected effects in the special orthogonal case. To be consistent with your notation, we'll write $\mathbf{O}(p,q) \subset {\rm{GL}}_n$ to denote the linear algebraic group over $\mathbf{R}$ "associated" to the standard quadratic form of signature $(p, q)$ (so its group of $\mathbf{R}$-points is
what you have denoted as ${\rm{O}}(p,q)$), and likewise for ${\mathbf{SO}}(p,q)$.
We will show that ${\rm{SO}}(p, q)$ has only inner automorphisms for odd $n$, and only the expected
outer automorphism group of order 2 (arising from reflection in any nonzero vector) for even $n$ in both the definite case and the case when $p$ and $q$ are each odd.
I will leave it to someone else to figure out (or find a reference on?) the case with $p$ and $q$ both even and positive.
We begin with some preliminary comments concerning
the definite (= compact) case for all $n \ge 3$, for which the Lie group ${\rm{SO}}(p,q) = {\rm{SO}}(n)$ is connected.
The crucial (non-trivial) fact is
that the theory of connected compact Lie groups is completely "algebraic'', and in particular
if $G$ and $H$ are two connected semisimple $\mathbf{R}$-groups for which
$G(\mathbf{R})$ and $H(\mathbf{R})$ are compact then every Lie group homomorphism
$G(\mathbf{R}) \rightarrow H(\mathbf{R})$ arising from a (unique) algebraic homomorphism $G \rightarrow H$.
In particular, the automorphism groups of $G$ and $G(\mathbf{R})$ coincide, so
the automorphism group of ${\rm{SO}}(n)$ coincides with that of $\mathbf{SO}(n)$.
Note that any linear automorphism preserving a non-degenerate quadratic form up to a nonzero scaling factor preserves its orthogonal and special orthogonal group. It is a general fact (due to Dieudonne over general fields away from characteristic 2) that if $(V, Q)$ is a non-degenerate quadratic space of dimension $n \ge 3$ over any field $k$ and if ${\mathbf{GO}}(Q)$ denotes the
linear algebraic $k$-group of
conformal automorphisms then the action of the algebraic group
${\mathbf{PGO}}(Q) = {\mathbf{GO}}(Q)/{\rm{GL}}_1$ on ${\mathbf{SO}}(Q)$ through conjugation gives exactly the automorphisms as an algebraic group. More specifically, $${\mathbf{PGO}}(Q)(k) = {\rm{Aut}}_k({\mathbf{SO}}(Q)).$$
This is proved using a lot of the structure theory of connected semisimple groups
over an extension field that splits the quadratic form, so it is hard to "see'' this fact
working directly over the given ground field $k$ (such as $k = \mathbf{R}$); that
is one of the great merits of the algebraic theory (allowing us to prove results over a field
by making calculations with a geometric object over an extension field, and using techniques
such as Galois theory to come back to where we began).
Inside the automorphism group of the Lie group ${\rm{SO}}(p,q)$, we have built the subgroup
${\rm{PGO}}(p,q) := {\mathbf{PGO}}(p,q)(\mathbf{R})$ of "algebraic'' automorphisms (and it gives all
automorphisms when $p$ or $q$ vanish). This subgroup is
$${\mathbf{GO}}(p,q)(\mathbf{R})/\mathbf{R}^{\times} = {\rm{GO}}(p,q)/\mathbf{R}^{\times}.$$
To analyze the group ${\rm{GO}}(p,q)$ of conformal automorphisms of the quadratic space, there are two possibilities: if $p \ne q$ (such as whenever $p$ or $q$ vanish) then any such automorphism must involve a positive conformal scaling factor due to the need to preserve the signature, and if $p=q$ (the "split'' case: orthogonal sum of $p$ hyperbolic planes) then signature-preservation imposes no condition and we see (upon choosing a decomposition as an orthogonal sum of $p$ hyperbolic planes) that there is an evident involution $\tau$ of the vector space whose effect is to negative the quadratic form. Thus, if $p \ne q$ then ${\rm{GO}}(p,q) = \mathbf{R}^{\times} \cdot {\rm{O}}(p,q)$ whereas ${\rm{GO}}(p,p) = \langle \tau \rangle \ltimes (\mathbf{R}^{\times} \cdot {\rm{O}}(p,p))$. Hence, ${\rm{PGO}}(p,q) = {\rm{O}}(p,q)/\langle -1 \rangle$ if $p \ne q$ and ${\rm{PGO}}(p,p) = \langle \tau \rangle \ltimes ({\rm{O}}(p,p)/\langle -1 \rangle)$ for an explicit involution $\tau$ as above.
We summarize the conclusions for outer automorphisms of the Lie group ${\rm{SO}}(p, q)$ arising from the algebraic theory.
If $n$ is odd (so $p \ne q$) then ${\rm{O}}(p,q) = \langle -1 \rangle \times {\rm{SO}}(p,q)$ and so the algebraic automorphisms are inner (as is very well-known in the algebraic theory). Suppose $n$ is even, so
$-1 \in {\rm{SO}}(p, q)$. If $p \ne q$ (with the same parity) then the group of algebraic automorphisms contributes a subgroup of order 2 to the outer automorphism group (arising from any reflection in a non-isotropic vector, for example). Finally, the contribution of algebraic automorphisms to the outer automorphism group of ${\rm{SO}}(p,p)$ has order 4 (generated by two elements of order 2: an involution $\tau$ as above and a reflection in a non-isotropic vector).
This settles the definite case as promised (i.e., all automorphisms inner for odd $n$ and outer automorphism group
of order 2 via a reflection for even $n$) since in such cases we know that all automorphisms are algebraic.
Now we may and do assume $p, q > 0$. Does ${\rm{SO}}(p, q)$ have any non-algebraic automorphisms?
We will show that if $n \ge 3$ is odd (i.e., $p$ and $q$ have opposite parity) or if $p$ and $q$ are both odd
then there are no non-algebraic automorphisms (so we would be done).
First, let's compute $\pi_0({\rm{SO}}(p,q))$ for any $n \ge 3$. By the spectral theorem,
the maximal compact subgroups of ${\rm{O}}(p,q)$ are the conjugates of the evident subgroup ${\rm{O}}(q) \times {\rm{O}}(q)$
with 4 connected components,
and one deduces in a similar way
that the maximal compact subgroups of ${\rm{SO}}(p, q)$ are the conjugates of the evident subgroup
$$\{(g,g') \in {\rm{O}}(p) \times {\rm{O}}(q)\,|\, \det(g) = \det(g')\}$$
with 2 connected components. For any Lie group $\mathscr{H}$
with finite component group (such as the group $G(\mathbf{R})$
for any linear algebraic group $G$ over $\mathbf{R}$), the maximal compact subgroups $K$
constitute a single conjugacy class (with every compact subgroup contained in one) and as a smooth manifold
$\mathscr{H}$ is a direct product of such a subgroup against a Euclidean space (see Chapter XV, Theorem 3.1 of
Hochschild's book "Structure of Lie groups'' for a proof). In particular, $\pi_0(\mathscr{H}) = \pi_0(K)$,
so ${\rm{SO}}(p, q)$ has exactly 2 connected components for any $p, q > 0$.
Now assume $n$ is odd, and swap $p$ and $q$ if necessary (as we may) so that $p$ is odd and $q>0$ is even.
For any $g \in {\rm{O}}(q) - {\rm{SO}}(q)$, the element $(-1, g) \in {\rm{SO}}(p, q)$ lies
in the unique non-identity component. Since $n \ge 3$ is odd, so ${\rm{SO}}(p, q)^0$ is the quotient
of the connected (!) Lie group ${\rm{Spin}}(p, q)$ modulo its order-2 center, the algebraic theory in characteristic 0 gives
$${\rm{Aut}}({\mathfrak{so}}(p,q)) = {\rm{Aut}}({\rm{Spin}}(p, q)) = {\rm{SO}}(p, q).$$
Thus, to find nontrivial elements of the outer automorphism group of the disconnected Lie group ${\rm{SO}}(p, q)$ we can focus
attention on automorphisms $f$ of ${\rm{SO}}(p, q)$ that induce the identity on ${\rm{SO}}(p, q)^0$.
We have arranged that $p$ is odd and $q>0$ is even (so $q \ge 2$).
The elements $$(-1, g) \in {\rm{SO}}(p, q) \cap ({\rm{O}}(p) \times {\rm{O}}(q))$$ (intersection inside ${\rm{O}}(p, q)$, so
$g \in {\rm{O}}(q) - {\rm{SO}}(q)$) have an intrinsic
characterization in terms of the Lie group ${\rm{SO}}(p, q)$
and its evident subgroups ${\rm{SO}}(p)$ and ${\rm{SO}}(q)$: these are the elements outside ${\rm{SO}}(p, q)^0$ that centralize
${\rm{SO}}(p)$ and normalize ${\rm{SO}}(q)$. (To prove this, consider the standard representation of
${\rm{SO}}(p) \times {\rm{SO}}(q)$ on $\mathbf{R}^{p+q} = \mathbf{R}^n$, especially the
isotypic subspaces for the action of ${\rm{SO}}(q)$ with $q \ge 2$.) Hence, for every $g \in {\rm{O}}(q) - {\rm{SO}}(q)$
we have $f(-1,g) = (-1, F(g))$ for a diffeomorphism $F$ of the connected manifold ${\rm{O}}(q) - {\rm{SO}}(q)$.
Since $f$ acts as the identity on ${\rm{SO}}(q)$, it follows that the elements $g, F(g) \in {\rm{O}}(q) - {\rm{SO}}(q)$
have the same conjugation action on ${\rm{SO}}(q)$. But ${\rm{PGO}}(q) \subset {\rm{Aut}}({\rm{SO}}(q))$,
so $F(g)g^{-1} \in \mathbf{R}^{\times}$ inside ${\rm{GL}}_q(\mathbf{R})$
with $q>0$ even. Taking determinants, this forces $F(g) = \pm g$ for a sign that may depend on $g$.
But $F$ is continuous on the connected space ${\rm{O}}(q) - {\rm{SO}}(q)$, so the sign is actually
independent of $g$. The case $F(g) = g$ corresponds to the identity automorphism of ${\rm{SO}}(q)$,
so for the study of non-algebraic contributions to the outer automorphism group of
${\rm{SO}}(p, q)$ (with $p$ odd and $q > 0$ even) we are reduced to showing that
the case $F(g) = -g$ cannot occur.
We are seeking to rule out the existence of an automorphism $f$ of ${\rm{SO}}(p, q)$ that is the identity on
${\rm{SO}}(p, q)^0$ and satisfies $(-1, g) \mapsto (-1, -g)$ for $g \in {\rm{O}}(q) - {\rm{SO}}(q)$.
For this to be a homomorphism, it is necessary (and sufficient) that the conjugation
actions of $(-1, g)$ and $(-1, -g)$ on ${\rm{SO}}(p, q)^0$ coincide for all $g \in {\rm{O}}(q) - {\rm{SO}}(q)$.
In other words, this requires that the element $(1, -1) \in {\rm{SO}}(p, q)$ centralizes
${\rm{SO}}(p, q)^0$. But the algebraic group ${\mathbf{SO}}(p, q)$ is connected (for the Zariski topology)
with trivial center and the same Lie algebra as ${\rm{SO}}(p, q)^0$, so by consideration
of the compatible algebraic and analytic adjoint representations we see that $(1, -1)$ cannot
centralize ${\rm{SO}}(p, q)^0$. Thus, no non-algebraic automorphism of ${\rm{SO}}(p, q)$ exists
in the indefinite case when $n \ge 3$ is odd.
Finally, suppose $p$ and $q$ are both odd, so ${\rm{SO}}(p,q)^0$ does not contain the element $-1 \in {\rm{SO}}(p,q)$ that generates the center of ${\rm{SO}}(p,q)$ (and even the center of ${\rm{O}}(p,q)$). Thus, we have ${\rm{SO}}(p,q) = {\rm{SO}}(p,q)^0 \times \langle -1 \rangle$ with ${\rm{SO}}(p,q)^0$ having trivial center. Any (analytic) automorphism of ${\rm{SO}}(p,q)$ clearly acts trivially on the order-2 center $\langle -1 \rangle$ and must preserve the identity component too, so such an automorphism is determined by its effect on the identity component. It suffices to show that every analytic automorphism $f$ of ${\rm{SO}}(p,q)^0$ arises from an algebraic automorphism of ${\rm{SO}}(p,q)$, as then all automorphisms of ${\rm{SO}}(p,q)$ would be algebraic (so the determination of the outer analytic automorphism group for $p, q$ odd follows as for the definite case with even $n \ge 4$).
By the theory of connected semisimple algebraic groups in characteristic 0, for any $p, q \ge 0$ with $p+q \ge 3$ every analytic automorphism of the connected (!) group ${\rm{Spin}}(p,q)$ is algebraic. Thus, it suffices to show that any automorphism $f$ of ${\rm{SO}}(p,q)^0$ lifts to an automorphism of the degree-2 cover $\pi:{\rm{Spin}}(p,q) \rightarrow {\rm{SO}}(p,q)^0$. (Beware that this degree-2 cover is not the universal cover if $p, q \ge 2$, as ${\rm{SO}}(p,q)^0$ has maximal compact subgroup ${\rm{SO}}(p) \times {\rm{SO}}(q)$ with fundamental group of order 4.) The Lie algebra automorphism ${\rm{Lie}}(f)$ of ${\mathfrak{so}}(p,q) = {\mathfrak{spin}}(p,q)$ arises from a unique algebraic automorphism of the group ${\mathbf{Spin}}(p,q)$ since this latter group is simply connected in the sense of algebraic groups. The induced automorphism of the group ${\rm{Spin}}(p,q)$ of $\mathbf{R}$-points does the job, since its compatibility with $f$ via $\pi$ can be checked on Lie algebras (as we are working with connected Lie groups).
This final argument also shows that the remaining problem for even $p, q \ge 2$ is to determine if any automorphism of ${\rm{SO}}(p,q)$ that is the identity map on ${\rm{SO}}(p,q)^0$ is itself the identity map. (If affirmative for such $p, q$ then the outer automorphism group of ${\rm{SO}}(p,q)$ is of order 2, and if negative then the outer automorphism group is bigger.)
Best Answer
Here's my favorite way to answer your question. Hopefully the answer to Robert Bryant's question is "yes".
Let $A$ be the ring of octonions (the "nonsplit" octonions over ${\mathbb R}$); it comes with an involution $\alpha \mapsto \bar \alpha$, from which there is a trace $Tr(\alpha) = \alpha + \bar \alpha$ and a norm $N(\alpha) = \alpha \cdot \bar \alpha$. From this, we get a trilinear form $T: A \otimes A \otimes A \rightarrow {\mathbb R}$ given by $$T(\alpha, \beta, \gamma) = Tr( (\alpha \beta) \gamma) = Tr(\alpha (\beta \gamma)).$$ (Multiplication is nonassociative, but the traces work out to the same result.)
The group $Spin(8)$ can be constructed as the group of triples $(g_1, g_2, g_3) \in SO(A,N)^3$ of "rotation matrices" with respect to the norm quadratic form, such that for all $(\alpha, \beta, \gamma) \in A^3$, $$T(g_1 \alpha, g_2 \beta, g_3 \gamma) = T(\alpha, \beta, \gamma).$$
The full group of outer automorphisms is now almost clear -- cyclic permutations of $(g_1, g_2, g_3)$ give automorphisms of $Spin(8)$ defined above since $T(\alpha, \beta, \gamma) = T(\beta, \gamma, \alpha)$.
Edit below to reflect comments and correspondence with Daemi, and the comment of Bryant
The full $S_3$ action on $Spin(8)$ is obtained from cyclic permutations and the following slightly subtle action of transpositions. Let $C$ denote the main involution of $A$ (the one for which $Tr(\alpha) = \alpha + C(\alpha)$). For any $g \in SO(A,N)$, define $\bar g = C \circ g \circ C$; then $\bar g \in SO(A,N)$ as well.
The action of a transposition on $Spin(8)$ follows: for the transposition $(12)(3)$, the associated outer automorphism of $Spin(8)$ sends $(g_1, g_2, g_3)$ to $(\bar g_2, \bar g_1, g_3)$. The other transpositions act in the analogous ways.
The Jordan algebra is the exceptional Jordan algebra of 3x3 Hermitian matrices with octonion entries: $$J = \left\lbrace \left( \begin{array}{ccc} a & \alpha & \bar \beta \cr \bar \alpha & b & \gamma \cr \beta & \bar \gamma & c \\ \end{array} \right) : \alpha, \beta, \gamma \in A, a,b,c \in {\mathbb R} \right\rbrace.$$
The group $Spin(8)$ acts on the triple of octonions $(\alpha, \beta, \gamma)$ via the natural representation from above. It acts trivially on the real numbers $a,b,c$, and this gives an action of $Spin(8)$ on the exceptional Jordan algebra. The outer automorphism group $S_3$ acts by permuting $(a,b,c)$ and $(\alpha, \beta, \gamma)$ simultaneously. Together, these give an action of $S_3 \ltimes Spin(8)$ on the exceptional Jordan algebra.
Reference update:
The material above can be found in my paper on $D_4$ modular forms, Amer. J. of Math. 128 (2006), 849-898.
The construction of $Spin(8)$ (over ${\mathbb Z}$) follows from Proposition 4.8 of M.-A. Knus, R. Parimala, and R. Sridharan, "On Compositions and Triality," J. reine angew. Math., 457:45–70, 1994.