The first definition corresponds to maximal tori and should be used; the second corresponds to maximal split tori.
The answer by ಠ_ಠ correctly states the definition of Cartan subalgebras for general Lie algebras: It is a subalgebra that is nilpotent and its own normaliser. In the case at hand, it is useful to introduce the following concepts:
Let $\mathfrak{g}$ be a semisimple Lie algebra over any field of characteristic 0. A subalgebra of $\mathfrak{g}$ is called toral if it is abelian and consists of semisimple elements. It is called split toral if it is abelian and consists of diagonalisable elements.
(Of course this is made to resemble tori and split tori in the group setting; I will just write "(split) torus" occasionally.)
Now one has:
Lemma: For $\mathfrak{g}$ as above, a subalgebra is maximal toral iff it is a Cartan subalgebra (= self-normalising & nilpotent).
(This is e.g. exercise 3 to ch. VII $\S$ 2 in Bourbaki's Lie Groups and Lie Algebras.)
As long as one works over algebraically closed fields, one rarely hears of toral and split toral subalgebras, since by algebraic closedness, toral is the same as split toral ("every torus is split"), so that by the lemma:
For a subalgebra of a semisimple Lie algebra over $\mathbb{C}$,
maximal toral = maximal split toral = Cartan subalgebra.
But over other fields, in our case $\mathbb{R}$, we have distinct notions of
- maximal toral subalgebras, and
- maximal split toral subalgebras.
By the lemma, 1. corresponds to the first (Knapp's) definition you give, and the generally accepted notion of Cartan subalgebras.
The second usage that you describe corresponds to 2. That is, what they call a Cartan subalgebra there is actually a maximal split toral subalgebra (in the group setting, it would be a maximal split torus, as opposed to a maximal torus). I have not seen this usage myself and would advise against it, since it does not match the general definition of Cartan subalgebra. Also, it would make the notion not invariant under scalar extension. Calling $\mathfrak{a}_0$ a maximal split torus is much better.
As to your last question, even in split Lie algebras, i.e. when there exists a split maximal torus [Beware the order of words: this is a maximal torus which happens to be split; not, as in notion 2, a maximal one among the split tori], the second usage would be more restrictive, since there can still be maximal tori which are not split.
-- Example: $\mathfrak{g_0} = \mathfrak{sl}_2(\mathbb{R}) = \lbrace \pmatrix{a & b \\
c &-a } : a,b,c \in \mathbb{R}\rbrace$. Then the second usage sees the split Cartan subalgebras (= one-dimensional subspaces) in $\mathfrak{p}_0 = \pmatrix{a & b \\
b &-a }$, but misses the non-split one that constitutes $\mathfrak{k}_0$, $\pmatrix{0 & b \\
-b &0 }$. --
If $\mathfrak{g}_0$ is not split, notion 2 does not even give a subset of notion 1, but they are disjoint: The ones in notion 2 have dimension strictly less than those in notion 1. And $\mathfrak{g}_0$ can still be far from compact. As an example, the following 8-dimensional real Lie algebra is a matrix representation of the quasi-split form of type $A_2$:
$\mathfrak{g}_0 = \lbrace
\begin{pmatrix}
a+bi & c+di & ei\\
f+gi & -2bi & -c+di\\
hi & -f+gi & -a+bi
\end{pmatrix} : a, ..., h \in \mathbb{R} \rbrace$; according to the nomenclature here, one might call this $\mathfrak{su}_{1,2}$.
One has $\mathfrak{k}_0 = \begin{pmatrix}
bi & -f+gi & hi\\
f+gi & -2bi & f+gi\\
hi & -f+gi & bi
\end{pmatrix}$ (i.e. $a=0, c=-f, g=d, h=e$) and
$\mathfrak{p}_0 = \begin{pmatrix}
a & c+di & ei\\
c-di & 0 & -c+di\\
-ei & -c-di & -a
\end{pmatrix}$ (i.e. $b=0, c=f, g =-d, h=-e$).
The maximal split tori $\mathfrak{a}_0$ in this case are the one-dimensional subspaces of $\mathfrak{p}_0$. But one can compute how each of them has a non-trivial centraliser in $\mathfrak{k}_0$ which has to be added to get a maximal torus = Cartan subalgebra in the generally accepted sense; the most obvious choice being
$\mathfrak{a}_0 = \begin{pmatrix}
a & 0 & 0\\
0 & 0 & 0\\
0 & 0 & -a
\end{pmatrix}$ which demands $\mathfrak{t}_0 = \begin{pmatrix}
bi & 0 & 0\\
0 & -2bi & 0\\
0 & 0 & bi
\end{pmatrix}$ as a complement, so that $\mathfrak{a}_0 \oplus \mathfrak{t}_0$ is a maximal torus and becomes the standard maximal split = split maximal torus in the complexification $\mathfrak{g}_{0}^\mathbb{C} \simeq \mathfrak{sl}_3(\mathbb{C})$.
Best Answer
The question is quite muddled and is based on a misconception. In the theory of Lie groups, a "torus" (as a subgroup) need not be homeomorphic to a topological torus.
Let me work with complex semisimple Lie groups $G$, since the discussion is cleaner in this case and this is what the question was about. I will assume that $G$ has finitely many connected components. Then $G$ has complex algebraic structure meaning that $G$ embeds as a Zariski closed subgroup of $GL(N, {\mathbb C})$ for some $N$ (Zariski closed means a subgroup defined via a system of polynomial equations), see here.
A torus in $G$ is a connected abelian complex Lie subgroup $H< G$ whose (almost faithful) linear representation is diagonalizable. Which representation you take, does not matter, you can take the above representation $G\to GL(N, {\mathbb C})$ or you can take the adjoint representation of $G$. (Almost faithful means that the kernel is finite.) A maximal torus (usually denoted $T$ or $A$) is a maximal (with respect to inclusion) subgroup with this property. All maximal tori are conjugate to each other. (Their Lie algebras are Cartan subalgebras, i.e. commutative self-normalizing subalgebras of the Lie algebra of $G$.) Since $H$ is diagonalizable, it is isomorphic to a closed subgroup of $({\mathbb C}^\times)^N$, which implies that $H\cong ({\mathbb C}^\times)^n$ for some $n$.
Conversely, suppose that $H< G$ is a subgroup isomorphic to $({\mathbb C}^\times)^n$. I claim that $H$ is diagonalizable as a subgroup of $GL(N, {\mathbb C})$. Prove this by induction on $N$. The case $N=1$ is clear. Then verify that every connected abelian subgroup of $GL(N, {\mathbb C})$ has an invariant line $L\subset {\mathbb C}^N$ (actually, this is even true for solvable subgroups). How do we know this? Since we are working over the algebraically closed field, each nontrivial element of $H$ has an eigenvector in ${\mathbb C}^N$. By commutativity of $H$, the eigenspace decomposition is $H$-invariant. Now, use the induction hypothesis. In any case, we got an invariant line, so we have another linear representation of $H$ on the vector space $V={\mathbb C}^N/L$. We need to prove that the sequence of $H$-modules splits: $$ 0\to L\to {\mathbb C}^N \to V\to 0. $$ So far our proof used only the fact that $H$ is abelian. Now, I will use that $H$ contains the "compact subtorus" $K_H=(S^1)^n$. The next part of the proof is called the "unitary trick" (more precisely, this is a very special case of the unitary trick). Since the subtorus $K_T$ is compact, it preserves some hermitian inner product on ${\mathbb C}^N$. Thus, $K_T$ preserves $L^\perp$, the orthogonal complement to $L$ in ${\mathbb C}^N$ defined via this inner product. But $H$ is the complexification of $K_T$, hence, the entire $H$ preserves $L^\perp$. (This is a pleasant exercise in complex analysis: If $f$ is a holomorphic function on $H$ vanishing on $K_H$, then $f$ is identically zero. Now, convert this into the statement about invariance of $L^\perp$.) As a $K$-module, $L^\perp$ is isomorphic to $V$ of course. Therefore, the above sequence splits. By the induction hypothesis, $V$ is diagonalizable as an $H$-module (i.e. we have an isomorphism of $H$-modules $V\cong L_1\oplus ... \oplus L_{n-1}$ where each $L_i$ is 1-dimensional) therefore, adding the extra factor $L$ to this decomposition we obtain a diagonalization of $H$. qed
Few more remarks in the non-anlegbraically closed case. A split torus (over some field) in the language of algebraic groups, is a group isomorphic to $(G_m)^n$ for some $n$. Thus, if we treat $H=({\mathbb C}^\times)^n$ as a real algebraic group, then the split torus in $H$ is its set of real points, i.e. the subgroup $({\mathbb R}^\times)^n$. The complementary subgroup $(S^1)^n$ is sometimes called a "compact torus". This part indeed is a torus in the topological sense. But this torus will never be maximal in this situation. One more thing to note: Start with a compact Lie group $K$ (without abelian factors). Then tori in $K$ indeed are all of the form $T=(S^1)^n$. Once you complexify $K$, you obtain a complex semisimple Lie group $G$. The tori in $G$ will be (up to conjugation) of the form $T^{\mathbb C}$, complexifications of tori in $K$.