The first definition corresponds to maximal tori and should be used; the second corresponds to maximal split tori.
The answer by ಠ_ಠ correctly states the definition of Cartan subalgebras for general Lie algebras: It is a subalgebra that is nilpotent and its own normaliser. In the case at hand, it is useful to introduce the following concepts:
Let $\mathfrak{g}$ be a semisimple Lie algebra over any field of characteristic 0. A subalgebra of $\mathfrak{g}$ is called toral if it is abelian and consists of semisimple elements. It is called split toral if it is abelian and consists of diagonalisable elements.
(Of course this is made to resemble tori and split tori in the group setting; I will just write "(split) torus" occasionally.)
Now one has:
Lemma: For $\mathfrak{g}$ as above, a subalgebra is maximal toral iff it is a Cartan subalgebra (= self-normalising & nilpotent).
(This is e.g. exercise 3 to ch. VII $\S$ 2 in Bourbaki's Lie Groups and Lie Algebras.)
As long as one works over algebraically closed fields, one rarely hears of toral and split toral subalgebras, since by algebraic closedness, toral is the same as split toral ("every torus is split"), so that by the lemma:
For a subalgebra of a semisimple Lie algebra over $\mathbb{C}$,
maximal toral = maximal split toral = Cartan subalgebra.
But over other fields, in our case $\mathbb{R}$, we have distinct notions of
- maximal toral subalgebras, and
- maximal split toral subalgebras.
By the lemma, 1. corresponds to the first (Knapp's) definition you give, and the generally accepted notion of Cartan subalgebras.
The second usage that you describe corresponds to 2. That is, what they call a Cartan subalgebra there is actually a maximal split toral subalgebra (in the group setting, it would be a maximal split torus, as opposed to a maximal torus). I have not seen this usage myself and would advise against it, since it does not match the general definition of Cartan subalgebra. Also, it would make the notion not invariant under scalar extension. Calling $\mathfrak{a}_0$ a maximal split torus is much better.
As to your last question, even in split Lie algebras, i.e. when there exists a split maximal torus [Beware the order of words: this is a maximal torus which happens to be split; not, as in notion 2, a maximal one among the split tori], the second usage would be more restrictive, since there can still be maximal tori which are not split.
-- Example: $\mathfrak{g_0} = \mathfrak{sl}_2(\mathbb{R}) = \lbrace \pmatrix{a & b \\
c &-a } : a,b,c \in \mathbb{R}\rbrace$. Then the second usage sees the split Cartan subalgebras (= one-dimensional subspaces) in $\mathfrak{p}_0 = \pmatrix{a & b \\
b &-a }$, but misses the non-split one that constitutes $\mathfrak{k}_0$, $\pmatrix{0 & b \\
-b &0 }$. --
If $\mathfrak{g}_0$ is not split, notion 2 does not even give a subset of notion 1, but they are disjoint: The ones in notion 2 have dimension strictly less than those in notion 1. And $\mathfrak{g}_0$ can still be far from compact. As an example, the following 8-dimensional real Lie algebra is a matrix representation of the quasi-split form of type $A_2$:
$\mathfrak{g}_0 = \lbrace
\begin{pmatrix}
a+bi & c+di & ei\\
f+gi & -2bi & -c+di\\
hi & -f+gi & -a+bi
\end{pmatrix} : a, ..., h \in \mathbb{R} \rbrace$; according to the nomenclature here, one might call this $\mathfrak{su}_{1,2}$.
One has $\mathfrak{k}_0 = \begin{pmatrix}
bi & -f+gi & hi\\
f+gi & -2bi & f+gi\\
hi & -f+gi & bi
\end{pmatrix}$ (i.e. $a=0, c=-f, g=d, h=e$) and
$\mathfrak{p}_0 = \begin{pmatrix}
a & c+di & ei\\
c-di & 0 & -c+di\\
-ei & -c-di & -a
\end{pmatrix}$ (i.e. $b=0, c=f, g =-d, h=-e$).
The maximal split tori $\mathfrak{a}_0$ in this case are the one-dimensional subspaces of $\mathfrak{p}_0$. But one can compute how each of them has a non-trivial centraliser in $\mathfrak{k}_0$ which has to be added to get a maximal torus = Cartan subalgebra in the generally accepted sense; the most obvious choice being
$\mathfrak{a}_0 = \begin{pmatrix}
a & 0 & 0\\
0 & 0 & 0\\
0 & 0 & -a
\end{pmatrix}$ which demands $\mathfrak{t}_0 = \begin{pmatrix}
bi & 0 & 0\\
0 & -2bi & 0\\
0 & 0 & bi
\end{pmatrix}$ as a complement, so that $\mathfrak{a}_0 \oplus \mathfrak{t}_0$ is a maximal torus and becomes the standard maximal split = split maximal torus in the complexification $\mathfrak{g}_{0}^\mathbb{C} \simeq \mathfrak{sl}_3(\mathbb{C})$.
Best Answer
To collect a few of the comments together into a coherent picture this boils down to the classification of maximal semisimple subalgebras of semisimple Lie algebras achieved by Dynkin as Moishe notes.
Let $\mathfrak{g}$ be a simple complex Lie algebra and let $\mathfrak{h}\leq \mathfrak{g}$ be a semisimple subalgebra. Firstly, we should note that the maximum possible rank of a semisimple subalgebra of $\mathfrak{g}$ is the rank of $\mathfrak{g}$. This follows from the observation that Torsten made which is that any Cartan subalgebra of $\mathfrak{h}$ is contained in a Cartan subalgebra of $\mathfrak{g}$. One way to see this would be to convince your self that semisimple elements of $\mathfrak{h}$ must be semisimple elements of $\mathfrak{g}$.
So a maximal rank subalgebra must have a Cartan subalgebra $\mathfrak{c}$ which is also a Cartan subalgebra of $\mathfrak{g}$. Note that this forces $\mathfrak{h}$ to be the span of $\mathfrak{c}$ together with some of the root spaces. In fact these roots must form a subsystem of our original root system. So the question becomes a combinatorial one: what are the maximal rank root subsystems of the root system of $\mathfrak{g}$?
Well there's a nice way to find these. Draw out the Dynkin diagram of $\mathfrak{g}$ then add a node to turn it into the extended diagram (This node corresponds to the lowest root and if there are two root lengths we can also extend by the lowest short root instead). Then remove any node and the lines attached to it. The remaining diagram is the Dynkin diagram of a maximal rank subalgebra. You can repeat this process recursively to find all the maximal rank subalgebras. If you try this with an $A_n$ root system you should see that you end up with another $A_n$ root system so that the only maximal rank subalgebra of $\mathfrak{sl}_{n+1}$ is itself. With other types you can get other possibilities though such as $C_i \times C_{n-i} \subset C_n$, $B_4 \subset F_4$ or $A_2\times A_2 \times A_2 \subset E_6$.
You can of course keep removing nodes without extending to find non-maximal subsystems but these won't cover all the non-maximal semisimple subalgebras and we need other techniques.
Here is a excellent answer by Torsten going into this process in more detail for full rank subsystems and here is an answer by me discussing the non-maximal ones. Both contain references to tables of the classification of root subsystems of a root system.