The first definition corresponds to maximal tori and should be used; the second corresponds to maximal split tori.
The answer by ಠ_ಠ correctly states the definition of Cartan subalgebras for general Lie algebras: It is a subalgebra that is nilpotent and its own normaliser. In the case at hand, it is useful to introduce the following concepts:
Let $\mathfrak{g}$ be a semisimple Lie algebra over any field of characteristic 0. A subalgebra of $\mathfrak{g}$ is called toral if it is abelian and consists of semisimple elements. It is called split toral if it is abelian and consists of diagonalisable elements.
(Of course this is made to resemble tori and split tori in the group setting; I will just write "(split) torus" occasionally.)
Now one has:
Lemma: For $\mathfrak{g}$ as above, a subalgebra is maximal toral iff it is a Cartan subalgebra (= self-normalising & nilpotent).
(This is e.g. exercise 3 to ch. VII $\S$ 2 in Bourbaki's Lie Groups and Lie Algebras.)
As long as one works over algebraically closed fields, one rarely hears of toral and split toral subalgebras, since by algebraic closedness, toral is the same as split toral ("every torus is split"), so that by the lemma:
For a subalgebra of a semisimple Lie algebra over $\mathbb{C}$,
maximal toral = maximal split toral = Cartan subalgebra.
But over other fields, in our case $\mathbb{R}$, we have distinct notions of
- maximal toral subalgebras, and
- maximal split toral subalgebras.
By the lemma, 1. corresponds to the first (Knapp's) definition you give, and the generally accepted notion of Cartan subalgebras.
The second usage that you describe corresponds to 2. That is, what they call a Cartan subalgebra there is actually a maximal split toral subalgebra (in the group setting, it would be a maximal split torus, as opposed to a maximal torus). I have not seen this usage myself and would advise against it, since it does not match the general definition of Cartan subalgebra. Also, it would make the notion not invariant under scalar extension. Calling $\mathfrak{a}_0$ a maximal split torus is much better.
As to your last question, even in split Lie algebras, i.e. when there exists a split maximal torus [Beware the order of words: this is a maximal torus which happens to be split; not, as in notion 2, a maximal one among the split tori], the second usage would be more restrictive, since there can still be maximal tori which are not split.
-- Example: $\mathfrak{g_0} = \mathfrak{sl}_2(\mathbb{R}) = \lbrace \pmatrix{a & b \\
c &-a } : a,b,c \in \mathbb{R}\rbrace$. Then the second usage sees the split Cartan subalgebras (= one-dimensional subspaces) in $\mathfrak{p}_0 = \pmatrix{a & b \\
b &-a }$, but misses the non-split one that constitutes $\mathfrak{k}_0$, $\pmatrix{0 & b \\
-b &0 }$. --
If $\mathfrak{g}_0$ is not split, notion 2 does not even give a subset of notion 1, but they are disjoint: The ones in notion 2 have dimension strictly less than those in notion 1. And $\mathfrak{g}_0$ can still be far from compact. As an example, the following 8-dimensional real Lie algebra is a matrix representation of the quasi-split form of type $A_2$:
$\mathfrak{g}_0 = \lbrace
\begin{pmatrix}
a+bi & c+di & ei\\
f+gi & -2bi & -c+di\\
hi & -f+gi & -a+bi
\end{pmatrix} : a, ..., h \in \mathbb{R} \rbrace$; according to the nomenclature here, one might call this $\mathfrak{su}_{1,2}$.
One has $\mathfrak{k}_0 = \begin{pmatrix}
bi & -f+gi & hi\\
f+gi & -2bi & f+gi\\
hi & -f+gi & bi
\end{pmatrix}$ (i.e. $a=0, c=-f, g=d, h=e$) and
$\mathfrak{p}_0 = \begin{pmatrix}
a & c+di & ei\\
c-di & 0 & -c+di\\
-ei & -c-di & -a
\end{pmatrix}$ (i.e. $b=0, c=f, g =-d, h=-e$).
The maximal split tori $\mathfrak{a}_0$ in this case are the one-dimensional subspaces of $\mathfrak{p}_0$. But one can compute how each of them has a non-trivial centraliser in $\mathfrak{k}_0$ which has to be added to get a maximal torus = Cartan subalgebra in the generally accepted sense; the most obvious choice being
$\mathfrak{a}_0 = \begin{pmatrix}
a & 0 & 0\\
0 & 0 & 0\\
0 & 0 & -a
\end{pmatrix}$ which demands $\mathfrak{t}_0 = \begin{pmatrix}
bi & 0 & 0\\
0 & -2bi & 0\\
0 & 0 & bi
\end{pmatrix}$ as a complement, so that $\mathfrak{a}_0 \oplus \mathfrak{t}_0$ is a maximal torus and becomes the standard maximal split = split maximal torus in the complexification $\mathfrak{g}_{0}^\mathbb{C} \simeq \mathfrak{sl}_3(\mathbb{C})$.
If $\mathfrak{g}$ is a complex, semisimple, finite-dimensional Lie algebra and $\mathfrak{h} \subseteq \mathbb{g}$ a Cartan subalgebra, then $\mathfrak{h}$ is abelian and self-centralizing, i.e. $Z_\mathfrak{g}(\mathfrak{h}) = \mathfrak{h}$ (this is, for example, show in Humphrey’s book).
If $\mathfrak{h}$ was not a maximal abelian subalgebra with respect to the inclusion, then $\mathfrak{h}$ would be properly contained is some maximal abelian subalgebra $\mathfrak{a}$ of $\mathfrak{g}$. But then $\mathfrak{a} \subseteq Z_\mathfrak{g}(\mathfrak{h})$, contradicting $\mathfrak{h}$ being self-centralizing. So if $\mathfrak{g}$ is finite-dimensional it is true that any Cartan subalgebra is a maximal abelian subalgebra. (I don’t know what happens in the infinite-dimensional case.)
The example in Wikipedia, namely
$$
\mathfrak{a} =
\left\{
\begin{pmatrix}
0 & A \\
0 & 0
\end{pmatrix}
\in \mathfrak{sl}_{2n}(\mathbb{C})
\,\middle|\,
A \in \mathfrak{gl}_n(\mathbb{C})
\right\},
$$
shows that while Cartan-subalgebras are maximal abelian subalgebras with respect to the inclusion, they are not necessarily of maximal dimension among all abelian subalgebras: Every Cartan subalgebra of $\mathfrak{sl}_{2n}(\mathbb{C})$ has dimension $2n-1$ (for example the traceless diagonal matrices), but $\mathfrak{a}$ has dimension $n^2$.
What this tells us that $\mathfrak{a}$ does not contain a Cartan subalgebra of $\mathfrak{sl}_{2n}(\mathbb{C})$ (strictly speaking we only get this for $n > 1$, but for $n = 1$ this is also easy to see).
Best Answer
Well, if you come to the definition of a Cartan subalgebra (in an arbitrary finite-dimensional Lie algebra over an arbitrary infinite field — denote by $d$ the dimension), you see that it is defined as $K_x=\mathrm{Ker}(\mathrm{ad}(x)^d)$, where $x$ is regular, and regular precisely means that $K_x$ has minimal dimension.
So, the Cartan rank (I don't like calling it the rank in this generality) is by definition $\inf_{x\in\mathfrak{g}}\dim\mathrm{Ker}(\mathrm{ad}(x)^d)$.
Moreover, if $\mathfrak{g}$ is semisimple in characteristic zero, then the Cartan rank is $\inf_{x\in\mathfrak{g}}\dim\mathrm{Ker}(\mathrm{ad}(x))$.
This is, at least, in principle, constructive: choose a basis $(e_i)$: consider $w=\sum_i t_ie_i$. Compute $\mathrm{ad}(w)^d$, treating the $t_i$ as indeterminates. Then you get a $d\times d$-matrix with entries in $K[t_1,\dots,t_n]$. Computing the determinant of all minors yields its rank (some number $k'$), and hence yields the Cartan rank (which is $d-k'$).
This shows, if $K$ is a computable field, that there is an algorithm whose input is $d$ and the $d^3$ structure constants of a $d$-dimensional Lie algebra, and outputs the Cartan rank.
In practice, this is not greatly efficient, because you don't want to compute $\mathrm{ad}(w)^d$ (which involves huge polynomials) and so many minors within it.
So there is a better algorithm. If $\mathfrak{g}$ is nilpotent, the Cartan rank is $d$. Otherwise, there exists $x$ with $\mathrm{ad}(x)$ is not nilpotent (this is a theorem, e.g., in Jacobson's book). The first step is thus to determine if $\mathfrak{g}$ is nilpotent, and otherwise to find $x$. One can efficiently compute the center (as equal to $\bigcap_i\mathrm{Ker}(\mathrm{ad}(e_i))$) and so on, so this computes the ascending central series, and its union $\mathfrak{z}$ ("hypercenter"). If $\mathfrak{z}=0$, then $\mathfrak{g}$ is nilpotent. Otherwise, one has to find $x$. Since generically $x$ is not ad-nilpotent, I'd say that an efficient non-deterministic way to find a non-ad-nilpotent element is to pick a "random" element and check if it's ad-nilpotent. Then one computes $\mathrm{Ker}(\mathrm{ad}(x)^d)$. If the latter is nilpotent, this is a Cartan subalgebra and we are done. Otherwise, we find a non-ad-nilpotent $x'$ therein and we go on (actually, if $x$ was chosen sufficiently random, one step should be enough).