To test for positive definiteness, as long as the principal submatrices are nested (and we have submatrices of all sizes, of course), they don't need to be the leading ones or the trailing ones. This is because by relabelling the rows and columns, any nested sequence of principal submatrices can be turned into a sequence of leading principal submatrices.
E.g. for any $S\subseteq\{1,2,3,4,5\}$, let $A(S)$ denotes the principal submatrix $(a_{ij})_{i,j\in S}$. Then $A(\{4\}),\,A(\{2,4\}),\,A(\{1,2,4\}),\,A(\{1,2,4,5\})$ and $A$ form a nested sequence of principal submatrices, and $A$ is positive definite if and only if the determinants of these submatrices are all positive.
In your example, $A_1=A(\{1\},\,B_2=A(\{4,5\}),\,A_3=A(\{1,2,3\}),\,A_4=A(\{1,2,3,4\})$ and $B_5=A$ do not form a nested sequence of principal submatrices. Even if their determinants are all positive, $A$ is not necessarily positive definite. For a counterexample, consider $A=\operatorname{diag}(1,-1,-1,1,1)$.
By the way:
- You lack vocabularies. I think you need to look up the meanings of the words "leading/trailing" and principal in your textbook.
- The test has a name. It is called Sylvester's criterion.
- You seem to have misunderstood how to use Sylvester's criterion for semidefiniteness. To verify that a Hermitian matrix is positive semidefinite, you need to show that all principal minors are nonnegative. It does not suffice to show that all leading principal minors are nonnegative. For a counterexample, consider $A=\operatorname{diag}(0,1,-1,0)$. All leading principal minors of this $A$ are zero (hence nonnegative), but $A$ is indefinite.
I assume that $M$ is an $N \times N$ block matrix and that each block is $N \times N$, as you have indicated (but not stated explicitly) in your post.
Any matrix with diagonal blocks (assuming the blocks have the same size) can be converted to a block-diagonal matrix. In particular, suppose that $a_{ijk}$ denotes the $k$th diagonal entry of the block $A_{ij}$, so that
$$
A_{ij} = \pmatrix{a_{ij1} \\ & \ddots \\ && a_{ijN}}.
$$
There exists a permutation matrix $P$ such that
$$
PMP^T = \pmatrix{B_1\\ & \ddots \\ && B_N},
$$
where
$$
B_k = \pmatrix{
a_{11k} & \cdots & a_{1Nk}\\
\vdots & \ddots & \vdots \\
a_{N1k} & \cdots & a_{NNk}}.
$$
It follows that $M$ is positive definite if and only if every $N \times N$ matrix $B_k$ is positive definite.
Where an ordinary Cholesky decomposition on the $N^2 \times N^2$ matrix would have complexity $O((N^2)^3) = O(N^6)$, attempting a separate Cholesky decomposition of each $B_k$ has complexity $N \cdot O(N^3) = O(N^4)$.
If you're interested in what the matrix $P$ looks like, it can be written as
$$
P = \sum_{i,j = 1}^N (e_{i} \otimes e_j)(e_j \otimes e_i)^T
$$
where $e_i$ denotes the $i$th canonical basis vector (the $i$th column of the identity matrix), and $\otimes$ denotes the Kronecker product.
Best Answer
We have \begin{aligned} \pmatrix{-1&a&-1\\ a&-4&a\\ -1&a&-1} &=-\pmatrix{1&-a&1\\ -a&4&-a\\ 1&-a&1}\\ &=-\pmatrix{1\\ -a\\ 1}\pmatrix{1&-a&1} -(4-a^2)\pmatrix{0\\ 1\\ 0}\pmatrix{0&1&0} \end{aligned} and hence $x^TBx=-(x_1-ax_2+x_3)^2 - (4-a^2)x_2^2$. Therefore $B$ is never positive semidefinite, and it is negative semidefinite (resp. negative definite) if and only if $4-a^2$ is nonnegative (resp. negative).