1) Let us replace the momentum indices ${\bf k}$ and $-{\bf k}$ with abstract indices $1$ and $2$, and ignore the momentum summation. The Hamiltonian then reads
$$\tag{1}H ~=~ \begin{pmatrix}a_1^{\dagger} & a_2\end{pmatrix}
M\begin{pmatrix}a_1 \\ a_2^{\dagger}\end{pmatrix},
$$
where
$$\tag{2} M~:=\begin{pmatrix}A & B\\B^{*} & A \end{pmatrix}~=~M^{\dagger},
\qquad A~\in~ \mathbb{R},\qquad B~\in~ \mathbb{C}.$$
A Bogoliubov transformation is on the form
$$\tag{3} \begin{pmatrix}a_1 \\ a_2^{\dagger}\end{pmatrix}
~=~U \begin{pmatrix}b_1 \\ b_2^\dagger\end{pmatrix}, \qquad U~:=~\begin{pmatrix}u_{11} & u_{12}\\u_{21} & u_{22}\end{pmatrix}.$$
Because a Bogoliubov transformation should preserve the canonical commutator relations (CCR), it is straightforward to check that the transformation matrix $U$ must belong to the Lie group
$$\tag{4} U(1,1)
~:=~\{U\in {\rm Mat}_{2\times 2}(\mathbb{C})
\mid U^{\dagger}\eta U~=~\eta \}, $$
where
$$\tag{5} \eta
~:=~\begin{pmatrix}1& 0\\0 & -1\end{pmatrix}$$
is the $1+1$ dimensional Minkowski metric. The Lie group $U(1,1)$ of Bogoliubov transformations is real and non-compact, and it is $4$-dimensional. In fact, one may prove that an element $U\in U(1,1)$ is of the form
$$\tag{6} U~=~\begin{pmatrix}u & v\\wv^* & wu^*\end{pmatrix},$$
where
$$\tag{7}u,v,w~\in~\mathbb{C},\quad |u|^2-|v|^2~=~1
\quad\text{and}\quad |w|~=~1. $$
The Hamiltonian then becomes of the form
$$\tag{8} H ~=~ \begin{pmatrix}b_1^{\dagger} & b_2\end{pmatrix}
N\begin{pmatrix}b_1 \\ b_2^{\dagger}\end{pmatrix},$$
where
$$\tag{9} N~:=~U^{\dagger}MU~=~N^{\dagger}.$$
Theorem I: We have the following two invariants under Bogoliubov transformations:
$$\tag{10} \det(N) ~=~\det(M)~=~A^2-|B|^2. $$
$$\tag{11} {\rm tr}(\eta N) ~=~{\rm tr}(\eta M)~=~0.$$
Proof of eq. (11): Use that $\eta N= U^{-1}\eta MU$ and $\eta M$ are connected via a similarity transformation. $\Box$
Corollary Ia:
The new $2\times 2$ matrix $N$ is of the form
$$\tag{12} N~:=\begin{pmatrix}A^{\prime} & B^{\prime}\\B^{\prime *} & A^{\prime} \end{pmatrix}~=~N^{\dagger},
\qquad A^{\prime}~\in~ \mathbb{R},\qquad B^{\prime}~\in~ \mathbb{C}.$$
It turns out that OP's assertion in the question formulation about diagonalizability is correct, cf. the following Corollary Ib.
Corollary Ib:
(i) The new $2\times 2$ matrix $N$ can only be diagonal if $|A|>|B|$ or $B=0$.
(ii) The new $2\times 2$ matrix $N$ can only be off-diagonal if $|A|<|B|$ or $A=0$.
(iii) If $|A|=|B|\neq 0$, then the new $2\times 2$ matrix $N$ is neither diagonal nor off-diagonal.
Proof of (i): The diagonal matrix $N$ must satisfy
$$ \tag{13} 0~\leq~ A^{\prime 2} ~=~ \det(N) ~=~\det(M)~=~A^2-|B|^2. $$
Hence $|A|>|B|$ or $|A|=|B|$. In the latter case, $A^{\prime}=0$, so $N=0$, and hence $M=0$, and in particular $B=0$. $\Box$
The proofs of (ii) and (iii) are left as exercises.
3) We will next argue that the case $A<|B|$ is not physically relevant, cf. Theorem II.
Theorem II:
(i) The Hamiltonian $H$ is positive definite if $A >|B|$.
(ii) The spectrum of $H$ is non-negative if $A\geq |B|$.
(iii) The spectrum of $H$ is unbounded from below if $A < |B|$.
A proof of Theorem II is left as an exercise. A proof at the classical level can be established by replacing the operators $a_1$ and $a_2$ with the corresponding classical complex variables, and then investigate the signature of the Hessian for the corresponding classical Hamiltonian function.
4) Finally, the Bogoliubov transformation (3) may be cast in a special relativistic language. One may view the matrix $M$, or equivalently $(A,B)$, as a point in $1+2$ dimensional Minkowski space with time coordinate $A\in \mathbb{R}$ and space coordinates $B\in \mathbb{C}\cong \mathbb{R}^2$. The invariant Minkowski length
$$\tag{14} \det(M)~=~A^2-|B|^2$$
is preserved under the action
$$\tag{15} \rho:U~\mapsto~ (U^{-1})^{\dagger}MU^{-1}$$
of the Lie group $U(1,1)$ of Bogoliubov transformations. Therefore $\rho$ is a Lie group homomorphism
$$\tag{16} \rho: \quad\to\quad O(1,2;\mathbb{R}) $$
into the $3$-dimensional Lorentz group $O(1,2;\mathbb{R})$. The condition $|A| >|B|$ ($|A| < |B|$) is the condition for being a time-like (space-like) vector, respectively. Intuitively, OP's observation concerning diagonalizability may be understood as the fact that one cannot turn a space-like vector into time-like vector by a Lorentz transformation.
That's essentially what we mean by a quasiparticle: a degree of freedom that has been otherwise decoupled from the rest of the system and which behaves exactly like a particle as far as quantum mechanics can tell. Since the (anti)commutation relations are the crucial part of the quantum mechanics of the relevant particles, those need to be carried as well.
Moreover, the canonical anticommutation relations are the best you could hope for anyways. This is because the transformation,
\begin{align}
b_j &= \sum_{k=1}^n \left(U_{jk}c_k + V_{jk}c_k^\dagger\right)\\
b_j^\dagger &= \sum_{k=1}^n \left( U^*_{jk}c_k^\dagger + V_{jk}^*c_k\right)
,
\end{align}
requires the anticommutator to read
\begin{align}
\{b_i^\dagger,b_j\}
&=
\left\{
\sum_{k=1}^n \left( U_{ik}^*c_k^\dagger + V_{ik}^*c_k\right),
\sum_{l=1}^n \left(U_{jl}c_l + V_{jl}c_l^\dagger\right)
\right\}
\\&=
\sum_{k=1}^n \sum_{l=1}^n \left[
U_{ik}^*U_{jl}
\left\{c_k^\dagger,c_l \right\}
+
V_{ik}^*V_{jl}
\left\{c_k,c_l^\dagger\right\}
\right]
\\&=
\sum_{k=1}^n \sum_{l=1}^n \left[
-U_{ik}^*U_{jl}
+
V_{ik}^*V_{jl}
\right]
\delta_{kl}
\\&=
\sum_{k=1}^n\left[ V_{ik}^*V_{jk} - U_{ik}^*U_{jk} \right]
\end{align}
in the most general case. Thus, in the worst-case scenario, you've bungled up the (anti)commutation relations, and you're going to need a nonzero (anti)commutator any time you need to slide a $b_j$ past a $b_i^\dagger$, even if $i\neq j$. Thus, you require at the very least that the anticommutator be diagonal.
Moreover, since $\{b,b^\dagger\}=bb^\dagger + b^\dagger b$ is the sum of two positive semidefinite hermitian operators, the diagonal anticommutator needs to be real and non-negative, so you've got two possibilities:
- it can be positive, in which case it can be rescaled to one, or
- it can be zero, in which case you're essentially losing information, probably via a degenerate matrix, and the algebra spanned by the $b_i,b_i^\dagger$ will be unable to fully span all of the $c_i,c_i^\dagger$.
More broadly speaking, you're obliged to "conserve the quantumness" of the problem (unless you explicitly want to make mean-field approximations or similar) and this requires you to maintain a full set of generators for the operator algebra and therefore to have nontrivial anticommutators between your new quasiparticle fermionic operators.
Best Answer
You are correct, Bogoliubov transformations are not unitary in general. By definition,
The algebraic relations are mainly the commutation/anticommutation relations which define the bosonic/fermionic operators. Nowhere in the definition did we specified that the transformation should be unitary. In fact, the Bogoliubov transformation (in its most generic form) is symplectic for bosons and orthogonal for fermions. In neither case is the Bogoliubov transformation unitary. The Bogoliubov transformation of bosons correspond to the linear canonical transformation of oscillators in classical mechanics (because bosons are quanta of oscillators), and we know the linear canonical transformations are symplectic due to the symplectic structure of the classical phase space.
So to be more specific, what are the restrictions on Bogoliubov transformations? Let us consider the case of $n$ single particle modes of either bosons $b_i$ or fermions $f_i$ (where $i=1,2,\cdots,n$ labels the single particle states, such as momentum eigenstates). Both $b_i$ and $f_i$ are not Hermitian operators, which are not quite convenient for a general treatment (because we can't simply treat $b_i$ and $b_i^\dagger$ as the independent basis since they are still related by the particle-hole transformation). Therefore we choose to rewrite the operators as the following linear combinations (motivated by the idea of decomposing a complex number into two real numbers like $z=x+\mathrm{i}y$): $$\begin{split}b_i&=a_i+\mathrm{i}a_{n+i}\\b_i^\dagger&=a_i-\mathrm{i}a_{n+i}\end{split}\qquad \begin{split}f_i&=c_i+\mathrm{i}c_{n+i}\\f_i^\dagger&=c_i-\mathrm{i}c_{n+i}\end{split}$$ where $a_i=a_i^\dagger$ and $c_i=c_i^\dagger$ (for $i=1,2,\cdots,2n$) are Hermitian operators (analogus to real numbers). They must inherit the commutation or anticommutation relations from the "complex" bosons $b_i$ and fermions $f_i$: $$\begin{split}[b_i,b_j^\dagger]=\delta_{ij},[b_i,b_j]=[b_i^\dagger,b_j^\dagger]=0&\Rightarrow[a_i,a_j]=\frac{1}{2}g_{ij}^a \\ \{f_i,f_j^\dagger\}=\delta_{ij}, \{f_i,f_j\}=\{f_i^\dagger,f_j^\dagger\}=0&\Rightarrow\{c_i,c_j\}=\frac{1}{2}g_{ij}^c\end{split}$$ where $g_{ij}^a$ and $g_{ij}^c$ are sometimes called the quantum metric for bosons and fermions respectively. In matrix forms, they are given by $$g^a=\mathrm{i}\left[\begin{matrix}0&\mathbb{1}_{n\times n}\\-\mathbb{1}_{n\times n}&0\end{matrix}\right] \qquad g^c=\left[\begin{matrix}\mathbb{1}_{n\times n}&0\\0&\mathbb{1}_{n\times n}\end{matrix}\right],$$ with $\mathbb{1}_{n\times n}$ being the $n\times n$ identity matrix. So to preserve the algebraic relations among the creation/annihilation operators is to preserve the quantum metric. General linear transformations of the operators $a_i$ and $c_i$ take the form of $$a_i\to \sum_{j}W_{ij}^a a_j\qquad c_i\to \sum_{j}W_{ij}^c c_j,$$ where the transformation matrix elements $W_{ij}^a, W_{ij}^c\in\mathbb{R}$ must be real, in order to ensure that the operators $a_i$ and $c_i$ remain Hermitian after the transformation. Then to preserve the quantum metric is to require $$W^a g^a W^{a\intercal}= g^a\qquad W^c g^c W^{c\intercal}= g^c.$$ So any real linear transformation satisfying the above conditions is a Bogoliubov transformation in the most general sense. Then depending on the property of the quantum metric, the Bogoliubov transformation is either symplectic or orthogonal. For the bosonic quantum metric, $g^a=-g^{a\intercal}$ is antisymmetric, so the transformation $W^a$ is symplectic. For the fermionic quantum metric, $g^c=g^{c\intercal}$ is symmetric, so the transformation $W^c$ is orthogonal.