1) Let us replace the momentum indices ${\bf k}$ and $-{\bf k}$ with abstract indices $1$ and $2$, and ignore the momentum summation. The Hamiltonian then reads
$$\tag{1}H ~=~ \begin{pmatrix}a_1^{\dagger} & a_2\end{pmatrix}
M\begin{pmatrix}a_1 \\ a_2^{\dagger}\end{pmatrix},
$$
where
$$\tag{2} M~:=\begin{pmatrix}A & B\\B^{*} & A \end{pmatrix}~=~M^{\dagger},
\qquad A~\in~ \mathbb{R},\qquad B~\in~ \mathbb{C}.$$
A Bogoliubov transformation is on the form
$$\tag{3} \begin{pmatrix}a_1 \\ a_2^{\dagger}\end{pmatrix}
~=~U \begin{pmatrix}b_1 \\ b_2^\dagger\end{pmatrix}, \qquad U~:=~\begin{pmatrix}u_{11} & u_{12}\\u_{21} & u_{22}\end{pmatrix}.$$
Because a Bogoliubov transformation should preserve the canonical commutator relations (CCR), it is straightforward to check that the transformation matrix $U$ must belong to the Lie group
$$\tag{4} U(1,1)
~:=~\{U\in {\rm Mat}_{2\times 2}(\mathbb{C})
\mid U^{\dagger}\eta U~=~\eta \}, $$
where
$$\tag{5} \eta
~:=~\begin{pmatrix}1& 0\\0 & -1\end{pmatrix}$$
is the $1+1$ dimensional Minkowski metric. The Lie group $U(1,1)$ of Bogoliubov transformations is real and non-compact, and it is $4$-dimensional. In fact, one may prove that an element $U\in U(1,1)$ is of the form
$$\tag{6} U~=~\begin{pmatrix}u & v\\wv^* & wu^*\end{pmatrix},$$
where
$$\tag{7}u,v,w~\in~\mathbb{C},\quad |u|^2-|v|^2~=~1
\quad\text{and}\quad |w|~=~1. $$
The Hamiltonian then becomes of the form
$$\tag{8} H ~=~ \begin{pmatrix}b_1^{\dagger} & b_2\end{pmatrix}
N\begin{pmatrix}b_1 \\ b_2^{\dagger}\end{pmatrix},$$
where
$$\tag{9} N~:=~U^{\dagger}MU~=~N^{\dagger}.$$
Theorem I: We have the following two invariants under Bogoliubov transformations:
$$\tag{10} \det(N) ~=~\det(M)~=~A^2-|B|^2. $$
$$\tag{11} {\rm tr}(\eta N) ~=~{\rm tr}(\eta M)~=~0.$$
Proof of eq. (11): Use that $\eta N= U^{-1}\eta MU$ and $\eta M$ are connected via a similarity transformation. $\Box$
Corollary Ia:
The new $2\times 2$ matrix $N$ is of the form
$$\tag{12} N~:=\begin{pmatrix}A^{\prime} & B^{\prime}\\B^{\prime *} & A^{\prime} \end{pmatrix}~=~N^{\dagger},
\qquad A^{\prime}~\in~ \mathbb{R},\qquad B^{\prime}~\in~ \mathbb{C}.$$
It turns out that OP's assertion in the question formulation about diagonalizability is correct, cf. the following Corollary Ib.
Corollary Ib:
(i) The new $2\times 2$ matrix $N$ can only be diagonal if $|A|>|B|$ or $B=0$.
(ii) The new $2\times 2$ matrix $N$ can only be off-diagonal if $|A|<|B|$ or $A=0$.
(iii) If $|A|=|B|\neq 0$, then the new $2\times 2$ matrix $N$ is neither diagonal nor off-diagonal.
Proof of (i): The diagonal matrix $N$ must satisfy
$$ \tag{13} 0~\leq~ A^{\prime 2} ~=~ \det(N) ~=~\det(M)~=~A^2-|B|^2. $$
Hence $|A|>|B|$ or $|A|=|B|$. In the latter case, $A^{\prime}=0$, so $N=0$, and hence $M=0$, and in particular $B=0$. $\Box$
The proofs of (ii) and (iii) are left as exercises.
3) We will next argue that the case $A<|B|$ is not physically relevant, cf. Theorem II.
Theorem II:
(i) The Hamiltonian $H$ is positive definite if $A >|B|$.
(ii) The spectrum of $H$ is non-negative if $A\geq |B|$.
(iii) The spectrum of $H$ is unbounded from below if $A < |B|$.
A proof of Theorem II is left as an exercise. A proof at the classical level can be established by replacing the operators $a_1$ and $a_2$ with the corresponding classical complex variables, and then investigate the signature of the Hessian for the corresponding classical Hamiltonian function.
4) Finally, the Bogoliubov transformation (3) may be cast in a special relativistic language. One may view the matrix $M$, or equivalently $(A,B)$, as a point in $1+2$ dimensional Minkowski space with time coordinate $A\in \mathbb{R}$ and space coordinates $B\in \mathbb{C}\cong \mathbb{R}^2$. The invariant Minkowski length
$$\tag{14} \det(M)~=~A^2-|B|^2$$
is preserved under the action
$$\tag{15} \rho:U~\mapsto~ (U^{-1})^{\dagger}MU^{-1}$$
of the Lie group $U(1,1)$ of Bogoliubov transformations. Therefore $\rho$ is a Lie group homomorphism
$$\tag{16} \rho: \quad\to\quad O(1,2;\mathbb{R}) $$
into the $3$-dimensional Lorentz group $O(1,2;\mathbb{R})$. The condition $|A| >|B|$ ($|A| < |B|$) is the condition for being a time-like (space-like) vector, respectively. Intuitively, OP's observation concerning diagonalizability may be understood as the fact that one cannot turn a space-like vector into time-like vector by a Lorentz transformation.
You are correct, Bogoliubov transformations are not unitary in general. By definition,
Bogoliubov transformations are linear transformations of creation/annihilation operators that preserve the algebraic relations among them.
The algebraic relations are mainly the commutation/anticommutation relations which define the bosonic/fermionic operators. Nowhere in the definition did we specified that the transformation should be unitary. In fact, the Bogoliubov transformation (in its most generic form) is symplectic for bosons and orthogonal for fermions. In neither case is the Bogoliubov transformation unitary. The Bogoliubov transformation of bosons correspond to the linear canonical transformation of oscillators in classical mechanics (because bosons are quanta of oscillators), and we know the linear canonical transformations are symplectic due to the symplectic structure of the classical phase space.
So to be more specific, what are the restrictions on Bogoliubov transformations? Let us consider the case of $n$ single particle modes of either bosons $b_i$ or fermions $f_i$ (where $i=1,2,\cdots,n$ labels the single particle states, such as momentum eigenstates). Both $b_i$ and $f_i$ are not Hermitian operators, which are not quite convenient for a general treatment (because we can't simply treat $b_i$ and $b_i^\dagger$ as the independent basis since they are still related by the particle-hole transformation). Therefore we choose to rewrite the operators as the following linear combinations (motivated by the idea of decomposing a complex number into two real numbers like $z=x+\mathrm{i}y$):
$$\begin{split}b_i&=a_i+\mathrm{i}a_{n+i}\\b_i^\dagger&=a_i-\mathrm{i}a_{n+i}\end{split}\qquad
\begin{split}f_i&=c_i+\mathrm{i}c_{n+i}\\f_i^\dagger&=c_i-\mathrm{i}c_{n+i}\end{split}$$
where $a_i=a_i^\dagger$ and $c_i=c_i^\dagger$ (for $i=1,2,\cdots,2n$) are Hermitian operators (analogus to real numbers). They must inherit the commutation or anticommutation relations from the "complex" bosons $b_i$ and fermions $f_i$:
$$\begin{split}[b_i,b_j^\dagger]=\delta_{ij},[b_i,b_j]=[b_i^\dagger,b_j^\dagger]=0&\Rightarrow[a_i,a_j]=\frac{1}{2}g_{ij}^a \\
\{f_i,f_j^\dagger\}=\delta_{ij}, \{f_i,f_j\}=\{f_i^\dagger,f_j^\dagger\}=0&\Rightarrow\{c_i,c_j\}=\frac{1}{2}g_{ij}^c\end{split}$$
where $g_{ij}^a$ and $g_{ij}^c$ are sometimes called the quantum metric for bosons and fermions respectively. In matrix forms, they are given by
$$g^a=\mathrm{i}\left[\begin{matrix}0&\mathbb{1}_{n\times n}\\-\mathbb{1}_{n\times n}&0\end{matrix}\right] \qquad g^c=\left[\begin{matrix}\mathbb{1}_{n\times n}&0\\0&\mathbb{1}_{n\times n}\end{matrix}\right],$$
with $\mathbb{1}_{n\times n}$ being the $n\times n$ identity matrix. So to preserve the algebraic relations among the creation/annihilation operators is to preserve the quantum metric. General linear transformations of the operators $a_i$ and $c_i$ take the form of
$$a_i\to \sum_{j}W_{ij}^a a_j\qquad c_i\to \sum_{j}W_{ij}^c c_j,$$
where the transformation matrix elements $W_{ij}^a, W_{ij}^c\in\mathbb{R}$ must be real, in order to ensure that the operators $a_i$ and $c_i$ remain Hermitian after the transformation. Then to preserve the quantum metric is to require
$$W^a g^a W^{a\intercal}= g^a\qquad W^c g^c W^{c\intercal}= g^c.$$
So any real linear transformation satisfying the above conditions is a Bogoliubov transformation in the most general sense. Then depending on the property of the quantum metric, the Bogoliubov transformation is either symplectic or orthogonal. For the bosonic quantum metric, $g^a=-g^{a\intercal}$ is antisymmetric, so the transformation $W^a$ is symplectic. For the fermionic quantum metric, $g^c=g^{c\intercal}$ is symmetric, so the transformation $W^c$ is orthogonal.
Best Answer
This is an eigenvalue problem.
Let's assume your Bogoliubov transformation is of the form: $(a_k,b_k)^T=X(c_k,d_k)^T$. What this transformation do is let your Hamiltonian become: $H_k=w_1c_k^\dagger c_k+w_2 d_k^\dagger d_k$, with the anti-commute relation holds for new field operators $c_k$ and $d_k$.
Now you can check that $X$ is just the matrix where its columns are just the Normalized Eigenvectors of your original matrix.