Let $A, B$ be square matrices of size $n$, $n \geq 2$, containing real entries.
$\DeclareMathOperator\rank{rank}$
If the following properties take place: $ A^3 + B^2 = I_n $ and $A^5=A^2$,
then $\det(A^2 + B^2 + I_n) \geq 0 $ and $\rank(I_n + AB^2) = \rank(I_n – AB^2) $.
So far, I've managed to solve the case when $ A $ is invertible.
When $A$ is not invertible, I've obtained $A^2B^2=O_n$, $\rank(A^2) + \rank(B^2) = n $, $\rank(A^2)=\rank(A^m),$ for every $m\geq 2$. All of the aforementioned these also hold when $A$ is invertible.
I would also appreciate it if anyone can recommend me a textbook/book with problems like these.
Best Answer
From the first condition, $B^2=1-A^3$, so that by the second, $$ 0=A^2-A^5=A^2B^2=AB^2A=B^2A^2.$$ Now $$(1-AB^2)(1+AB^2)=1-AB^2AB^2=1 $$ so that $$\operatorname{rank}(1\pm AB^2)=n. $$
Note that $$ B^4=(1-A^3)^2=1-2A^3+A^6=1-A^3=B^2.$$ Let $U=\operatorname{im} B^2$ and $W=\operatorname{im} (1-B^2)$. Then for $u\in U$, we have $u=B^2v$ for some $v\in \Bbb R^n$ and so $B^2u=B^4v=B^2v=u$, i.e., $B^2$ acts as identity on $U$. Likewise, $B^2$ acts as zero map on $W$. We conclude $U\cap W=\{0\}$, and as each $v\in \Bbb R^n$ can be written as $v=B^2v+(1-B^2)v$, we conclude obtain the $B^2$-invariant direct sum decomposition $$\Bbb R^n=U\oplus W,$$ where also $U=\ker(1-B^2)$, $W=\ker B^2$. This decomposition is also $A^2$-invariant: As $A^2B^2=B^2A^2=0$, we see that $A^2$ maps $\Bbb R^n$ to $\ker B^2=W$ while acting as zero map on $U$. Then $U,W$ are also $(1+A^2+B^2)$-invariant and we can compute the determinant blockwise. As $A^2$ acts as zero and $B^2$ as identity on $U$, we see that $(A^2+B^2+1)u$ acts as twice the identity on $U$. On $W$, $1+A^2+B^2$ coincides with $A^2$. Therefore $$\begin{align}\det(A^2+B^2+1)&=\det( (A^2+B^2+1)|_U)\det( (A^2+B^2+1)|_W)\\&=\det(2|_U)\det(A^2|_W)\\&=2^{\dim U}\det( A|_W)^2\\&\ge 0.\end{align}$$