So this question is part (c) to the problem:
Let $T: V \to W$ and $S: W \to U$ be linear transformations of finite-dimensional vectors spaces over the field $\Bbb F$.
- Show that $\ker(T)\subseteq\ker(S\circ T)$ and $\operatorname{im}(S\circ T)\subseteq\operatorname{im}(S)$.
- Show that $\operatorname{rank}(S\circ T)≤\operatorname{rank}(T)$ and $\operatorname{rank}(S\circ T)≤\operatorname{rank}(S)$.
Now I showed part a and b and I am assuming that we should use these two conclusions to answer part c. So in that case it makes sense to try and define
$\ker(A)\subseteq{\ker(BA)}$ because of the definition of linear maps.
We know the definiton of $\ker(T) := \{v\in V\; |\; T(v) = 0\}$ and hence
if $\det(A) = 0$ then $\ker(\det) :=\{A\in M_{n\times n}(F)\;|\; \det(A) = 0\}$. And when $\det(BA) = 0$, we have from the result in part 1, i.e. $\ker(\det)\subseteq \ker(\det\circ \det)$. But I cannot makes sense of part 1 and 2 any further. I do know that $BA=AB$ when $A$ is invertible, but since $\det(A) = 0$ then we can't have that equality.
Best Answer
$\det A=0\Longleftrightarrow\text{rank}A=\dim(\text{Im}(S))<n\Longleftrightarrow\text{Im}(S)\subset U$. You know from part $(b)$ that $\text{Im}(S\circ T)\subseteq\text{Im}(S)\subset U$, which implies $\text{Im}(S\circ T)\subset U$ and $\text{rank}AB<n$. What does this tell you about $\det AB$?