I am working on a proof of the following problem and I am not sure if I am making the appropriate reduction to a smaller space.
Let $F$ be a subspace of the vector space $\mathbb{C}^{4\times4}$ such that for every $A,B \in F$ we have $AB = BA$. If there exists a matrix $A$ in $F$ having at least two distinct characteristic values prove $\dim(F) \leq 4$
Sketch:
It is easy to rule out the case when the characteristic has four distinct roots since every operator that commutes with A in this case is a polynomial in T and so the family of matrices in $F$ can be simultaneously diagonalized so $\dim(F) \leq 4$ in this case.
We have two cases other for the characteristic polynomial $p_A(x) = (x-c_1)(x-c_2)(x-c_3)^2$
or $P_A(x) = (x-c_1)^{r_1}(x-c_2)^{r_2}$ where $r_1 = 1,2,3$ and $r_2 = 3,2,1$ respectively.
I think I know how to handle the first case but the case with only two distinct characteristic values I am not sure about. I will list the proof I know when the characteristic polynomial has three distinct characteristic values in order to give motivation for my question.
let $W_1$ be the nullspace of$A-c_1I$, $W_2$ the nullspace of $A-c_2I$ and $W_3$ the nullspace of $(A-c_3I)^2$. Observe that $W_1 \cap W_2 \cap W_3 = \{ 0\}$ so that $\mathbb{C}^4 = W_1 \oplus W_2 \oplus W_3$.
Notice any B that commutes with $A$ implies that $W_1$, $W_2$, $W_3$ are invariant subspaces for $B$.
It then follows that the dimension of $F$ is the sum of the dimension of the space of matrices acting via left multiplication on $W_1$ plus the sum of the dimension of the space of matrices acting via left multiplication on $W_2$ plus the sum of the dimension of the space of matrices acting via left multiplication on $W_3$.
Notice dimension of the space the dimesion of the space of matrices acting via left multiplication on $W_1$ is just 1 since $dim W_1 =1$ and similarity for $dim W_2= 1$.
Now the dimension of the family of commuting operators on a 2-dimensional vectors space over $C$ is at most 2. Indeed each of the commuting operators on $W_2$ will have a repeated eigenvalue.
Therefore for the case $ p_A(x) = (x-c_1)(x-c_2)(x-c_3)^2$ we see that $\dim(F) \leq 4$.
I am a little unsure about the case $p_A(x) = (x-c_1)^{r_1}(x-c_2)^{r_2}$ where $r_1 = 1,2,3$ and $r_2 = 3,2,1$. In particular I am not sure if the above argument can deal with the case $r_1 =1 , r_2=3$ or $r_1=3, r_2=1$. If I apply the same arguments to the above as above can I make the following claim when i am studying the dimension of the space of matrices restricted to $W_1$ and if $\dim(W_3) =3$ can I still make the following statement as I did above in the two dimension case:
Question: Is the dimension of the family of commuting operators on a 3-dimensional vector space over $\mathbb{C}$ at most 3? Will each of the commuting operators on $W_2$ have a double or triple repeated eigenvalue?
Best Answer
This is too long to be a comment, so I make it an answer instead. Not a solution, just remarks.
I think that you are on the right track in the sense that this approach will lead to a proof. The observation that the other matrices $B$ need to keep the subspaces $W_i,i=1,2,3$ invariant is the key, and you can do the rest case by case. However, if I were grading your exam I would want you to justify the following claim that you made:
Ok. Probably this was meant to be $W_3$, i.e. the 2-dimensional characteristic space of $A$. Your first claim is correct in that a commuting family of operators on $W_3$ has dimension at most $2$, but you haven't proven it yet! It may be a lemma or an example in your textbook, in which case you are ok. But to be on the safe side, I want you to consider the case that $A$ acts as identity operator (or zero - any scalar will do!) on the subspace on $W_3$. Then $A$ will commute with all the operators on $W_2$, and those form a 4-dimensional space. Granted, they don't commute with each other, and in the end the maximum dimension will be 2, but you haven't shown it yet.
My suspicion was raised by your second claim that each operator on $W_2$ will have a repeated eigenvalue. This is most certainly false! For example, the space $F$ could consists of the diagonal matrices, but you just happened to pick as $A$ the diagonal matrix with entries $1,2,3,3$. Then $W_3$ consists of the span of the last two basis vectors, but the diagonal matrix $B$ with entries $2,3,5,8$ does not have a repeated characteristic value on $W_3$.
Then to a hint: There is no way to guarantee that you could pick $A$ in the right way, so another matrix $B$ may have several eigenvalues on a characteristic space of $A$. But, then the relation $AB=BA$ means that $A$ has to keep those eigenspaces of $B$ in $W_3$ invariant! In other words $A$ has to be a scalar on $W_2$, whenever such a $B$ exists in $F$. But we can use $B$ to split the space $W_3$ further into a direct sum of smaller components. By the preceding arguments, all the matrices in $F$ then need to respect this finer direct sum decomposition, and you are back to the case of simultaneously diagonalizable set.
[Edit: Clarifying the hint] So I would begin by showing that the 4-d space can be split into a direct sum of subspaces $\oplus\sum_iW_i$ in such a way that every element of $F$ keeps each summand invariant, and has only a single characteristic value on any component.]
But another trick is needed, when $A$ is not diagonalizable, and (consequently) no $B\in F$ has several characteristic values in a characteristic space of $A$. Hint: Any $B\in F$ must map the null space of $(A-\lambda I)^k$ to itself. [Edit: In view of the general result cited by Pierre-Yves this requires careful work in the case that the multiplicity of the characteristic value is three, as my hint only shows that all the matrices of $F$ are upper triangular w.r.t. to the basis that puts $A$ into its Jordan form. That set of upper triangular 3x3 matrices with identical entries along the diagonal forms a 4-d space. But is not commutative!]