Chandru1: Here is a hint to get you started: If the equation $(I-(AB-BA))^n=0$ holds, then for any $v$, if $w=(I-(AB-BA))^{n-1}v$, then $(I-(AB-BA))w=0$, or $(AB-BA)w=w$. If $w=0$ for all $v$, then $(I-(AB-BA))^{n-1}=0$. Proceeding inductively this way, you get that either $AB-BA=I$, or else 1 is an eigenvalue of $AB-BA$.
Now, the first option is impossible, because the traces of $AB$ and $BA$ coincide, but the trace of $I$ is $n$ rather than 0.
I don't know whether you are dealing with $2\times 2$ matrices or general $n \times $n$ matrices. The result is true in either case.
It may not be clear to you what these spaces are. Define addition of matrices by adding corresponding entries. So for example
$$\begin{bmatrix}
1 & 2\\
0 & 3
\end{bmatrix}
+
\begin{bmatrix}
5 & 3\\
0 & 1
\end{bmatrix}
=
\begin{bmatrix}
6 & 5\\
0 & 4
\end{bmatrix}
$$
If $c$ is a constant (a scalar, a number) then you multiply a matrix by $c$ by multiplying each entry by $c$. So for example
$$3\begin{bmatrix}
1 & 2\\
0 & 3
\end{bmatrix}
=
\begin{bmatrix}
3 & 6\\
0 & 9
\end{bmatrix}
$$
A vector space of matrices is a collection $V$ of matrices (of the same size) such that if $A$ and $B$ are matrices in the collection, then so is the sum $A+B$, and also if $c$ is any scalar, then $cA$ is in the collection.
So typically a vector space of matrices will have many matrices in it. The only vector space of matrices that consists of a single matrix is the space whose only element is the all $0$'s matrix.
In particular, the identity matrix by itself ($1$'s down the main diagonal, $0$'s elsewhere) is not a subspace of the collection of $2\times 2$ matrices, for if the identity matrix $I$ is in the subspace, then $cI$ has to be in the subspace for all numbers $c$. The collection of all matrices which are $0$ off diagonal, and have all diagonal terms equal is a subspace of the space of all matrices. Maybe that will take care of part of your objection.
Let $V$ be any vector space, and take a collection $U$ of some of the elements of $V$. Then $U$ is called a subspace of $V$ if $U$ by itself is a vector space, meaning that the sum of any two elements of $U$ is in $U$, and any constant times an element of $U$ is in $U$.
You quoted something to the effect that a certain $D$ is a subspace of the space of upper triangular matrices. That's not true. The collection of all matrices of the shape you described, with everything off diagonal equal to $0$, is a subspace. So $D$ is supposed to be not a single matrix, it is a largish collection of matrices.
Now let's look at your particular problem. Let $V$ be the collection of all upper triangular matrices. Is this a vector space? Take any two upper triangular matrices $A$ and $B$. Is $A+B$ upper triangular? Yes. If $c$ is a constant, and $A$ is upper triangular, is $cA$ upper triangular? Yes. So $V$ is a vector space.
Let $D$ be the collection of all diagonal matrices? Is this a vector space? Yes, the sum of two diagonal matrices is diagonal, a constant times a diagonal matrix is a diagonal matrix. $D$ is a subspace of the upper triangular matrices, because any diagonal matrix is in particular upper triangular, it is a special upper triangular matrix.
Best Answer
Hint: Suppose the diagonal elements are $\{a_1,a_2,\ldots,a_n\}$. The determinant of $A$ is $a_1a_2\dotsm a_n$. So the determinant of $A^k$ is $a_1^ka_2^k\dotsm a_n^k$. But the determinant of $I$ is $1$, so $a_1^ka_2^k\dotsm a_n^k=1$.