Sorry for the lengthy post, there is a summary at the bottom.
For example, let's say we have two matrices:
$$A = \pmatrix{1 & 2\\ 3 & 4\\ 5 & 6}, \quad B = \pmatrix{4 & -2 & 1\\0 & 2 & 3}$$
Apologies for my lack of formatting skills, to clarify: $A$ is a $3\times2$ matrix where the first row has the two elements $1$ and $2$. $B$ is a $2 \times 3$ matrix where the first row has the three elements $4$, $-2$, and $1$.
The way I think about multiplying two matrices is: take the dot product of the first row of the first matrix with every column of the second matrix. The result of each is an element in the first row of the resulting matrix. Once that's done, take the dot product of the second row of the first matrix with every column of the second matrix. These will make up the second row of the resulting matrix. Do this until you've gone through every row in the first matrix.
A rule I learned is that you can know the size of the resulting matrix by writing out the sizes of the two matrices being multiplied. In this case, that would look like: $3\times2$ and $2\times3$. The outside numbers ($3$ and $3$, when written that way) will be the size of the product.
But what if you wrote the sizes in the other order, i.e. $2\times3$ and $3\times2$? That would be saying that the resulting matrix is $2\times2$. And indeed, depending on how I look at the question, I can get both a $2\times2$ and $3\times3$ matrix as an answer.
Going back to my strategy for multiplication, if I think of taking the dot product of the rows of $A$ with the columns of $B$, then it's easy to see that I'll get a $3\times3$ matrix. However, if I think about taking the dot product of the rows of $B$ with the column of $A$, I get a $2\times2$ matrix.
Which is the right answer? Or is it that the order matters, i.e. if I'm looking for the result of $AB$ then I take the dot products of $A$'s rows and $B$'s columns, and if I'm looking for the result of $BA$ I take the dot products of $B$'s rows and $A$'s columns?
Or am I hugely misunderstanding something?
TLDR When I multiply the rows of a $3\times2$ matrix with the columns of a $2\times3$ matrix, I get a $3\times3$ matrix as the answer. But when I look at it the other way and multiply the rows of the $2\times3$ matrix with the columns of the $3\times2$ matrix, I get a $2\times2$ matrix as the answer. Which is correct?
I've been sitting here for hours trying to figure this out. Any help whatsoever is greatly appreciated!
Best Answer
Matrix multiplication is not commutative, therefore $AB \ne BA$ in general (note that they can be the same for some specific cases).
Which one is correct depends on what you are looking for and in which concept you are multiplying those matrices (for example in the case of basis change or some other applications, these two multiplications are completely different). In your example they are totally different and whichever is asked you to find, you should use that one.