Suppose we have a linear operator $T:\Bbb F^n -> \Bbb F^n$. If we fix the basis as the standard basis for both the domain and the co-domain then it turns out that the matrix of an orthogonal transformation is an orthogonal matrix.
Q-1 Suppose I change the basis to a non standard basis, that is, something other than the standard basis, does the above still hold? That is to say, is the matrix of an orthogonal transformation an orthogonal matrix?
Q-2 Let $T: V \to W$ be a linear transformation where $V$ and $W$ can be any vector spaces. Fix bases for the domain and the co-domain, say $B$ and $C$. Does the concept of an orthogonal transformation still exist in this case because we would require the concept of a dot product and that the dot product must be preserved. In the case of $\Bbb F^n$ the vectors are column vectors with entries in the corresponding field and so the dot product can be defined. But for an arbitrary vector space the vectors are not column vectors although we can talk about the dot product of the coordinate vectors. So does such a thing as an orthogonal transformation exist for arbitrary spaces? If yes then does that mean the dot product $(X,Y)$ is preserved where $X$ and $Y$ are coordinate vectors?
Q-3 This is a continuation of Q-2. $T:V \to W$ is a linear transformation. Fix bases $B$ and $C$ for $V$ and $W$ respectively. Is the matrix with respect to the chosen bases orthogonal if the transformation is orthogonal?
Best Answer
Q-1: I wasn't sure of the answer to your first question, so I did some searching around. Specifically, the group of $n \times n$ orthogonal matrices, denoted $O(n)$, does not appear to be a normal subgroup of the group of all invertible $n \times n$ matrices, denoted $GL_n(\mathbb{R})$.
What I mean by the above is that there exist orthogonal matrices $A$ and invertible matrices $S$ such that $SAS^{-1}$ is not orthogonal. Recall that the matrix of the linear tranformation given by $A$ under a different basis is given by $SAS^{-1}$ for some change-of-basis matrix $S$. Thus, the answer to your first question is no.
However, if we only use orthogonal change-of-basis matrices, i.e. we assume that the new basis is orthonormal, then the answer is yes. Observe, if $S$ is orthogonal, then so is $S^{-1}$, and we have $$(SAS^{-1})^T(SAS^{-1}) = (S^{-1})^TA^TS^TSAS^{-1} = (S^{-1})^TA^TAS^{-1} = (S^{-1})^TS^{-1} = I$$ (we have just used that $A^TA = I$ for orthogonal matrices $A$). This shows that $SAS^{-1}$ is still orthogonal, so changing to a different orthonormal basis preserves orthogonality of the linear transformation.
Q-2: We do not have a concept of orthogonality, including orthogonal transformations, unless our vector spaces are indeed inner product spaces. As you point out, there is always a way to impose an inner product on a given vector space $V$, namely by picking a basis, using that basis to construct an isomorphism to $\mathbb{R}^n$, and then taking the dot product in $\mathbb{R}^n$. This is an example of something we usually call "non-canonical," which roughly means there were choices involved in the definition of this inner product. Namely, we had to choose a basis for $V$, and there are many different ways to do this, yielding many different inner products. Therefore, we do not typically use this inner product. Rather, we would hope that $V$ comes with a more "natural" or "canonical" inner product to define orthogonal transformations between arbitrary spaces.
Q-3: Again, if $V$ and $W$ are inner product spaces, only then may we discuss orthogonality, so let's assume they do indeed have inner product structures. Then as we found above, while a transformation may be orthogonal, its matrix with respect to a particular basis need not be. Once again, we will need to assume our bases $\mathcal{B}$ and $\mathcal{C}$ for $V$ and $W$ resp. are orthonormal with respect to the inner product structures on $V$ and $W$ resp. Under these circumstances, I believe we can conclude that the matrix for the orthogonal transformation $T$ is an orthogonal matrix (although I have not proven this fact. You should try to find a proof, or try to write a proof yourself!)