Here $Z$ is an $n \times n$ real matrix. First we can use SVD such that
$$ Z = U \Sigma V' $$
where $\Sigma = \mbox{diag} (\sigma_1, \sigma_2,\cdots, \sigma_n)$ and $\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_n \geq 0$. Then we, have
$$ \mbox{Tr}(RZ) = \mbox{Tr}(RU\Sigma V') = \mbox{Tr}((V'RU)\Sigma) $$
Now the problem becomes maximizing $\mbox{Tr}(O\Sigma)$, where
$V'RU = O \in SO(n)$ or $O \in O(n)-SO(n)$. The first case is easy, just take $O=E$. I guess in the second case $O$ should be $\mbox{diag} (1,1,\dots,1,-1)$, how to prove this?
Best Answer
Here's a proof based off of von Neumann Trace Inequality and polar decomposition
$Z= UP$ by Polar decomposition where $P$ is real symmetric positive semidefinite
$P = M\Sigma M^T$ for $M\in SO_n$ and I assume we always have the ordering $\sigma_1\geq \sigma_2 \geq ...\geq \sigma_n$
(i.) $\det\big(Z\big) \gt 0$
for the relaxed problem involving $Q\in O_n(\mathbb R)$
$\text{trace}\big(QZ\big)=\text{trace}\big((QU)P\big)\leq \sum_{k=1}^n \sigma^{(QU)}_k\cdot \sigma^{(P)}_k=\sum_{k=1}^n 1\cdot \sigma^{(P)}_k= \text{trace}\big(P\big)$
by von Neumann Trace Inequality and setting $Q:=U^T$ meets the upper bound with equality and thus gives the max for OP's problem since $\det\big(Z\big)\gt 0\implies U \in SO_n(\mathbb R)$
(ii.) $\det\big(Z\big) = 0$
The argument is essentially the same except polar decompositon is not unique. However we can assume WLOG that $U \in SO_n(\mathbb R)$ per the argument here How to show that the unitary matrix in a polar value decomposition $A=UP$ is unique if and only if the matrix $A$ is invertible? so again setting $Q:= U^T$ meets the upper bound of von Neumann Trace Inequality with equality and hence is maximal.
(iii.) $\det\big(Z\big) \lt 0$
consider the relaxed problem of maximizing
$\text{trace}\big(VP\big) + \text{trace}\big(P\big)=\text{trace}\big((V+I)P \big)$
with the constraint that $V\in O_n(\mathbb R)$ and $V$ has at least one eigenvalue of $-1$
$\text{trace}\big((V+I)P \big)\leq \sum_{k=1}^n \sigma^{(V+I)}_k\cdot \sigma^{(P)}_k \leq \sum_{k=1}^{n-1} 2\cdot \sigma^{(P)}_k $
by von Neumann Trace Inequality and then considering the case of $\big(V+I\big)$ having $n-1$ eigenvalues of $2$ and a single zero. Put differently $\big(V+I\big)$ has eigenvalues of $\lambda_k^{(V)} + 1$ and the singular values are the modulus of said eigenvalues (since $V+I$ is normal) so $\sigma_k^{(V+I)} = \vert \lambda_k^{(V)}+1\vert\leq \vert \lambda_k^{(V)}\vert +\vert 1\vert =2$ and since $V$ has an eigenvalue of $-1$ we know $\det\big(V+I\big )=0$ so $\sigma_n^{(V+I)}=0$.
The upper bound on the right is met with equality when $V:=M \begin{bmatrix} I_{n-1} & \mathbf 0 \\ \mathbf 0&-1 \end{bmatrix}M^T$. Since $P$ is fixed we may subtract $\text{trace}\big(P \big)$ from each side to get
$\text{trace}\big(VP \big)\leq \big(\sum_{k=1}^{n-1} 1\cdot \sigma^{(P)}_k\big) -\sigma^{(P)}_n $
For OP's Problem of maximizing
$\text{trace}\big(QZ\big)=\text{trace}\big((QU)P\big)$ then $(QU) \in O_n(\mathbb R)-SO_n(\mathbb R)$ so it has at least one eigenvalue of $-1$ hence the upper bound of $\big(\sum_{k=1}^{n-1} 1\cdot \sigma^{(P)}_k\big) -\sigma^{(P)}_n$ is met with equality when $(QU) = M \begin{bmatrix} I_{n-1} & \mathbf 0 \\ \mathbf 0&-1 \end{bmatrix}M^T$ i.e. when
$Q := M \begin{bmatrix} I_{n-1} & \mathbf 0 \\ \mathbf 0&-1 \end{bmatrix}M^TU^T$