I try to answer starting from the case of square matrices. There is some care to take while considering a "hidden" isomorphism of vector spaces. In any case, let $V$ be a finite dim. vector spaces over a field $\mathbb K$ (for simplicity $\mathbb R$ ), with basis $\{e_i\}$ of cardinality $n$.
It is well known that there exists an isomorphism of vector spaces
$$\Phi:\operatorname{Hom}_\mathbb K(V,V)\rightarrow V^{*}\otimes V, $$
with $$\Phi(\phi)=a_{ij}f_i\otimes e_j,$$
where $\phi\in \operatorname{Hom}_\mathbb K(V,V)$ and $\phi(e_i):=a_{ij}e_j$ for all $i,j=1,\dots,n$.
$\{f_i\}$ is the dual basis on $V^{*}$ of the basis $\{e_i\}$ on $V$, i.e. $f_i(e_j)=\delta_{ij}$.
We use the Einstein convention for repeated indices.
We know how to define the trace operator $\operatorname{Tr}$ on the space $\operatorname{Hom}_\mathbb K(V,V)$; the trace is computed on the square matrix representing each linear map in $\operatorname{Hom}_\mathbb K(V,V)$. Let us move to the r.h.s. of the isomorphism $\Phi$.
- trace operator on $V^{*}\otimes V$
Let
$$\operatorname{Tr}_1: V^{*}\otimes V\rightarrow \mathbb K, $$
be given by $\operatorname{Tr}_1(g\otimes v):=g(v)$.
Lemma $\operatorname{Tr}_1$ is linear and satisfies
$$\operatorname{Tr}_1\circ \Phi=\operatorname{Tr}.$$
proof: just use definitions.
- trace operator on $(V^{*}\otimes V)\otimes\dots\otimes (V^{*}\otimes V) $
Using the $n=1$ case we introduce
$$\operatorname{Tr}_n: \underbrace{(V^{*}\otimes V)\otimes\dots\otimes (V^{*}\otimes V)}_{n-\text{times}} \rightarrow \mathbb K, $$
with $\operatorname{Tr}_n(f_1\otimes v_1\otimes\dots\otimes f_n\otimes v_n):=\prod_{i=1}^n f_i(v_i)$.
Lemma $\operatorname{Tr}_n$ is linear and invariant under permutations on $(V^{*}\otimes V)^{\otimes n}$;
it satisfies
$$\operatorname{Tr}_n\left(\Phi(\phi_1)\otimes\dots\otimes\Phi(\phi_n)\right)=\prod_{i=1}^n \operatorname{Tr}(\phi_i), $$
for all $\phi_i\in \operatorname{Hom}_\mathbb K(V,V)$.
proof: we prove the second statement. We introduce the notation
$$\Phi(\phi_k):=
a^k_{i_kj_k}f_{i_k}\otimes e_{i_k}\in V^{*}\otimes V,$$
for all $k=1,\dots,n$.
We arrive at $$\operatorname{Tr}_n\left( (a^1_{i_1j_1}f_{i_1}\otimes e_{i_1})\otimes\dots\otimes
(a^n_{i_nj_n}f_{i_n}\otimes e_{i_n})\right)=a^1_{i_1j_1}\dots a^n_{i_nj_n}f_{i_1}(e_{i_1})\dots
f_{i_n}(e_{i_n})=\text{remember the definition of dual basis}=
a^1_{i_1j_1}\dots a^n_{i_nj_n}\delta_{i_1j_1}\dots\delta_{i_nj_n}=
a^1_{i_1i_1}\dots a^n_{i_ni_n}\\=\prod_{i=1}^n \operatorname{Tr}(\phi_i),$$
as claimed.
It helps to think of this as a partitioned matrix. We have
$$
A \otimes B =
\left[\begin{array}{cc|cc}
a_{11,11} & a_{11,12} & a_{11,21} & a_{11,22}\\
a_{12,11} & a_{12,12} & a_{12,21} & a_{12,22}\\
\hline
a_{21,11} & a_{21,12} & a_{21,21} & a_{21,22}\\
a_{22,11} & a_{22,12} & a_{22,21} & a_{22,22}
\end{array}\right]
$$
In the pair $ij$, $i$ specifies which block you're in, and $j$ specifies where you are in the block.
Now, we compute (for instance)
$$
b_{21} = a_{21,11} + a_{22,12}
$$
which you'll recognize as the trace of the lower-left block.
Best Answer
We have: $$ X_{ii}' = (RXR')_{ii} = (RX)_{ik}R_{ki}' = R_{ij}X_{jk}R_{ki}' = R_{ij}X_{jk}R_{ik} = R_{ij}R_{ik}X_{jk} $$ In the usual notation, we might write $$ \operatorname{Trace}(RXR') = \operatorname{Trace}[(RX)R'] = \operatorname{Trace}[R'(RX)] = \operatorname{Trace}[(R'R)X] = \operatorname{Trace}(X) $$