Let $V$ be euclidean (hermitian) space. The set of vectors $\{a_1,a_2,\dots,a_k\}$ can be mapped under orthogonal operator to the set of vectors $\{b_1,b_2,\dots,b_k\}$ iff Gram matrices of each sets are equal, i.e. $$G(a_1,\dots,a_k)=G(b_1,\dots,b_k).$$
It seems to me quite a nice problem. Probably it should not be so difficult and let me show my attempts.
$\Rightarrow$ This is trivial because if the first set can be mapped to the second via orthogonal operator $f$ then $f(a_i)=b_i$ and $(b_i,b_j)=(f(a_i),f(a_j))=(a_i,a_j)$ the last inequality from definition of orthogonal operator $f$. It shows that corresponding elements of matrices $G_a$ and $G_b$ are equal which means that $G(a_1,\dots,a_k)=G(b_1,\dots,b_k).$
$\Leftarrow$ Suppose that $G(a_1,\dots,a_k)=G(b_1,\dots,b_k)$ and let $(e)=\{e_1,\dots,e_n\}$ be an orthonormal basis of $V$. Suppose $a_i$ has coordinates $(a^1_i,\dots,a^n_i)$ then $[G_a]_{ij}=(a_i,a_j)=\sum \limits_{k=1}^{n}a^k_ia^k_j=[A^TA]_{ij}$ which means that $G_a=A^TA$ and analogously $G_b=B^TB$ where $A$ and $B$ are $n\times k$ matrices, whose columns are coordinates of vectors $\{a_1,\dots,a_k\}$ and $\{b_1,\dots,b_k\}$, respectively. So we have that $A^TA=B^TB$.
My goal is to construct an operator $f:V\to V$ such that its matrix in the orthonormal basis $(e)$ is orthogonal matrix. Let's call this matrix $M_f^{(e)}=M$. As I said $M$ should be orthogonal and $MA=B$. I was trying to play with $M=BA^T$ or $M=AB^T$ but I failed.
So I would be very grateful if anyone can give useful idea or show the solution, please!
Best Answer
A partial answer for the case that the sets $\{a_1,\dots,a_k\}$ and $\{b_1,\dots,b_k\}$ are linearly independent (or equivalently, the Grammian matrices are invertible).
Suppose that $G(a_1,\dots,a_k) = G(b_1,\dots,b_k)$. Let $\{a_{k+1},\dots,a_n\}$ and $\{b_{k+1},\dots,b_n\}$ be orthonormal bases for $\{a_1,\dots,a_k\}^\perp$ and $\{b_1,\dots,b_k\}^\perp$. Verify that $G(a_1,\dots,a_n) = G(b_1,\dots,b_n)$.
Note that a linear map $f:V \to V$ is orthogonal if and only if $(f(x),f(y)) = (x,y)$ for all $x,y \in V$. Show that if we take $f$ to be the unique linear map satisfying $f(a_j) = b_j$ for $j=1,\dots,n$, then $f$ satisfies this property and is therefore orthogonal.
An extension of this solution to the general case:
Because $A^TA = B^TB$, we have $\ker A = \ker B$. It follows that a set of vectors $a_{j_1},\dots,a_{j_d}$ will be linearly indepnendent if and only if the corresponding set $b_{j_1},\dots,b_{j_d}$ is linearly independent.
With that in mind, we can select a set $a_{j_1},\dots,a_{j_d}$ that forms a basis of $\operatorname{span}(\{a_1,\dots,a_k\})$ (which has dimension $d$). The corresponding set $b_{j_1},\dots,b_{j_d}$ forms a basis for $\operatorname{span}(\{b_1,\dots,b_k\})$. As before, we select vectors $a_{d+1},\dots,a_{n}$ and $b_{d+1},\dots,b_n$ that form bases for the respective orthogonal complements of the spans.
Now, it suffices to define $f$ to be the linear map satisfying $f(a_{j_\ell}) = b_{j_\ell}$ for $\ell = 1,\dots,d$ and $f(a_\ell) = b_\ell$ for $\ell = d+1,\dots,n$.