Say the columns of $A$ are $A_1,\dots,A_n$.
If $B = (b_{ij}) = A^TA$, then:
$b_{ij} = \langle (A_i)^T,A_j\rangle$, and since these (the $A_k$) are orthonormal relative to our inner product, we have:
$b_{ij} = \delta_{ij}$ (the kronecker delta), which is $1$ when $i = j$, and $0$ otherwise, that is to say $B$ is the identity matrix.
So $A^TA = I$, from which we conclude $A^T = A^{-1}$ (If you insist on showing $A$ has a two-sided inverse, see below).
On the other hand, if $A^TA = I$, then (running our argument in reverse), we see the columns of $A$ (and thus the rows of $A^T$) form an orthonormal basis (they form a basis since $A$ is invertible).
By considering $AA^T = I$ in the same way, we see the columns of $A^T$, and thus the rows of $(A^T)^T = A$ also form an orthonormal basis.
Writing matrix A = $\begin{bmatrix}
{v_1} & {v_2} ... & {v_n} \\
\end{bmatrix}$ as built from column vectors and making computation $A^TA$ (which we know is equal $I$) gives result:
$\begin{bmatrix}
{v_1}^T\\
{v_2}^T\\
... \\
{v_n}^T\\
\end{bmatrix}\begin{bmatrix}
{v_1} & {v_2} & ...& {v_n}\\
\end{bmatrix}=\begin{bmatrix}
{v_1}^T{v_1} & {v_1}^T{v_2} & ...& {v_1}^T{v_n}\\
{v_2}^T{v_1} & {v_2}^T{v_2}& ...&{v_2}^T{v_n}\\
... & .... & ... &.... \\
{v_n}^T{v_1}& {v_n}^T{v_2} & ... & {v_n}^T{v_n}\\
\end{bmatrix}$ =$\begin{bmatrix}
1 & 0 & ... &0\\
0 & 1 & ... &0\\
... & .... & ...&.... \\
0 & 0 & ...& 1\\
\end{bmatrix}$
what gives wanted interpretation.
Best Answer
An orthogonal matrix may be defined as a square matrix the columns of which forms an orthonormal basis. There is no thing as an "orthonormal" matrix.
The terminology is a little confusing, but it is well established.