Your approach is OK. As you said, the columns of $A$, denote them by $C^A_1,...,C^A_n$, form an orthonormal basis of $F^n$ w.r.t the standard inner product. Denote $C^A_j=(a_{1,j},...,a_{n,j})$.
Since $A$ is upper-triangular, $a_{k,1}=0$ for all $k>1$. Since $A$ is unitary, $a_{1,1}\neq0$. Now compute the inner-product $\langle C^A_1,C^A_j\rangle$ for all $j>1$: you get:
$$0=\langle C^A_1,C^A_j\rangle=a_{1,1}\cdot a_{1,j}+0+...+0$$
Since $a_{1,1}\neq0$, we have $a_{1,j}=0$ for all $j>1$. (this shows that the first row is $(a_{1,1}\hspace{2pt}0\hspace{2pt}...\hspace{2pt}0)$)
Now use induction on columns.
As @Julien pointed out, every square matrix admits a $PLU$ decomposition, where $P$ is a permutation matrix. We have: $A = P \cdot L \cdot U$, such that:
$A=\begin{bmatrix}1 & 2 & 3 & 4 \\5 & 6 & 7 & 8\\1 & -1 & 2 & 3 \\2 & 1 & 1 &2 \end{bmatrix}= \begin{bmatrix} 1 & 0 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\\0 & 1 & 0 & 0 \end{bmatrix} \cdot \begin{bmatrix} 1 & 0 & 0 & 0\\1 & 1 & 0 & 0\\2 & 1 & 1 & 0\\5 & \dfrac{4}{3} & \dfrac{5}{3} & 1 \end{bmatrix} \cdot \begin{bmatrix} 1 & 2 & 3 & 4\\0 & -3 & -1 & -1\\0 & 0 & -4 & -5\\0 & 0 & 0 & -\dfrac{7}{3} \end{bmatrix}$
You could try manually cranking this one to find its $LU$ factorization. We want:
$L \cdot U = \begin{bmatrix} 1 & 0 & 0 & 0\\l_{21} & 1 & 0 & 0\\l_{31} & l_{32} & 1 & 0\\l_{41} & l_{42} & l_{43} & 1 \end{bmatrix} \cdot \begin{bmatrix} u_{11} & u_{12} & u_{13} & u_{14}\\0 & u_{22} &u_{23} & u_{24}\\0 & 0 & u_{33} & u_{34}\\0 & 0 & 0 & u_{44} \end{bmatrix} = \begin{bmatrix}1 & 2 & 3 & 4 \\5 & 6 & 7 & 8\\1 & -1 & 2 & 3 \\2 & 1 & 1 &2 \end{bmatrix}$
We start off by solving the first row, so we get:
$$u_{11} = 1, u_{12} = 2, u_{13} = 3, u_{14} = 4$$
The portion of the multiplication that determines the remaining entries in the first column of $A$ yields:
$$l_{21}u_{11} = 5 \rightarrow l_{21} = 5$$
$$l_{31}u_{11} = 1 \rightarrow l_{31} = 1$$
$$l_{11}u_{11} = 2 \rightarrow l_{41} = 2$$
At this point rewrite all the variables you solved for and then continue the process and see if you can solve the remaining variables. Of course it is easy to check the result if you can solve all of the equations.
So, we currently have:
$L \cdot U = \begin{bmatrix} 1 & 0 & 0 & 0\\5 & 1 & 0 & 0\\1 & l_{32} & 1 & 0\\2 & l_{42} & l_{43} & 1 \end{bmatrix} \cdot \begin{bmatrix} 1 & 2 & 3 & 4\\0 & u_{22} &u_{23} & u_{24}\\0 & 0 & u_{33} & u_{34}\\0 & 0 & 0 & u_{44} \end{bmatrix} = \begin{bmatrix}1 & 2 & 3 & 4 \\5 & 6 & 7 & 8\\1 & -1 & 2 & 3 \\2 & 1 & 1 &2 \end{bmatrix}$
Try solving for $u_{22},u_{23}, u_{24}$, and then $l_{32}, l_{42}$ and continue this process.
Best Answer
I will assume that $*$ stands for matrix multiplication.
If $A.B^t$ is a diagonal matrix $D$, then $B^t=A^{-1}.D$. But both $A^{-1}$ and $D$ are upper triangular and therefore their product is upper triangular too. In other words, $B^t$ is upper triangular. But $B^t$ is lower triangular. So, $B$ is diagonal, and so $A(=B^{-1}.D)$ is diagonal too.