One remark first, the determinant is defined for square matrices, not for vectors. Maybe the problem here is you view $D$ as a vector, whereas what is meant is "the square matrix with the same diagonal as $U$ and zeros everywhere else".
The LU decomposition yields a lower triangular matrix $L$ and an upper triangular matrix $U$ with
$$A=LU$$
The the determinant of a product is always the product of the determinants, it's perfectly safe to write
$$\det A=\det L \det U$$
Now, the determinant of a triangular matrix is the product of it's diagonal elements, and $L$ has only ones in its diagonal, whereas the diagonal of $U$ may be called $D$, and
$$\det A=\det D$$
For example, with $A=\left(\begin{matrix}
1 & 0 & 1 \\
2 & 1 & 0 \\
1 & 2 & 2 \\
\end{matrix}\right)$, you get the factorization
$$A=\left(\begin{matrix}
1 & 0 & 0 \\
2 & 1 & 0 \\
1 & 2 & 1 \\
\end{matrix}\right)\cdot\left(\begin{matrix}
1 & 0 & 1 \\
0 & 1 & -2 \\
0 & 0 & 5 \\
\end{matrix}\right)$$
And of course
$$\det \left(\begin{matrix}
1 & 0 & 0 \\
2 & 1 & 0 \\
1 & 2 & 1 \\
\end{matrix}\right)=\det \left(\begin{matrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{matrix}\right)=1$$
$$\det \left(\begin{matrix}
1 & 0 & 1 \\
0 & 1 & -2 \\
0 & 0 & 5 \\
\end{matrix}\right)=\det \left(\begin{matrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 5 \\
\end{matrix}\right)=5$$
Hence $\det A=5$.
There is another $LU$ factorization, with ones in the diagonal of $U$ instead of $L$, but of course that does not change the answer, if $D$ is the diagonal of $L$.
There may be another concern: often, the $LU$ decomposition is written
$$PA=LU$$
Where $P$ is a permutation matrix. If happens when pivoting is used in de $LU$ decomposition.
Then $\det P \det A=\det L\det U$, but $\det P=\pm1$, since a permutation matrix is always orthogonal. Thus you must be careful with the sign of $\det A$. And $\det P$ is simply the sign of the permutation on which $P$ is based.
Best Answer
If $A$ is an $n\times n$ matrix, then $$ \det(-A)=(-1)^n\det A $$ because of the more general result you cite: $\det(cA)=c^n\det(A)$, with $c=-1$.
This follows from the fact that, denoting by $I$ the identity matrix, we have $cA=(cI)A$ and so Binet's theorem says that $\det(cA)=\det((cI)A)=\det(cI)\det(A)$. Now $cI$ is a diagonal matrix, so its determinant is the product of the entries along the diagonal, which are all equal to $c$ and there are $n$ of them. Thus $\det(cI)=c^n$.
In your case $n=5$, so you can say that $$ \det(-A^3B^TA^{-1})=-\det(A^3B^TA^{-1}) $$ By Binet's theorem and taking into account that $\det(B^T)=\det(B)$ and $\det(A^{-1})=\det(A)^{-1}$, \begin{align} \det(-A^3B^TA^{-1}) &=-\det(A^3B^TA^{-1})\\[4px] &=-\det(A^3)\det(B^T)\det(A^{-1})\\[4px] &=-\det(A)^3\det(B)\det(A)^{-1}\\[4px] &=-5^3\cdot\frac{1}{8}\cdot\frac{1}{5}=-\frac{25}{8} \end{align}
There is no “multiplication by $5$” to do.