As commented by Jyrki Lahtonen, the statement is true and it is immediately implied by the following relation between the minors of $A^{-1}$ and the minors of $A$.
Proposition: If $A$ is an invertible $n\times n$ matrix, and if $i_1,\dots,i_n$ and $j_1,\dots,j_n$ be two permutations of $1,\dots,n$, then the minor of $A$ corresponding to rows $i_1,\dots,i_k$ and columns $j_1,\dots,j_k$, denoted by $d$, and the minor of $A^{-1}$ corresponding to rows $j_{k+1},\dots,j_n$ and columns $i_{k+1},\dots,i_n$, denoted by $d'$,satisfy that
$$d=\pm d'\det A.$$
Proof: Let $e_1,\dots,e_n$ be a basis of $\mathbb{R}^n$, and let $f_i= A e_i$, $i=1,\dots,n$. Then on the one hand,
$$\omega:=Ae_{j_1}\wedge\cdots\wedge Ae_{j_k}\wedge e_{i_{k+1}}\wedge \cdots\wedge e_{i_n}=\pm d\cdot e_1\wedge\cdots\wedge e_n,$$
on the other hand,
$$\omega=f_{j_1}\wedge\cdots\wedge f_{j_k}\wedge A^{-1}f_{i_{k+1}}\wedge \cdots\wedge A^{-1}f_{i_n}=\pm d'\cdot f_1\wedge\cdots\wedge f_n.$$
Since
$$f_1\wedge\cdots\wedge f_n=\det A\cdot e_1\wedge\cdots\wedge e_n,$$
the conclusion follows.
One of the various formulas for the determinant of an $n\times n$ matrix $A$ is
$$\det(A) = \sum_{\sigma \in S_n} \mathrm{sgn}(\sigma) \prod_{i=1}^n A_{i,\sigma_i} $$
where as usual $A_{i,j}$ denotes the entry of $A$ in the $i$th row and $j$th column, and $\mathrm{sgn}(\sigma)$ is either $+1$ or $-1$, depending on $\sigma$. Thus, if every $A_{i,j}$ is an integer, then $\det(A)$ is just a huge sum of products of integers, and is therefore an integer itself.
Best Answer
The answer is "yes" if $U$ is nonsingular and totally unimodular, but in general, the answer is "no", as shown by the random counterexample below: $$ \pmatrix{1&0&0\\ 1&1&0\\ -1&1&1}^{-1}=\pmatrix{1&0&0\\ -1&1&0\\ 2&-1&1}. $$