Here $U(n)$ is the unitary group, consisting of all matrix $A \in M_n (\mathbb{C})$ such that $AA^*=I$
Problem How to calculate the integer cohomology group $H^*(U(n))$ of $U(n)$? What if $O(n)$ replace $U(n)$?
My primitive idea is that: as for $U(n)$, it is a Lie group and can
acts transitively on the unit sphere with a isotropy group (or say
stabilizer) $U(n-1)$. Hence, $U(n)/U(n-1) = S^{2n-1}$ and we have a
fibre bundle $$ \pi : U(n) \to S^{2n-1}$$ with fibre $U(n-1)$. Then,
we try to use spectral sequence and Leray's theorem to calculate.
Although it seems that we should calculate it by induction, I don't know how to continue. Moreover, someone could use any other method as long as it work.
Best Answer
I'll expand on the hint given by Robert. $U(1)$ is just $S^1 \subset \mathbb{C}$, and its cohomology ring is the exterior algebra $\Lambda[c_1]$. Here, the subscript of the generator indicates its degree.
For $n = 2$, we have, as you described, a fibre bundle $U(1) \to U(2) \to S^3$. Thus there is a Serre spectral sequence with \begin{equation} E_2^{p,q} = H^p(S^3,\mathcal{H}^q U(1)) \cong H^p(S^3) \otimes H^q(U(1)) \Rightarrow H^{p+q}(U(2)). \end{equation}
We have $E_2^{p,q} \cong \Lambda[c_1,c_3]$. By lacunary reasons, this spectral sequence collapses on the second page, and so we deduce $H^*(U(2)) \cong \Lambda[c_1,c_3]$.
In general, the spectral sequence for the fiber bundle $U(n-1) \to U(n) \to S^{2n-1}$ always collapses on the second page, and you can use induction to prove the proposition $H^*(U(n)) \cong \Lambda[c_1,c_3,\ldots,c_{2n-1}]$.
I like Mimura and Toda's The topology of Lie groups as a reference for these types of calculations. If I recall, they also discuss how to obtain the above proposition using Morse theory (or maybe just a rational decomposition of $U(n)$).
You also asked about $O(n)$, which is trickier. First, $O(n)$ is not path-connected, but has two path components each homeomorphic to $SO(n)$, so we may as well consider $SO(n)$ instead. There is a section in Hatcher (3.D) that is dedicated to computing the mod 2 cohomology of $SO(n)$, using a cell decopmosition.
You can also compute the mod 2 cohomology of $O(n)$ and $SO(n)$ using spectral sequences, but the problem here is that the analogous spectral sequences need not collapse, so the analysis is more subtle. One can instead induct on Stiefel manifolds, which generalize $O(n)$. I learnt how to do this from May et al's notes on characteristic classes. In particular, see Theorem 2.4 for the cohomology groups of $SO(n)$, although the ring structure requires consideration of Steenrod operations. May also discusses cohomology away from the prime 2.