Is there a group $G$ with the property that $G$ is a smooth manifold, the multiplication map of $G$ is smooth, but the inversion map of $G$ is not smooth?
[Math] Group G hasn’t all conditions of Lie group
lie-groups
Related Solutions
I hate to throw cold water on the party, but surprisingly, the formula that the OP was trying to prove ($[X',Y']=[X,Y]'$) is actually false when $G$ acts on the left. This formula is correct if $G$ acts on the right on $M$; but if $G$ acts on the left, then the correct formula is $[X',Y']=-[X,Y]'$.
Here's how to prove it. Suppose first that $G$ acts smoothly on $M$ on the right. Fix $p\in M$, and consider the orbit map $\alpha^{(p)}: G\to M$ defined by $\alpha^{(p)}(g) = p\cdot g$. Then I claim that for each $X\in \mathfrak g$, the fundamental vector field $X'$ is $\alpha^{(p)}$-related to $X$. To see this, note that the group law $p\cdot gg' = (p\cdot g)\cdot g'$ translates to $$\alpha^{(p)}\circ L_g (g') = \alpha^{(p\cdot g)}(g')$$ (where $L_g$ is left multiplication by $g$). Let $g\in G$ be arbitrary and set $q=p\cdot g=\alpha^{(p)}(g)$. Because $X$ is left-invariant, $$ X'_q = d\bigl(\alpha^{(q)}\bigr)_1(X_1) = d\bigl(\alpha^{(p)}\bigr)_g\circ d(L_g)_1(X_1) = d\bigl(\alpha^{(p)}\bigr)_g(X_g)$$ (where the first equality is essentially the definition of $X'$, and the second follows from the previous equation by taking differentials at the identity). This proves the claim.
Because brackets of $\alpha^{(p)}$-related vector fields are themselves $\alpha^{(p)}$-related, it follows that $[X',Y']_p = ([X,Y]')_p$, and since this is true for every $p$ it is true globally.
Now if $G$ acts on $M$ on the left, the above argument doesn't work; if $\tilde\alpha^{(p)}$ denotes the orbit map for the left action, then $X'$ is not $\tilde\alpha^{(p)}$-related to $X$. Instead, we can create a right action by setting $p\cdot g = g^{-1}\cdot p$. Letting $\alpha^{(p)}$ denote the orbit map for this new right action, we have $\tilde\alpha^{(p)}=\alpha^{(p)}\circ\iota$, where $\iota: G\to G$ is inversion; and the argument above shows that $X$ and $X'$ are $\alpha^{(p)}$-related. Since $d\alpha^{(p)}$ preserves brackets and $d\iota$ reverses them, it follows that $d\tilde\alpha^{(p)}$ reverses brackets.
The problem with Eric's argument has to do with the identification between $\operatorname{Lie}(\operatorname{Diff}(M))$ and $\mathfrak X(M)$. Essentially the same argument as above shows that the natural map $\operatorname{Lie}(\operatorname{Diff}(M)) \to \mathfrak X(M)$ is actually a Lie algebra anti-isomorphism (if we consider $\operatorname{Diff}(M)$ as an infinite-dimensional Lie group acting on $M$ on the left).
The problem with Victor Protsak's argument is that it doesn't actually show that $(X,0)$ (considered as a vector field on $G\times M$) is $\alpha$-related to $X'$ -- to show this, you'd have to prove that $(d\alpha)_{(g,m)}(X,0)=X'$ for every $g$ in the group, not just $g=1$.
I had to sort all this out recently because I'm adding a section on infinitesimal group actions in the second edition of my book Introduction to Smooth Manifolds. When I tried to prove the same formula the OP was trying to prove, I was surprised to find that it led to contradictions. Eventually I came up with the argument I sketched here, and then found at least one other book that confirms the result. I'm at home now and don't have access to my books, but if you'd like I'll find the reference and post it tomorrow.
With field coefficients, homology and cohomology are dual to each other. The cohomology of a space is an algebra, the homology of a space is a coalgebra. When the space is a group (or loop space or $H$-space...) both its homology and cohomology are Hopf algebras.
"The literature" is full of computations of the cohomology of the classifying space $BG$. In some cases by the way, Hopf algebras help, for example if you compute with coefficients in $\mathbb{Q}$. Then graded Hopf algebras over the rational field are classified, which gives you a strong indication on what the cohomology of $G$ looks like, and a spectral sequence argument tells you that $H^*(BG, \mathbb{Q})$ is a polynomial ring, in the end. Details in McCleary's book A user's guide to spectral sequences.
Best Answer
Robert L. Bryant "An Introduction to Lie Groups and Symplectic Geometry" requires in the definition of a Lie group only that the multiplication map be smooth, and then proves that the inversion map must be smooth also. (Proposition 1, page 14.)