There was a recent question on intuitions about sheaf cohomology, and I answered in part by suggesting the "genetic" approach (how did cohomology in general arise?). For historical material specific to sheaf cohomology, what Houzel writes in the Kashiwara-Schapira book Sheaves on Manifolds for sheaf theory 1945-1958 should be adequate.
The question really is about the earlier period 1935-1938. According to nLab, cohomology with local coefficients was proposed by Reidemeister in 1938 (http://ncatlab.org/nlab/show/history+of+cohomology+with+local+coefficients). The other bookend comes from Massey's article in History of Topology edited by Ioan James, suggesting that from 1895 and the inception of homology, it took four decades for "dual homology groups" to get onto the serious agenda of topologists. It happens that 1935 was also the date of a big international topology conference in Stalin's Moscow, organised by Alexandrov. This might be taken as the moment at which cohomology was "up in the air".
Now de Rham's theorem is definitely somewhat earlier. Duality on manifolds is quite a bit earlier in a homology formulation.
It is apparently the case that At the Moscow conference of 1935 both Kolmogorov and Alexander announced the definition of cohomology, which they had discovered independently of one another. This is from http://www.math.purdue.edu/~gottlieb/Bibliography/53.pdf at p. 11, which then mentions the roles of Čech and Whitney in the next couple of years. This is fine as a narrative, as far as it goes. I have a few questions, though:
1) Is the axiomatic idea of cocycle as late as Eilenberg in the early 1940s?
2) What was the role of obstruction theory, which produces explicit cocycles?
Further, Weil has his own story. Present at the Moscow conference and in the USSR for a month or so after, his interest in cohomology was directed towards the integration of de Rham's approach into the theory. He comments in the notes to his works that he pretty much rebuffed Eilenberg's ideas. Bourbaki was going to write on "combinatorial topology" but the idea stalled (I suppose this is related). So I'd also like to understand better the following:
3) Should we be accepting the topologists' history of cohomology, if it means restricting attention to the "algebraic" theory, or should there be more differential topology as well as sheaf theory in the picture?
As said, restriction to a short period looks like a good idea to get some better grip on this chunk of history.
Best Answer
What strikes me about the first fifty years of homology theory (from Poincaré to Eilenberg-Steenrod's book) is that the development was as much about stripping away unnecessary complication as about increasing sophistication. A famous example is singular homology, which was found very late, by Eilenberg. The construction as we know it presumably seemed too naive to Lefschetz, who misguidedly devised a theory of oriented simplices, and inadequate to those who were interested in general (not locally path connected) metric spaces.
I want to suggest that this process of stripping away is relevant to the introduction of cohomology and its product. (Cf. Dieudonné's "History of algebraic and differential topology", pp.78-81). I won't directly answer the questions, but will suggest that one of the motivations for cohomology came from an application of Pontryagin duality which was rendered obsolete by the new theory.
Alexander wrote up his Moscow conference talk, with improvements suggested by Cech, in a 1936 Annals paper (vol. 37 no. 3) (JSTOR link). In it, he proposes the cohomology ring ("connectivity ring") as a fundamental homological invariant of a space. In the introduction he hints at the line of thought that led him to the cohomology ring. The relation between cycles and differential forms is mentioned (without citation of de Rham), but what looks more surprising to modern eyes is the comment that the theory of cycles "has been very greatly perfected by Pontrjagin's cycles with real coefficients reduced modulo 1".
Pontryagin had recently developed his duality theory for locally compact abelian groups (Annals, 1934) in order to apply it to Alexander duality (again, Annals, 1934). If $K$ is a compact polyhedral complex in $\mathbb{R}^n$, there is a linking form which gives a pairing between $k$-cycles of $K$ and $(n-k-1)$-cycles of $\mathbb{R}^n-K$ and, in modern terms, induces an isomorphism of $H_k(K)$ with $H^{n-k-1}(\mathbb{R}^n-K)$. Alexander's formulation equated the Betti numbers over a field (mod 2, initially - Dieudonné p. 57) of $K$ and its complement, but it was understood that the full homology groups of $K$ and $\mathbb{R}^n-K$ need not be isomorphic. Pontryagin showed that if one takes a Pontryagin-dual pair of metric abelian groups, say $\mathbb{Z}$ and $\mathbb{T}$, so that each is the character group of the other, then $H_k(K;\mathbb{T})$ is Pontryagin-dual to $H_{n-k-1}(\mathbb{R}^n-K;\mathbb{Z})$ via the linking form.
From Alexander's introduction:
Guided by Pontryagin's generalisation of his own duality theorem, Alexander finds a simple construction that supersedes Pontryagin's as a basic invariant. (The universal coefficient theorem gives a modern perspective on why Pontryagin's choice of coefficient groups works. I must admit, his formulation of duality is very clean.)
Can anyone comment on Kolmogorov's route to cohomology?
ADDED. On obstruction theory: Charles Matthews's comments draw attention to a 1940 paper of Eilenberg. The MathSciNet review of that paper (by Hurewicz, whose homotopy groups, useful for obstruction theory, date from 1935-36) points me to its 1937 forerunner by Whitney, "The maps of an $n$-complex into an $n$-sphere" (Duke M.J. 3 (no.1), 51-55). This work, too, was presented at the Moscow conference in 1935. Though the topic is different, Whitney's introduction closely resembles Alexander's: