First note that
$$\int_{-\infty}^{\infty} \frac{1-\cos ax}{x^2} \; dx
= \left[ -\frac{1-\cos ax}{x}\right]_{-\infty}^{\infty} + a \int_{-\infty}^{\infty} \frac{\sin ax}{x} \; dx
= \pi \, |a|,$$
by the Dirichlet integral. Also, by mathematical induction we can easily prove that
$$ \prod_{k=1}^{n} \cos \theta_k = \frac{1}{2^n} \sum_{\mathrm{e}\in S} \cos\left( e_1 \theta_1 + \cdots + e_n \theta_n \right),$$
where the summation runs over the set $S = \{ -1, 1\}^n$. Thus we have
$$ \begin{align*}
I_n = \int_{-\infty}^{\infty} \frac{1-\cos x \cdots \cos nx}{x^2} \; dx
&= \frac{1}{2^n} \sum_{\mathrm{e}\in S} \int_{-\infty}^{\infty} \frac{1-\cos(e_1 x + \cdots + e_n nx)}{x^2} \; dx \\
&= \frac{\pi}{2^n} \sum_{\mathrm{e}\in S} \left|e_1 + \cdots + e_n n\right|.
\end{align*}$$
For example, if $n = 3$, we have $\left|\pm 1 \pm 2 \pm 3\right| = 0, 0, 2, 2, 4, 4, 6, 6$ and hence
$$I_3 = \frac{\pi}{8}(0 + 0 + 2 + 2 + 4 + 4 + 6 + 6) = 3\pi.$$
Let the summation part as
$$ A_n = \sum_{\mathrm{e}\in S} |e_1 + \cdots + e_n n|.$$
The first 10 terms of $(A_n)$ are given by
$$ \left(A_n\right) = (2, 8, 24, 72, 196, 500, 1232, 2968, 7016, 16280, \cdots ), $$
and thus the corresponding $(I_n)$ are given by
$$ \left(I_n\right) = \left( \pi ,2 \pi ,3 \pi ,\frac{9 \pi }{2},\frac{49 \pi }{8},\frac{125 \pi }{16},\frac{77 \pi }{8},\frac{371 \pi }{32},\frac{877 \pi
}{64},\frac{2035 \pi }{128} \right).$$
So far, I was unable to find a simple formula for $(A_n)$, and I guess that it is not easy to find such one.
p.s. The probability distribution of $S_n = e_1 + \cdots + e_n n$ is bell-shaped, and fits quite well with the corresponding normal distribution $X_n \sim N(0, \mathbb{V}(S_n))$. Thus it is not bad to conjecture that
$$ \frac{A_n}{2^n} = \mathbb{E}|S_n| \approx \mathbb{E}|X_n| = \sqrt{\frac{n(n+1)(2n+1)}{3\pi}},$$
and hence
$$ I_n \approx \sqrt{\frac{\pi \, n(n+1)(2n+1)}{3}}.$$
Indeed, numerical experiment shows that
I was able to prove a much weaker statement:
$$ \lim_{n\to\infty} \frac{I_n}{n^{3/2}} = \sqrt{\frac{2\pi}{3}}. $$
First, we observe that for $|x| \leq 1$ we have
$$ \log \cos x = -\frac{x^2}{2} + O\left(x^4\right).$$
Thus in particular,
$$
\sum_{k=1}^{n} \log\cos\left(\frac{kx}{n}\right)
= \sum_{k=1}^{n}\left[-\frac{k^2 x^2}{2n^2} + O\left(\frac{k^4x^4}{n^4}\right)\right]
= -\frac{nx^2}{6} + O\left(x^2 \vee nx^4\right).$$
Now let
$$ \begin{align*}\frac{1}{n^{3/2}} \int_{-\infty}^{\infty} \frac{1 - \prod_{k=1}^{n}\cos (kx)}{x^2} \; dx
&= \frac{1}{\sqrt{n}} \int_{-\infty}^{\infty} \frac{1 - \prod_{k=1}^{n}\cos \left(\frac{kx}{n}\right)}{x^2} \; dx \qquad (nx \mapsto x) \\
&= \frac{1}{\sqrt{n}} \int_{|x|\leq 1} + \frac{1}{\sqrt{n}} \int_{|x| > 1}
=: J_n + K_n.
\end{align*}$$
For $K_n$, we have
$$ \left|K_n\right| \leq \frac{1}{\sqrt{n}} \int_{1}^{\infty} \frac{2}{x^2}\;dx = O\left(\frac{1}{\sqrt{n}}\right).$$
For $J_n$, the substitution $\sqrt{n} x \mapsto y$ gives
$$ \begin{align*}
J_n
&= \frac{1}{\sqrt{n}} \int_{|x|\leq 1} \left( 1 - \exp\left( -\frac{nx^2}{6} + O\left(x^2 \vee nx^4\right) \right) \right) \; \frac{dx}{x^2} \\
&= \int_{|y|\leq\sqrt{n}} \left( 1 - \exp\left( -\frac{y^2}{6} + O\left(\frac{y^2}{n}\right) \right) \right) \; \frac{dy}{y^2} \\
&\xrightarrow[]{n\to\infty} \int_{-\infty}^{\infty} \frac{1 - e^{-y^2/6}}{y^2} \; dy \\
&= \left[-\frac{1-e^{-y^2/6}}{y}\right]_{-\infty}^{\infty} + \frac{1}{3} \int_{-\infty}^{\infty} e^{-y^2/6} \; dy
= \sqrt{\frac{2\pi}{3}}.
\end{align*}$$
This completes the proof.
By letting $t=x/2$, we have that
$$\int_{1/n}^{n}\cos(x/2)\frac{\ln (x)}{x}dx=\int_{1/(2n)}^{n/2}\cos(t)\frac{\ln(t) +\ln(2)}{t}dt$$
Hence
$$\begin{align}\int_{1/n}^{n}\left(\cos x-\cos(x/2)\right)\frac{\ln(x)}{x}\,dx&=
\int_{n/2}^{n}\frac{\cos(x)}{x}\ln(x)dx-\int_{1/(2n)}^{1/n}\frac{\cos(x)}{x}\ln(x)\,dx \\
&\qquad-\ln(2)\int_{1/(2n)}^{n/2}\frac{\cos(x)}{x}\,dx.\end{align}$$
Now show that
$$\begin{align}
&\lim_{n\to \infty}\int_{n/2}^{n}\frac{\cos(x)}{x}\ln(x)\,dx=0,\\
&\lim_{n\to \infty}\int_{1/(2n)}^{1/n}\frac{\cos(x)-1}{x}\ln(x)\,dx=0.
\end{align}$$
Morever
$$\lim_{n\to \infty}\left(\int_{1/(2n)}^{n/2}\frac{\cos(x)}{x}\,dx-\ln(n)\right)
=\lim_{n\to \infty} (-\text{Ci}(\frac{1}{2n})-\ln(n))=\ln(2)-\gamma.$$
where $\text{Ci}(x)$ is the cosine integral (recall that
$\text{Ci}(x)=\ln(x)+\gamma+o(1)$, as $x\to 0$).
Therefore
$$\int_{1/(2n)}^{n/2}\frac{\cos(x)}{x}dx=\ln(n)+\ln(2)-\gamma+o(1),$$
and
$$\int_{1/(2n)}^{1/n}\frac{\cos(x)}{x}\ln(x)\,dx
=-\ln(2)\ln(n)-\frac{\ln^2(2)}{2}+o(1).$$
Thus we may conclude that
$$\lim_{n \to \infty}\int_{1/n}^{n}\left(\cos x-\cos(x/2)\right)\frac{\ln (x)}{x} \, dx=\boxed{\gamma \ln (2)-\frac{\ln^2(2)}{2}}.$$
Best Answer
Let $u = e^{ix}$. Then $\cos x \cos 2x ... \cos nx = \frac{1}{2^n}\prod_{k=1}^n (u^k + u^{-k})$.
Write $$\prod_{k=1}^n (z^k + z^{-k}) = \sum_{j=-N}^N a_j z^j$$
where $N=\frac{n(n+1)}2$.
Then the integral above is just $\frac{2\pi}{2^n}a_0$.
Combinatorially, we can see that $a_0$ is the number of ways of picking a subset of $\{1,2,...,n\}$ which adds up to $\frac{n(n+1)}{4}$.
Now, the set of subsets of $\{1,...,n\}$ that add up to a particular value is an antichain, so we have a bound on the size of $a_0$, namely, $$0\leq a_0 \leq \binom {n}{\lfloor n/2\rfloor}$$.
And we know that $$\lim_{n\to\infty}\frac{1}{2^n}\binom {n}{\lfloor n/2\rfloor} =0$$
So we know your limit is zero.
(The maximum size of an antichain in the Boolean poset is Sperner's theorem.)
Note that when $n\equiv 1,2\pmod 4$, $\frac{n(n+1)}4$ is not an integer, so in those cases, $a_0=0$. Anybody have an "interesting" lower bound on $a_0$ for the other cases?
[In comments below, Robert Israel shows that when $n\equiv 0,3\pmod 4$, $a_0\geq 2^{\lfloor (n+1)/4\rfloor}$. I suspect that there is a polynomial, $q$, such that $a_0\geq \frac{2^n}{q(n)}$.]
This proof generalizes. Given any sequence of positive integers, $\{b_1,...,b_n\}$, we have the same result:
$$\int_{0}^{2\pi} \cos b_1 x \cos b_2 x ... \cos b_n x dx= \frac{2\pi}{2^n}A$$
where $A$ is the count of an antichain, so $0\leq A\leq \binom {n}{\lfloor n/2\rfloor}$
If $b_i=1$ for all $i$, then we get exact equality when $n$ is even, which is to say that:
$$\int_{0}^{2\pi} \cos^{2m} x dx = \frac{2\pi}{2^{2m}} \binom {2m}{m}$$