You are mistaking the inequality. Tao is saying that there is a constant $C > 0$ with $$|\frac{1}{N}\sum_{n \le N} a_n| \le C(\frac{1}{H}\sum_{h \le H} |\frac{1}{N}\sum_{n \le N} a_{n+h}\overline{a_n}|)^{1/2} + C\frac{H}{N}.$$
Actually, I think there should be a $C\frac{1}{H^{1/2}}$ on the right hand side as well (see the link I gave).
Based on where you got to, and using $\sqrt{x+y} \le \sqrt{x}+\sqrt{y}$, it suffices to show $$\frac{1}{N}\sum_{n \le N} |\frac{1}{H}\sum_{h \le H} a_{n+h}|^2 \le \frac{2}{H}\sum_{h \le H} |\frac{1}{N}\sum_{n \le N} a_{n+h}\overline{a_n}|+O(\frac{1}{H})+O(\frac{H}{N}).$$
Merely by expanding the square, the left hand side is $$\frac{1}{H^2}\sum_{h_1,h_2 \le H} \frac{1}{N}\sum_{n \le N} a_{n+h_1}\overline{a_{n+h_2}}.$$ The term $h_1 = h_2$ has contribution $$\frac{1}{H^2}\sum_{h \le H} \frac{1}{N}\sum_{n \le N} |a_{n+h}|^2 = O(\frac{1}{H}),$$ so we can ignore it. We are left with
\begin{align*}
\frac{2}{H^2}\sum_{1 \le h_1 < h_2 \le H} \frac{1}{N}\sum_{n \le N} a_{n+h_1}\overline{a_{n+h_2}} &= \frac{2}{H^2}\sum_{1 \le h_1 < h_2 \le H} \frac{1}{N}\sum_{n=h_1+1}^{N+h_1} a_n\overline{a_{n+h_2-h_1}} \\ &= \frac{2}{H^2}\sum_{1 \le h_1 < h_2 \le H} \left[\frac{1}{N}\sum_{n=1}^N a_n\overline{a_{n+h_2-h_1}} + O(\frac{H}{N})\right].
\end{align*}
The $O(\frac{H}{N})$ term merely comes out of the triple sum (since everything is averaged), so we can ignore it. We are left with $$\frac{2}{H^2} \sum_{1 \le h_1 < h_2 \le H} \frac{1}{N}\sum_{n=1}^N a_n\overline{a_{n+h_2-h_1}} = \frac{2}{H^2}\sum_{h \le H}\sum_{\substack{1 \le h_1 < h_2 \le H \\ h_2-h_2 = h}} \frac{1}{N}\sum_{n=1}^N a_n\overline{a_{n+h}}.$$
Now, we can pull out the sum in $n$ and note that the number of $1 \le h_1 < h_2 \le H$ with $h_2-h_1 = h$ is $H-h$ to get $$\frac{2}{H^2}\sum_{h \le H} (H-h)\frac{1}{N}\sum_{n=1}^N a_n\overline{a_{n+h}},$$ which is upper bounded by $$\frac{2}{H^2}\sum_{h \le H} (H-h)\left|\frac{1}{N}\sum_{n=1}^N a_n\overline{a_{n+h}}\right|,$$ which is of course upper bounded by $$\frac{2}{H}\sum_{h \le H} \left|\frac{1}{N}\sum_{n=1}^N a_n\overline{a_{n+h}}\right|,$$ as desired.
In regards to what this inequality is trying to capture. First, note that it holds in any Hilbert space, i.e. $||\frac{1}{N}\sum_{n \le N} v_n|| \le C(\frac{1}{H}\sum_{h \le H} ||\frac{1}{N}\sum_{n \le N} \langle v_{n+h},v_n \rangle||)^{1/2}+C\frac{1}{H^{1/2}}+C\frac{H}{N}$ (just go through the proof). Now suppose the $v_n$'s are mutually orthogonal. Then the left hand side is $\frac{1}{\sqrt{N}}$, while the right hand side is $O(\frac{1}{H^{1/2}})+O(\frac{H}{N})$ (which actually shows the need for the $O(\frac{1}{H^{1/2}})$ term), where the $O(\frac{1}{H^{1/2}})$ term came from $\frac{1}{N}\sum_{n \le N} \langle v_n,v_n \rangle$, the "diagonal term" in the proof above. The point is that the Van der Corput inequality allows one to get good bounds on $||\frac{1}{N}\sum_{n \le N} v_n||$ when the $v_n$'s are "almost orthogonal" (the preceding sentence shows it specializes to actual orthogonality).
We need to prove that
$$\int_0^1 \frac{1}{x^2 + \ln^2 x}\, \mathrm{d} x > 1.$$
Fact 1: For all $y \in (0, \mathrm{e}]$,
$$\frac{-y^4 + 64y^3 - 318y^2 + 400y + 35}{132y + 48} \ge y(1 - \ln y)^2.$$
(Note: LHS is the (4, 1)-Pade approximant of $y(1 - \ln y)^2$ at $y = 1$.)
With the substitution $y = \mathrm{e}x$, using Fact 1, we have
\begin{align*}
& \int_0^1 \frac{1}{x^2 + \ln^2 x}\, \mathrm{d} x\\
=\, & \int_0^\mathrm{e} \frac{\mathrm{e}}{y^2 + \mathrm{e}^2 (1 - \ln y)^2}\, \mathrm{d} y\\
\ge\, & \int_0^\mathrm{e} \frac{132\mathrm{e}y^2 + 48\mathrm{e}y}{(132 - \mathrm{e}^2)y^4 + (64\mathrm{e}^2 + 48)y^3 - 318\mathrm{e}^2 y^2 + 400\mathrm{e}^2y + 35\mathrm{e}^2}\, \mathrm{d} y\\
\ge\, & \int_0^\mathrm{e} \frac{132\mathrm{e}y^2 + 48\mathrm{e}y}{(132 - \mathrm{e}^2)(y + 1/12)(y + 463/64)[(y - 69/44)^2 + (55/54)^2]}\, \mathrm{d} y \tag{1} \\[8pt]
=\, &
\frac{1253273472\mathrm{e}}{911936989\mathrm{e}^2 - 120375682548} \int_0^\mathrm{e} \frac{1}{12y + 1}\, \mathrm{d} y\\[8pt]
&\quad + \frac{29129459463143424\mathrm{e}}{38952637956653\mathrm{e}^2 - 5141748210278196}
\int_0^\mathrm{e} \frac{1}{64y + 463}\, \mathrm{d} y \\[8pt]
&\quad + \frac{372594816\mathrm{e}}{2487336612548656836 - 18843459185974673\mathrm{e}^2}\\[8pt]
&\quad\quad \times \int_0^\mathrm{e} \frac{842186377401732y + 746090344086823}{1411344y^2 - 4426488y + 4934869}\, \mathrm{d} y \tag{2}\\
>\, & 1
\end{align*}
where in (1) we have used (easy): for all $y\in [0, \mathrm{e}]$,
\begin{align*}
&(132 - \mathrm{e}^2)y^4 + (64\mathrm{e}^2 + 48)y^3 - 318\mathrm{e}^2 y^2 + 400\mathrm{e}^2y + 35\mathrm{e}^2\\
\le\, & (132 - \mathrm{e}^2)(y + 1/12)(y + 463/64)[(y - 69/44)^2 + (55/54)^2].
\end{align*}
Note: The three integrals in (2) admit closed form expressions in terms of $\ln $ and $\arctan$ only.
We are done.
Best Answer
Remark: Although we get a closed-form bound (in terms of elementary functions), the solution is still quite complicated.
We have $$\int_{-\infty}^0 (\cosh x - 1)^x\, \mathrm{d}x = \int_0^\infty 2^{-x}\left(\sinh\frac{x}{2}\right)^{-2x}\,\mathrm{d} x = \int_0^\infty 2^{1 - 2x}(\sinh x)^{-4x}\, \mathrm{d}x.$$
Fact 1: $\int (\sinh x)^{-5}\, \mathrm{d}x = -\frac{\cosh x}{4\sinh^4 x} + \frac{3\cosh x}{8\sinh^2 x} - \frac{3}{8}\ln\frac{\mathrm{e}^x + 1}{\mathrm{e}^x - 1} + C$.
Fact 2: $\int (\sinh x)^{-4}\, \mathrm{d}x = -\frac{\cosh x}{3\sinh^3 x} + \frac{2\cosh x}{3\sinh x} + C$.
Fact 3: $\sinh x \ge x + \frac16 x^3$ for all $x\in [0, 1]$.
Let $T(f(x),a,n)$ denote the $n$-th order Taylor approximation of $f(x)$ at $x = a$.
Fact 4: For all $x\in [0, 1]$, $(1 + x^2/6)^{-4x} \le T((1 + x^2/6)^{-4x}, 0, 12)$.
(Proof: Take logarithm, then take derivative and the 2nd derivative.)
Fact 5: For all $x\in [0, 1]$, $2^{-2x} \le T(2^{-2x}, 0, 6)$.
(Proof: Take logarithm, then take derivative.)
Fact 6: For all $u \in [-1, 0]$, $\mathrm{e}^{-4u} \le T(\mathrm{e}^{-4u}, 0, 8) + \frac{512}{315}u^8$.
(Proof: Take logarithm, then take derivative.)
First, using Fact 1, we have \begin{align*} \int_{5/4}^\infty 2^{1 - 2x}(\sinh x)^{-4x}\, \mathrm{d}x &\le \int_{5/4}^\infty 2^{1 - 2\cdot \frac{5}{4}}(\sinh x)^{-4\cdot \frac{5}{4}}\, \mathrm{d}x \\ &= -\frac{\sqrt2}{4}\left(-\frac{\cosh (5/4)}{4\sinh^4 (5/4)} + \frac{3\cosh (5/4)}{8\sinh^2 (5/4)} - \frac{3}{8}\ln\frac{\mathrm{e}^{5/4} + 1}{\mathrm{e}^{5/4} - 1}\right)\\ &< \frac{3}{500}. \end{align*}
Second, using Fact 2, we have \begin{align*} \int_1^{5/4} 2^{1 - 2x}(\sinh x)^{-4x}\, \mathrm{d}x &\le \int_1^{5/4} 2^{1 - 2\cdot 1}(\sinh x)^{-4\cdot 1}\, \mathrm{d}x\\ &= -\frac{\cosh (5/4)}{6\sinh^3 (5/4)} + \frac{\cosh (5/4)}{3\sinh (5/4)} - \left(-\frac{\cosh 1}{6\sinh^3 1} + \frac{\cosh 1}{3\sinh 1}\right)\\ &< \frac{3}{80}. \end{align*}
Third, using Facts 3-6, we have \begin{align*} &\int_0^1 2^{1 - 2x}(\sinh x)^{-4x}\, \mathrm{d}x\\ \le\,& \int_0^1 2^{1 - 2x}(x + x^3/6)^{-4x}\, \mathrm{d}x\\ =\, & \int_0^1 2^{1 - 2x}(1 + x^2/6)^{-4x}x^{-4x}\, \mathrm{d}x\\ \le\, & 2\int_0^1 T(2^{-2x}, 0, 6) \, T((1 + x^2/6)^{-4x}, 0, 12) \, \left[T(\mathrm{e}^{-4u}, 0, 8) + \frac{512}{315}u^8\right]_{u = x\ln x}\, \mathrm{d} x \tag{1}\\ <\, & 1549/500. \end{align*} Note: The integral in (1) admits a closed form $a_0 + a_1\ln 2 + a_2\ln^2 2 + a_3\ln^3 2 + a_4\ln^4 2 + a_5\ln^5 2 + a_6\ln^6 2$ where $a_i$'s are all rational numbers.
Thus, we have $\int_0^\infty 2^{1 - 2x}(\sinh x)^{-4x}\, \mathrm{d}x < \frac{3}{500} + \frac{3}{80} + 1549/500 < \pi$.
We are done.