As Noam correctly mentioned, the upper bound has to be exponential. However, the exponent can, indeed, be improved.
One (fairly cheap) trick is to look at $|f(x)|^2=2^{|S|}\prod_S(1+\cos 2\pi sx)$. If we can show that the product is bounded from above by $e^{-c|S|}$ outside a set of measure $e^{-c|S|}$, we can get away with Cauchy-Schwarz. Now, $\log(1+y)\le y-cy^2$ for $y\in[-1,1]$ with some $c>0$. Since $2\cos^2 y=1+\cos 2y$, we see that the square term will give us a fixed linear in $|S|$ push down, so it remains to show that for every $a>0$, the set of $x$ for which $\left|\sum_{s\in S} \cos 2\pi sx \right|$ is at least $a|S|$ has exponentially small measure (we'll need to use that twice: once for $x$ and once for $2x$).
This is actually pretty easy if we recall the reverse inequality $\log(1+x)\ge x-Cx^2$ for $x\in[-\frac 12,\frac 12]$, say. Note that the unique representation as a sum property implies that
$$
\int_0^1 \prod(1+t\cos 2\pi sx)\,dx=1
$$
for every $t\in\mathbb R$. Assuming that $\sum_{s\in S} \cos 2\pi sx>a|S|$ on a set $E$, choose $t\in(0,\frac 12)$ so small that $b=at-Ct^2>0$. Then, clearly, the product is at least $e^{b|S|}$ on $E$ and the desired exponential bound on the measure of $E$ follows at once. The other set where $\sum_{s\in S} \cos 2\pi sx<-a|S|$ is treated the same way using small negative $t$.
This all is extremely crude, of course. However, it answers the original question in the affirmative, so I'll stop here :).
Regarding your first question, observe that (2) is equivalent to
$$ |S_f(N)|^2 \le 4N^2q^{-1}+4N\log q+q\log^2 q. $$
Hence, by (1), it suffices to show
$$ N +2N^2q^{-1}+4N\log q + 4q\log q \le 4N^2q^{-1}+4N\log q+q\log^2 q,$$
which in turn is equivalent to
$$ N+4q\log q \le 2N^2q^{-1}+q\log^2 q.$$
Here we have $N\leq 2N^2q^{-1}$ by $1\leq q\leq 2N$, hence we are done when $\log q\geq 4$.
For $\log q<4$ we argue more carefully. In this case, we can rewrite the last inequality as
$$ (4-\log q)(q\log q)\leq 2N^2q^{-1}-N,$$
$$ 32\log q-8\log^2 q\leq 16N^2q^{-2}-8Nq^{-1},$$
$$ 1+32\log q-8\log^2 q\leq (4Nq^{-1}-1)^2,$$
$$ q\left\{1+\sqrt{1+32\log q-8\log^2 q}\right\}\leq 4N.$$
The left hand side is clearly at most $e^4(1+\sqrt{33})$, hence we are done if $N\geq 93$.
The remaining pairs $(N,q)$ satisfy $1\leq q\leq 2N\leq 184$, and the question reduces to whether
$$ \min(N^2,N +2N^2q^{-1}+4(N+q)\log q)\leq 4N^2q^{-1}+4N\log q+q\log^2 q$$
holds for these pairs or not. Equivalently, we need to test the inequality
$$ \min(N^2-4N\log q,N +2N^2q^{-1}+4q\log q)\leq 4N^2q^{-1}+q\log^2 q\tag{$\ast$}$$
for $1\leq q\leq 2N\leq 184$.
Regarding your second question, observe that the right hand side of (2) is about $\sqrt{N}\log N$ when $q=N$, while the bound needs to be at least $\sqrt{N}$ due to (8.12) in Iwaniec-Kowalski. In particular, when Iwaniec-Kowalski say that "slightly better results hold true for almost all coefficients", they mean that on average $\log N$ can be either omitted or replaced by $\sqrt{\log N}$. See also (8.13) and (8.14) in the book.
Update 1. Checking with SAGE, it turns out that $(\ast)$ fails for some pairs $(N,q)$, the smallest and largest such pairs being $(21,13)$ and $(41,44)$. This means that the proof of Theorem 8.1 in Iwaniec-Kowalski is incomplete, but it is fine for $N\geq 42$ (and also for $N\leq 20$).
Update 2. Replacing $4(N+q)\log q$ in (1) by its source (cf. p.199 in Iwaniec-Kowalski)
$$ 2(N+q)\Bigl(\sum_{1\leq\ell\leq\frac{q}{2}}\ell^{-1}+\sum_{1\leq\ell<\frac{q}{2}}\ell^{-1}\Bigr),$$
we get with SAGE that Theorem 8.1 holds for $N\geq 32$ (and also for $N\leq 23$).
Update 3. Here is a further improvement that proves Theorem 8.1 in the remaining cases. In (8.8) of Iwaniec-Kowalski, we can replace $2N$ by $2(N-\ell)$ in the light of the preceding display. Hence, in (1), we can replace $2N^2q^{-1}$ by $\sum_{1\leq\ell'\leq N'}2(N-q\ell')$, where $N':=\lfloor N/q\rfloor$. This new sum equals
$$\sum_{1\leq\ell'\leq N'}2(N-q\ell')=2N'N-qN'(N'+1) < 2N'N-N'N = N'N \leq N^2q^{-1}.$$
Therefore, in (1), we can replace $2N^2q^{-1}$ by $N^2q^{-1}$, and then we get with SAGE that Theorem 8.1 holds without exception. It is worthwhile to note that the improvement in "Update 2" is also needed to reach this conclusion.
P.S. It is possible that with the improvements of (1) outlined above, one can reach (2) in a simpler way. I have not examined this possibility (for lack of time).
Best Answer
This seems easier than you might have expected. Up to normalization, your quantities $\alpha(m,D)$ are Fourier coefficients of the indicator function of $D$ (for which reason many people would rather use the notation $\hat 1_D(m)$). As such, they satisfy the Parseval identity $$ \sum_m |\alpha(m,D)|^2 = n|D|. $$ In view of $|\alpha(m,D)|\le|D|$, this yields $$ n|D| \le \sum_m |D| |\alpha(m,D)|, $$ whence $$ \sigma = \frac1n \sum_m |\alpha(m,D)| \ge 1. $$ Moreover, for equality to hold, one needs all $|\alpha(m,D)|$ to be equal to either $0$ or $|D|$, which is only possible if $D$ is a coset of a subgroup of ${\mathbb Z}/n{\mathbb Z}$.