EDIT: Don't bother reading my partial solution. Brian Conrad pointed out that an easier way to do what I did is to use the equivalent definition of formally unramified in terms of Kähler differentials. And later on, fpqc posted below a complete solution passed on by Mel Hochster, who got it from Luc Illusie, who got it from ???.
OLD ANSWER:
Here is a half-answer. I'll prove half the conclusion, but on the plus side I'll use only half the hypothesis! Namely, I will prove that if $R \to S_{\mathfrak{p}}$ is formally unramified for all primes $\mathfrak{p} \subset S$, then $R \to S$ is formally unramified.
Let $A$ be an $R$-algebra, and let $I \subseteq A$ be a nilpotent ideal. Given $R$-algebra homomorphisms $f,g \colon S \to A$ that become equal when composed with $A \to A/I$, we must prove that $f=g$. Fix $\mathfrak{p}\subset S$. Then the localizations $A_{\mathfrak{p}} := S_{\mathfrak{p}} \otimes_{S,f} A$ and $S_{\mathfrak{p}} \otimes_{S,g} A$ of $A$ (defined viewing $A$ as an $S$-algebra in the two different ways) are naturally isomorphic, since adjoining the inverse of an $a \in A$ to $A$ automatically makes $a+\epsilon$ invertible for any nilpotent $\epsilon$ (use the geometric series). Now $f$ and $g$ induce $R$-algebra homomorphisms $f_{\mathfrak{p}},g_{\mathfrak{p}} \colon S_{\mathfrak{p}} \to A_{\mathfrak{p}}$ that become equal when we compose with $A_{\mathfrak{p}} \to A_{\mathfrak{p}}/I A_{\mathfrak{p}}$. Since $R \to S_{\mathfrak{p}}$ is formally unramified, this means that $f_{\mathfrak{p}} = g_{\mathfrak{p}}$. In other words, for every $s \in S$, the difference $f(s)-g(s)$ maps to zero in $A_{\mathfrak{p}}$ for every $\mathfrak{p}$. An element in an $S$-module that becomes $0$ after localizing at each prime ideal of $S$ is $0$, so $f(s)=g(s)$ for all $s$. So $f=g$.
Your question is:
Why, morally, do we need to look at the S-modules over the cofibrant replacement
of S to capture the lifting data from the rest of the square-zero extensions
that we would need to characterize formal smoothness (resp. formal étaleness)?
Here's how I think about it, though I don't know if it addresses what's bothering you about this.
You'd like to understand surjective maps of rings $f:T\to T'$ whose kernel $M$ (an $S$-module) is square-zero, but where $f$ might not admit a section.
There's a universal example of such an extension (in $R$-algebras over $S$); it lives in simplicial rings, i.e., in $(s(R\operatorname{-Alg})\downarrow S)$. Namely, form the simplicial $S$-module $BM$ by shifting (so $\pi_1BM=M$, $\pi_iBM=0$ for $i\neq1$), then consider the simplicial ring $S\oplus BM$, formed with trivial multiplication on the $BM$ part.
Now let $g: S\to S\oplus BM$ be the evident inclusion map into the first factor. The "kernel" of this map of simplicial rings is the homotopy fiber of $g$ (calculated in simplicial $S$-modules), and this homotopy fiber is equivalent to $M$ (concentrated in dimension $0$).
So $g$ has "kernel" $M$.
The claim is that isomorphism classes of square-zero extensions (possibly not split) over $S$ is the same as homotopy classes of maps $S\to S\oplus BM$ in $(s(R\operatorname{-Alg})\downarrow S)$. Calculating homotopy classes requires using a cofibrant model for the domain $S$. (In fact, there are no nontrivial maps $S\to S\oplus BM$ at all.)
Given such a map $f:QS\to S\oplus BM$ from a cofibrant replacement, pull back $g$ along it to get an extension $g'\colon E\to QS$. The simplcial ring $E$ has homotopy concentrated in degree $0$, and in fact there's an exact sequence $0\to M\to \pi_0 E\to \pi_0 QS=S$.
(Note that $(s(R\operatorname{-Alg})\downarrow QS)$ and $(s(R\operatorname{-Alg})\downarrow S)$ are Quillen equivalent model categories, so in fact giving an "extension" over $QS$ is the same as giving one for $S$.)
This story is morally just like the one you can tell for extension of $R$-modules: to compute extensions of $M$ by $N$, note that there's a universal extension by $N$ in the derived category of $R$-modules, of the form $N\to C\to N[1]$ (where $H_*C=0$). So extensions over $M$ are $\mathrm{Hom}_{D(R)}(M, N[1])$, and you can compute this group by replacing $M$ with a projective resolution (i.e., cofibrant replacement).
Best Answer
I think this works. Suppose we have a ring $R$ and an $R$-module $M$ all of whose localisations are projective and consider $S=S^\ast_RM$, the symmetric algebra on $M$. Then $R \rightarrow S$ is formally smooth precisely when $M$ is projective. Let $\mathfrak p$ be a prime ideal of $S$ and $\mathfrak q$ the prime ideal of $R$ lying below $\mathfrak p$. Then $S_{\mathfrak p}$ is a localisation of $S_{\mathfrak q}=S^\ast_{R_{\mathfrak q}}M_{\mathfrak q}$ which is a formally smooth $R_{\mathfrak q}$-algebra and hence $S_{\mathfrak p}$ is a formally smooth $R$-algebra (localisations being formally étale). To show that the statement is false it thus is enough to find a non-projective $R$-module all of localisation are projective.
If $R$ is a boolean ring (i.e., $r^2=r$ for all $r\in R$) then all its localisations are isomorphic to $\mathbb Z/2$ all of whose modules are projective. Hence it suffices to find a boolean ring $R$ and a non-projective $R$-module. This is easy: Let $S$ be a totally disconnected compact topological space and $s\in S$ a non-isolated point and put $R$ equal to the boolean ring of continuous maps $S \rightarrow \mathbb Z/2$. Evaluation at $s$ gives a ring homomorphism $R \rightarrow \mathbb Z/2$ making $\mathbb Z/2$ an $R$-module. If it were projective there would be a module section $\mathbb Z/2 \rightarrow R$. Let $e$ be the image of $1$. Then for any $f\in R$ we have $fe=f(s)e$. This means that $e$ must be the characteristic function of $s$. Indeed, as $e \mapsto 1$ under evaluation at $s$ we have $e(s)=1$. On the other hand, if $t\neq s$ there is an $f\in R$ with $f(t)=1$ and $f(s)=0$ giving $e(t)=f(t)e(t)=f(s)e(t)=0$. Hence $\{s\}=\{t\in S:e(t)=1\}$ is open so that $s$ is isolated.
Note that we may take for $S$ any infinite compact totally disconnected set as some point of it must be non-isolated. Nice examples are the Cantor set (all of whose points are non-isolated) and the spectrum of $\prod_T\mathbb Z/2$ for any infinite set $T$ (which is the ultrafilter compactification of $T$, any non-principal ultrafilter is non-isolated I think).