EDIT: Don't bother reading my partial solution. Brian Conrad pointed out that an easier way to do what I did is to use the equivalent definition of formally unramified in terms of Kähler differentials. And later on, fpqc posted below a complete solution passed on by Mel Hochster, who got it from Luc Illusie, who got it from ???.
OLD ANSWER:
Here is a half-answer. I'll prove half the conclusion, but on the plus side I'll use only half the hypothesis! Namely, I will prove that if $R \to S_{\mathfrak{p}}$ is formally unramified for all primes $\mathfrak{p} \subset S$, then $R \to S$ is formally unramified.
Let $A$ be an $R$-algebra, and let $I \subseteq A$ be a nilpotent ideal. Given $R$-algebra homomorphisms $f,g \colon S \to A$ that become equal when composed with $A \to A/I$, we must prove that $f=g$. Fix $\mathfrak{p}\subset S$. Then the localizations $A_{\mathfrak{p}} := S_{\mathfrak{p}} \otimes_{S,f} A$ and $S_{\mathfrak{p}} \otimes_{S,g} A$ of $A$ (defined viewing $A$ as an $S$-algebra in the two different ways) are naturally isomorphic, since adjoining the inverse of an $a \in A$ to $A$ automatically makes $a+\epsilon$ invertible for any nilpotent $\epsilon$ (use the geometric series). Now $f$ and $g$ induce $R$-algebra homomorphisms $f_{\mathfrak{p}},g_{\mathfrak{p}} \colon S_{\mathfrak{p}} \to A_{\mathfrak{p}}$ that become equal when we compose with $A_{\mathfrak{p}} \to A_{\mathfrak{p}}/I A_{\mathfrak{p}}$. Since $R \to S_{\mathfrak{p}}$ is formally unramified, this means that $f_{\mathfrak{p}} = g_{\mathfrak{p}}$. In other words, for every $s \in S$, the difference $f(s)-g(s)$ maps to zero in $A_{\mathfrak{p}}$ for every $\mathfrak{p}$. An element in an $S$-module that becomes $0$ after localizing at each prime ideal of $S$ is $0$, so $f(s)=g(s)$ for all $s$. So $f=g$.
Using some of BCnrd's ideas together with a different construction, I'll give a positive answer to Kevin Buzzard's stronger question; i.e., there is a counterexample for any non-etale smooth morphism.
Call a morphism $X \to Y$ wicked smooth if it is locally of finite presentation and for every (square-zero) nilpotent thickening $T_0 \subseteq T$ of $Y$-schemes, every $Y$-morphism $T_0 \to X$ lifts to a $Y$-morphism $T \to X$.
Theorem: A morphism is wicked smooth if and only if it is etale.
Proof:
Anton already explained why etale implies wicked smooth.
Now suppose that $X \to Y$ is wicked smooth. In particular, $X \to Y$ is smooth, so it remains to show that the geometric fibers are $0$-dimensional. Wicked smooth morphisms are preserved by base change, so by base extending by each $y \colon \operatorname{Spec} k \to Y$ with $k$ an algebraically closed field, we reduce to the case $Y=\operatorname{Spec} k$. Moreover, we may replace $X$ by an open subscheme to assume that $X$ is etale over $\mathbb{A}^n_k$ for some $n \ge 0$.
Fix a projective variety $P$ and a surjection $\mathcal{F} \to \mathcal{G}$ of coherent sheaves on $P$ such that some $g \in \Gamma(P,\mathcal{G})$ is not in the image of $\Gamma(P,\mathcal{F})$. (For instance, take $P = \mathbb{P}^1$, let $\mathcal{F} = \mathcal{O}_P$, and let $\mathcal{G}$ be the quotient corresponding to a subscheme consisting of two $k$-points.) Make $\mathcal{O}_P \oplus \mathcal{F}$ an $\mathcal{O}_P$-algebra by declaring that $\mathcal{F} \cdot \mathcal{F} = 0$, and let $T = \operatorname{\bf Spec}(\mathcal{O}_P \oplus \mathcal{F})$. Similarly, define $T_0 = \operatorname{\bf Spec}(\mathcal{O}_P \oplus \mathcal{G})$, which is a closed subscheme of $T$ defined by a nilpotent ideal sheaf. We then may view $g = 0+g \in \Gamma(P,\mathcal{O}_P \oplus \mathcal{G}) = \Gamma(T_0,\mathcal{O}_{T_0})$.
Choose $x \in X(k)$; without loss of generality its image in $\mathbb{A}^n(k)$ is the origin. Using the infinitesimal lifting property for the etale morphism $X \to \mathbb{A}^n$ and the nilpotent thickening $P \subseteq T_0$, we lift the point $(g,g,\ldots,g) \in \mathcal{A}^n(T_0)$ mapping to $(0,0,\ldots,0) \in \mathbb{A}^n(P)$ to some $x_0 \in X(T_0)$ mapping to $x \in X(k) \subseteq X(P)$. By wicked smoothness, $x_0$ lifts to some $x_T \in X(T)$. The image of $x_T$ in $\mathbb{A}^n(T)$ lifts $(g,g,\ldots,g)$, so each coordinate of $x_T$ is a global section of $\mathcal{F}$ mapping to $g$, which is a contradiction unless $n=0$. Thus $X \to Y$ is etale.
Best Answer
It seems that what you are looking for is theorem 5.11 here. See also example (e) on section 5.12. Also if you don't feel like reviewing from EGA you can look at section 1.5 of "Introduction to algebraic stacks" by A. Canonaco which I think covers the relevant facts (including 17.5.1)