With some category/topos theory we can now put infinitesimals on a rigorous ground, as in Bell's A Primer of Infinitesimal Analysis, where the author introduces $\epsilon$ satisfying \begin{equation}
\epsilon\ne 0, \epsilon^2=0.
\end{equation}
However, he also points out that this version of infinitesimal is not compatible with the law of excluded middle.
Meanwhile, the author seems convinced that this $\epsilon$ is the infinitesimal in the eyes of Newton and Leibniz among many others, when they were attacking problems like instantaneous speed and area under a curve.
I wonder whether this is true. I know people like Newton and Leibniz did not use limiting argument. But this does not mean they think of infinitesimals as nilsquare elements as described by Bell, because there are still other models of infinitesimals available.
Thanks very much.
Best Answer
I think (but am very open to correction by those who know the history better than I do) that the situation is basically this. Newton, Leibniz and other early practitioners made a bunch of assumptions (both overt and implicit) about infinitesimals that can't coherently all be held true together. That's why from the earliest days, starting with Bishop Berkeley's mockery of the very idea, there was an unresolved anxiety about infinitesimals that lasted right into the twentieth century. They "worked" wonderfully well, but seemingly shouldn't have.
So the name of the game isn't to find the coherent account of infinitesimals that Newton and Leibniz had in view. Rather the game is one of optimal rational reconstruction. What coherent view gives us a theory which satisfies the most important features that Newton and Leibniz wanted in a calculus of infinitesimals, and does so in the neatest, most mathematically fruitful way?
But note there needn't be a fact of the matter about what are the most important features, about what's neatest, about what's most fruitful. These are going to be judgement calls, and maybe there will be different ways to go, with rather different costs and benefits. Even if we decide that the kind of theory which Bell presents is overall a "best buy", it would certainly be pushing the principle of charitable interpretation to breaking point to say that it was therefore what Newton and Leibniz "really" meant all along. Almost certainly, any coherent theory will have to deny things they would have taken for granted.
But then is Bell's smooth infinitesimal analysis the best buy? As the OP indicates, there are other and more popular modern versions of nonstandard analysis which stick closer to Robinson's original. For an extended development of one, see Nader Vakil's impressive and illuminating Real Analysis Through Modern Infinitesmials (in the Cambridge Encyclopedia of Mathematics series). But I wouldn't want to say either that such a theory rather than Bell's -- attractive though it is -- is what Newton and Leibniz really were talking about! Being the best buy in the neighbourhood is recommendation enough.