It is well known among historians of Fermat that, while his technique of adequality prepared the ground for the general framework later developed by Leibniz and Newton, Fermat himself gave very little in the way of explanation of his technique exploiting a symbol $E$ that appears to us to behave like an infinitesimal.
There is an interesting issue concerning Fermat's opponents and whether he may have felt a need to be watchful in his actions correspondingly, and specifically with regard to how he presented his method of adequality that could potentially run the risk of being found heretical because of its connection to the infinitely small, indivisibles and the decomposition of the continuum into indivisibles, and atomism. The latter were thought to be at tension with established canon (specifically Trent 13.2 and the problems atomism constitutes for transubstantiation and eucharist similar to Grassi's criticism of Galileo, as discussed by both Redondi and Festa) and were repeatedly banned; for details see the books by Redondi and Alexander.
(1) Fermat was a judge on the Parlement de Toulouse. He must have aggravated some people at the Parlement de Toulouse by seeking out help in Paris to get the appointment on the Chambre d'Edit in Castres. The king overruled the decision of the Parlement and named Fermat instead.
(2) The president of the Parlement seems to have been an opponent of Fermat. There is a document that was once secret that contains a character report on Fermat describing him as "interessé" and implying that he was unreliable. The damaging secret report was written by a crony of the President's.
(3) Descartes attacked Fermat's technique of adequality. This must have been well known at the time because Descartes was famous. It would be interesting to determine whether Fermat's enemies ever used this against him.
(4) According to Barner's interpretation of a capital case at the Parlement de Toulouse involving a priest named Delpoy, the latter was executed apparently over helping a Calvinist girl hide after she escaped from a re-education school. The execution apparently went against Fermat's recommendations. Fermat was reporting on this case but was not the one who ruled to execute the priest.
(5) According to Barner, Fermat was shocked by the Delpoy thing and couldn't concentrate on his math for at least a month, according to Kenelm Digby. There was a correspondence between Wallis and Fermat at the time through Digby that came to a standstill following the Delpoy affair.
(6) Barner also documents a sharp drop in Fermat's productivity at the Parlement for a month following the execution of Delpoy.
(7) As far as Weil's comment with regard to Mahoney's claims concerning Digby's letter to Wallis: Weil wrote: "an inquisitive historian might do worse than try to find out whether the above story was not a figment of [DIGBY'S] lively imagination." While Mahoney did not put in such an effort, Barner did, and found archival evidence that Fermat was in distress over the Delpoy affair.
I am seeking such information about Fermat's possible opponents and their possible actions against him in the context of sensitive events such as the indictment and execution of the priest Delpoy in order to gauge whether Fermat may have also been hesitant to be too frank about the foundations of his method of adequality because of possible theological risks involved and the possible use of this by his opponents. Do we have evidence of such actions by his opponents beyond the president's attempt to block Fermat's appointment at Castres, Descartes' attack against adequality, and disagreements over Delpoy?
Note 1. See this related thread on Redondi, Galileo, and atomism.
Note 2. See a related 2018 publication in Foundations of Science.
Best Answer
Maybe the following article http://arxiv.org/abs/1306.5973 (Is mathematical history written by the victors?) and references therein will be useful :-).
Fermat's life and work is carefully investigated in the book "The Mathematical Career of Pierre de Fermat, 1601-1665" by Michael Sean Mahoney: http://press.princeton.edu/titles/5449.html See, however, Gerry Myerson's comment below and https://mathematicswithoutapologies.wordpress.com/2015/03/25/andre-weil-vs-history-of-mathematics/
P.S. I have just found two papers by Klaus Barner (in French) which might be useful (in case you are not already aware of them):
https://eudml.org/doc/10129 (Pierre Fermat Sa vie privée et professionnelle)
http://www.zeitschrift-rechtsgeschichte.de/de/article_id/581 (Fermat et l'affaire Delpoy)
P.P.S. Two more papers by Klaus Barner:
http://www.degruyter.com/view/j/dmvm.2001.9.issue-3/dmvm-2001-0070/dmvm-2001-0070.xml (Das Leben Fermats)
http://www.math-in-europe.eu/images/information_pdf/hist_phil_pdf/hist_pdf/barner.pdf (Pierre de Fermat (1601? - 1665) His life beside mathematics)
Russian science fiction writer Alexander Kazantsev wrote a historical novel about Fermat's life: http://www.litmir.me/bd/?b=261604 (Острее шпаги). A German translation of it does exist: https://www.amazon.de/Alexander-Kasanzew-Sch%C3%A4rfer-als-Degen/dp/3355003638 (Schärfer als Degen).
From the more scientific point of view, maybe the following book will be useful: https://www.amazon.co.uk/Public-Life-Toulouse-1463-1789-Cosmopolitan/dp/0801421918 (Public Life in Toulouse, 1463-1789: From Municipal Republic to Cosmopolitan City, by R.~A.~Schneider).