For the history of first-order logic I strongly recommend "The Road to Modern Logic-An Interpretation", José Ferreirós, Bull. Symbolic Logic v.7 n.4, 2001, 441-484. Author's website, ASL, JSTOR, DOI: 10.2307/2687794.
Apart from first order logic and higher order logic there are several less well known logics that I can mention:
Constructive logics used to formalize intuitionism and related areas of constructive mathematics. One key example is Martin-Löf's intuitionistic type theory, which is also very relevant to theoretical computer science.
Modal logics used to formalize modalities, primarily "possibility" and "necessity". This is of great interest in philosophy, but somehow has not drawn much interest in ordinary mathematics.
Paraconsistent logics, which allow for some inconsistencies without the problem of explosion present in most classical and constructive logics. Again, although this is of great philosophical interest, it has not drawn much attention in ordinary math.
Linear logic, which is more obscure but can be interpreted as a logic where the number of times that an hypothesis is used actually matters, unlike classical logic.
Yes, when someone says that ZFC is a suitable foundation for mathematics, that means that all standard mathematical notions can be expressed in the language of ZFC using first-order logic and all standard mathematical arguments can be written as derivations from ZFC using first-order logic.
For your particular example of torsion groups, there is a first-order formula $\psi(x)$ with one parameter in the language of set theory such that, for any $x,$ $\psi(x)$ holds iff $x$ is a torsion group.
The statement that torsion groups can't be characterized in first-order logic means something different, namely:
There is no first-order sentence $\varphi$ in the language of group theory such that, for every group $G,$ $G$ is a torsion group iff $G$ satisfies the sentence $\varphi.$
Note that $\varphi$ here is required to be in the language of group theory; it can only quantify over elements of the group (not subsets of the group, or natural numbers, or anything else), and it can't refer to any functions or relations other than the multiplication operation of the group.
So how would you write the original formula $\psi$ in the language of set theory? The formula $\psi(x)$ would say something like the following (you'd have to expand all the circumlocutions fully, of course, but, although that's a bit unwieldy, it's routine to do):
$x$ is an ordered pair $\langle G, \circ\rangle.$
$\circ$ is a function mapping $G\times G$ to $G$ satisfying the axioms for a group.
For every $a\in G,$ there exists a natural number $n$ and a function $f: n+1\to G$ such that:
(i) $f(0) = a;$
(ii) $(\forall k\lt n)(f(k)\circ a = f(k+1));$
(iii) $f(n)$ is the identity element of the group $\langle G, \circ\rangle.$
This is a first-order formula in the language of set theory, but you can see that it is not a formula in the language of group theory; it talks about objects other than elements of $G,$ and it involves quantifying over natural numbers (and also functions, or finite tuples from $G).$
(For anybody who hasn't seen the usual set-theoretic convention regarding natural numbers, I should mention that $\psi$ as written above uses the formulation that for any natural number $m,$ we have $m = \{k \mid k \lt m\}.)$
Best Answer
The point is that using ZFC we can interpret1 second-order logic. ZFC gives us a way to understand the notion of sets and relations which are necessary for second-order logic.
We can prove, as a first-order statement in the language of ZFC, that there is a unique Dedekind-complete ordered field (up to isomorphism). This is a second-order statement about the real numbers, yes, but we can interpret that in ZFC, which transforms it into a first-order logic statement in the language of set theory.
1. In the non-technical sense.