The simplest answer is that he is using certain objects such as natural numbers and functions via their abstracted properties (called interfaces in programming languages), instead of pinning down any specific objects that have those properties (called implementations in programming languages). The reason is that we usually care about the interfaces, and rarely about how exactly they are implemented.
Remember that usually mathematical objects do not come alone but as members of some structure, and the properties we want them to have is better understood as properties of the structure as a single whole. That also means that we are free to change the structure as long as it satisfies the same properties.
The typical example is the natural numbers. Almost all the time we only care about the properties that they have in relation to one another, like $0 < 1 < 2 < 3 < \cdots$, and induction over the natural numbers. These properties are all required to hold regardless of how the collection of natural numbers is actually formed. ZFC has many ways of exhibiting a model of PA, and in fact Zermelo's original construction of one of these models was different from the now standard von Neumann construction. The reason we use the latter now is one of the rare cases where we find that it is more convenient to have the natural numbers be a transitive set, so that they are von Neumann ordinals. There are technical advantages to that, because we can use the inbuilt symbol $\in$ for the ordering on ordinals. One can say that we like the structure of natural numbers to internally satisfy PA and externally be a substructure of the structure of ordinals, so there is special reason to privilege the von Neumann construction of natural numbers.
Another typical example is the real numbers. This time we never in practice care how they are constructed. As long as they satisfy the field properties and the second-order completeness axiom, that is good enough. One can say that we only need the structure of the reals to satisfy internal properties, and we don't care about any external properties. So whether use choose the Cauchy sequences of rationals or Dedekind cuts of the rationals or decimal sequences, it doesn't matter at all. The only point at which it matters is the when you want to construct the structure to show that there is some instance satisfying the desired properties, where in some weak systems some of these are not possible to construct. Thereafter we use real numbers only via their interface.
Similarly for functions. In ZFC they can be encoded as special kinds of subsets of the cartesian product of the domain and codomain, but in practice we don't care, and sometimes don't want that restriction, when dealing with class functions for example. You see, you think of the powerset symbol $\mathcal{P}$ as a function, don't you? But it can't be a set in ZFC unless you want a contradiction. Same for the cardinality map, and in general for any function-like thing whose domain or codomain is not a set. One could work in the strictly stronger MK set theory to avoid that issue, but the point remains that we don't really care how functions are implemented as long as we can use them the way we expect them to function.
To ensure that the mathematical structures one uses can be constructed in ZFC, one just has to make sure that either there is a set in ZFC that satisfies the desired interface (such as for natural numbers and reals), or there is a syntactic translation from one's proof to a proof in ZFC (such as proofs using class functions, which often but not always can be translated systematically into proofs that work in ZFC, simply by replacing the use of those functions by formulae), or that one is working in a conservative extension of ZFC (such as using full abbreviation power).
Here is a model of your "finite set theory" (including Foundation) in which there is an infinite set and Power Set and Replacement fail. Let $A=\{\emptyset,\{\emptyset\},\{\{\emptyset\}\},\{\{\{\emptyset\}\}\},\dots\}$ and let $M$ be the closure of $V_\omega\cup\{A\}$ under Pairing, Union, and taking subsets (so if $X\in M$ and $Y\subseteq X$ then $Y\in M$). It is clear that $M$ satisfies all of your axioms except possibly the negation of Infinity. To prove the negation of Infinity, note that if $X\in M$, then the transitive closure of $X$ contains only finitely many elements of cardinality $>1$ (since this is true of $A$ and for every element of $V_\omega$, and is preserved by taking pairs, unions, and subsets). So $M$ cannot contain any inductive set.
However, $M$ does contain an infinite set, namely $A$. It is also clear that $M$ fails to satisfy Power Set, since $M$ contains every subset of $A$ but $\mathcal{P}(A)\not\in M$ (either by the criterion mentioned above, or by noting that every element of $M$ is countable). Replacement also fails, since the usual recursive definition of the obvious bijection $A\to\omega$ can be implemented in $M$, so Replacement would imply $\omega$ is a set.
This model does satisfy Choice in the form "if $X$ is a set of disjoint nonempty sets then there is a set that contains one element from each of them" (since $M$ contains all subsets of $\bigcup X$). It does not satisfy Choice in the form "if $X$ is a set of nonempty sets then there exists a choice function $X\to\bigcup X$", basically because it is very hard to construct functions as sets in $M$ (for instance, if $X=A\setminus\{\emptyset\}$, the unique choice function for $X$ would have infinitely many 2-element sets in its transitive closure). Probably it is possible to build a model where any reasonable form of Choice fails, but I don't know how exactly to do that at the moment.
Best Answer
You're misunderstanding the role of the $\mathsf{ZFC}$ axioms. Their goal is to basically be a "once-and-for-all" framework for doing mathematics; rather than describe any particular mathematical object necessarily, the idea is that they should describe the whole universe of mathematics. This is a universe which, among other things, contains vector spaces - so one of the things the $\mathsf{ZFC}$ axioms should be able to do is prove things about vector spaces. By contrast this just isn't something the vector space axioms are intended to do: we don't e.g. ask "Can the vector space axioms prove Fermat's last theorem?"
The $\mathsf{ZFC}$ axioms therefore need to be considered along with general implementation strategies - ideas for how a priori non-set-theoretic mathematical concepts can be implemented in set theory. These strategies - or rather strategy really, there's never much of a trick to it, it's just really tedious - mean that we can express all of mathematics in the language of sets alone in a very real sense.
Having fixed a way of expressing mathematical statements in the pure language of set theory, we now want to actually prove/disprove them. The success of $\mathsf{ZFC}$ is measured in large part by its ability to prove things about other mathematical objects, implemented according to the previously-mentioned strategies, which we can prove in "naive mathematics" already. This is what the quote is referring to: while there are indeed mathematical questions which the $\mathsf{ZFC}$ axioms do not resolve, they are few and far between (outside of mathematical logic anyways :P).