Here is a model of your "finite set theory" (including Foundation) in which there is an infinite set and Power Set and Replacement fail. Let $A=\{\emptyset,\{\emptyset\},\{\{\emptyset\}\},\{\{\{\emptyset\}\}\},\dots\}$ and let $M$ be the closure of $V_\omega\cup\{A\}$ under Pairing, Union, and taking subsets (so if $X\in M$ and $Y\subseteq X$ then $Y\in M$). It is clear that $M$ satisfies all of your axioms except possibly the negation of Infinity. To prove the negation of Infinity, note that if $X\in M$, then the transitive closure of $X$ contains only finitely many elements of cardinality $>1$ (since this is true of $A$ and for every element of $V_\omega$, and is preserved by taking pairs, unions, and subsets). So $M$ cannot contain any inductive set.
However, $M$ does contain an infinite set, namely $A$. It is also clear that $M$ fails to satisfy Power Set, since $M$ contains every subset of $A$ but $\mathcal{P}(A)\not\in M$ (either by the criterion mentioned above, or by noting that every element of $M$ is countable). Replacement also fails, since the usual recursive definition of the obvious bijection $A\to\omega$ can be implemented in $M$, so Replacement would imply $\omega$ is a set.
This model does satisfy Choice in the form "if $X$ is a set of disjoint nonempty sets then there is a set that contains one element from each of them" (since $M$ contains all subsets of $\bigcup X$). It does not satisfy Choice in the form "if $X$ is a set of nonempty sets then there exists a choice function $X\to\bigcup X$", basically because it is very hard to construct functions as sets in $M$ (for instance, if $X=A\setminus\{\emptyset\}$, the unique choice function for $X$ would have infinitely many 2-element sets in its transitive closure). Probably it is possible to build a model where any reasonable form of Choice fails, but I don't know how exactly to do that at the moment.
If we add to ZFC every statement that is proved to be unprovable in ZFC, we would obtain an inconsistent system, because there are sentences that are both not-provable and not-disprovable in ZFC.
Take the particular case of the continuum hypothesis, CH. Both CH and its negation are unprovable in ZFC. So we can only add one of them to ZFC at a time. But then we have to decide which one. Of course, we can assume either CH or its negation on a temporary basis, but if we wanted to add one "once and for all" we would have to decide whether there is a consensus to assume CH, or a consensus to assume the negation. Because there is not a widely accepted argument for either of these options, we don't add either axiom to ZFC on a permanent basis.
The axioms that are in ZFC are all consistent with a particular understanding of sets as forming a cumulative hierarchy. Indeed, ZFC as a theory is set up for studying the cumulative hierarchy, rather than any sets that might exist outside of it. But our understanding of the cumulative hierarchy is not complete, so that we have no idea whether CH (for example) should hold.
There has been a lot written on whether new axioms should be added as "basic axioms" in mathematics, extending ZFC. For example, see
Feferman, S., Friedman, H., Maddy, P., & Steel, J. (2000). Does Mathematics Need New Axioms? Bulletin of Symbolic Logic, 6(4), 401-446. doi:10.2307/420965
Best Answer
The reflection principle is a theorem schema in ZFC, meaning that for each formula $\phi(\vec x)$ we can prove in ZFC a version of the principle for $\phi$. In particular, it gives us that if $\phi$ holds (in the universe of sets) then there is some ordinal $\alpha$ such that $V_\alpha\models \phi$.
It follows from this that (assuming its consistency) $\mathsf{ZFC}$ is not finitely axiomatizable. Otherwise, $\mathsf{ZFC}$ would prove its own consistency, violating the second incompleteness theorem. The (standard) list of axioms you presented is actually an infinite list, with replacement being in fact an axiom schema (one axiom for each formula).
It is perhaps worth mentioning that no appeal to the incompleteness theorem is needed: If $\mathsf{ZFC}$ is consistent, and finitely axiomatizable, then it would prove (because of reflection) that there are $\alpha$ such that $V_\alpha\models\mathsf{ZFC}$. It would then follow that there is a least such $\alpha$. But inside $V_\alpha$ there must be some $\beta$ such that $$V_\alpha\models\mbox{``}V_\beta\models\mathsf{ZFC}\mbox{''}$$ (because $V_\alpha$ is a model of set theory, so it satisfies reflection), and easy absoluteess arguments give us that then $\beta<\alpha$ is indeed an ordinal, and $V_\beta$ is really a model of $\mathsf{ZFC}$, contradicting the minimality of $\alpha$.