Underlying logic is bi-sorted first order logic with identity, added to it the primitive symbol of set membership $\in$ and of binary relation strict smaller than $<$. Where $\in$ is a relation from first sort objects (represented by lower cases standing for naturals) to second sort objects (represented by upper cases denoting sets of naturals), while < relation is a binary relation restricted to first sort objects. The identity relation is un-restricted.
Sorting axioms:
Disjointedness: $\forall x,Y (x \neq Y)$
Existence: $\exists x (x=x)$
Extra-logical axioms:
-
Well ordering axiom:
Asymmetric: $ x < y \to \neg (y < x) $
Transitive: $ x < y \land y < z \to x < z$
Connective: $ x \neq y \leftrightarrow (x < y \lor y < x)$
Well founded: $\exists n \in X \to \exists n \in X \forall m \in X (n \leq m)$
-
Finiteness: $\exists n \in X \to \exists n \in X \forall m \in X (m \leq n)$
-
Potential Infinity: $\forall x \exists y (x < y)$
-
Extensionality: $\forall X \forall Y [\forall z (z \in X \leftrightarrow z \in Y) \to X=Y]$
-
Set formation: if $\phi(m)$ is a formula in which $m$ is free, but $X$ not free, then all closures of: $$\forall n \exists X \forall m (m \in X \leftrightarrow m < n \land \phi(m))$$, are axioms.
Is this theory a conservative extension of $\sf PA$?
The idea is that this theory can easily define the arithmetical primitives: $0$ is provable from axioms of existence and well ordering axiom, both the successor and predecessor functions are provable from axioms of well foundedness and finiteness and axiom of potential infinity. Induction principle is provable from set formation and well ordering axiom. Now "<" being a well founded relation over first sort object would support recursive definitions, since they'd have a solution, so we'll use this property to define a counting recursive function $\#$ on sets and numbers, the idea is for that function to count the numbers, so for any set K the first element in $K$ (with respect to $<$ relation) would be sent by this counting function to $1$, the next bigger element in $K$ would be sent to number $2$ and so on, the counting function is recursive it sends each element of $K$ to the the successor of the value of that function on the preceding element in $K$ till we reach min(k) which would be assigned number $1$. Using this counting function then the value assigned to the maximal element in $K$ would be the cardinality of $K$, i.e. the number of elements in $K$. Then we can easily define summation of naturals in the usual set theoretic manner as the size of the union of disjoint sets having their cardinalities being the arguments of summation. Multiplication is a little bit more complex, but it uses partitioning sets which has equal intervals between its elements, and it is easy to get multiplication using that function. Here is the formal workup:
Define recursively: $\#^K (x) = n \leftrightarrow [x=min(K) \land n=1] \lor [x \in K \land x > min(K) \land n= S[\#^K(P^K(x))]$
Where $P^K(x) = y \leftrightarrow x \in K \land y \in K \land y < x \land \not \exists z \in K (y < z < x)] $
Define Successor as: $x=S(y) \iff y < x \land \not \exists z (y < z < x)$
Define: $|K|=n \iff [K \text { is non empty } \land n= \#^K(max(K))] \lor [K \text { is empty } \land n=0]$.
Define: $x + y = z \iff \exists X,Y,Z (X \text { disjoint } Y \land Z= X \cup Y \land |X|=x \land |Y|=y \land |Z|= z )$
Define: $X \text { is d partitioning set } \iff \forall L (\exists a,b \in X (a=P^X (b) \land L=\{n|a < n \leq b\}) \lor L=\{n| 0< n \leq min(X)\} \to |L|=d )$
Define: $X^*=\{n| \exists m \in X (0< n \leq m) \}$
Define: $x \times y = z \iff \exists X (X \text { is x partitioning set } \land |X|=y \land z= |X^*|) \lor [x=0 \land y \neq 0 \land z=0]$
Best Answer
Seriously edited to address mistakes in original version; see the history of interested. I do think that all the claims I made originally are true, FWIW.
First, let's put the question as stated to bed. Consider the "standard model" $\mathcal{N}$ of your theory with first-order part $\mathbb{N}$ and second-order part $\mathcal{P}_{fin}(\mathbb{N})$ (and the usual interpretations of $<$ and $\in$). We trivially have set formation and well-foundedness here, since we've included everything we could want (set formation will pose a subtlety below, which tripped me up originally).
But $\mathcal{N}$ is decidable, that is, its full theory $Th(\mathcal{N})$ is computable (see here). No theory with a decidable model can interpret PA, since PA is essentially undecidable, so in particular your theory doesn't. Indeed we can push this vastly below PA: your theory doesn't even interpret Robinson arithmetic.
Now it seems that a fairly simple argument should lift this observation to a proof that in fact PA (and indeed a much weaker theory) proves the consistency of your theory. However, there are some subtleties here which I haven't yet nailed down comfortably, so I'll leave that for a later edit. The argument that I gave in a previous edit (that any model of $Th(\mathbb{N}; <)$ expands to a model of your theory once we add all bounded definable sets) would do this, but has a serious gap: comprehension in the expansion is not trivial, since we need to argue that when we add the ability to quantify over bounded definable subsets of our starting structure we don't generate any new bounded definable sets, and - while I think this is true - I don't have a complete argument for this yet.
Now let me address the more general question which arose in the comments and subsequent edit:
Certainly it can't in general, even along genuine well-orderings: consider $\mathcal{N}_P=(\mathbb{N};<,+)$. Clearly we can give a recursive definition of multiplication here, but $\mathcal{N}_P$ is decidable while $(\mathbb{N};<,+,\times)$ very much isn't, so multiplication can't actually be first-order definable in $\mathcal{N}_P$.
So at this point it's a good idea to look back at how definition by recursion is sometimes possible, e.g. in PA. What we do there is use finite sequences, and we do this via pairing (or something morally equivalent). That's an important dependency: coding sequences precedes definition by recursion.
Now finite sequences can be used to provide "derivations" of instances of recursive definitions. For instance, assuming we have addition "in hand," a derivation of "$a$ times $b$ equals $c$" would be a sequence of length $b$ whose first term is $a$, last term is $c$, and with difference $a$ between successive terms. Induction principles can then be invoked to prove that appropriate derivations exist; for instance, the totality of multiplication amounts to
and this is proved by induction, roughly as follows:
Fixing $a$, suppose $b$ is the minimal counterexample for the principle holding for $a$.
By induction (the base case being boring), let $d$ be the unique number such that there is a derivation of "$a$ times $b-1$ equals $d$," and let $\sigma$ be that derivation.
Appending $a+d$ to $\sigma$ gives a derivation of "$a$ times $b$ equals $a+d$" (so take $c=a+d$). To prove uniqueness, note that any derivation of "$a$ times $b$ equals $e$" (for any $e$) must have the form of $e$ appended to a derivation of "$a$ times $b-1$ equals $u$" for some $u$. But by induction that $u$ is $d$, so $e=u+a=d+a=c$.
But we can only even begin to do this after we've shown that all the relevant stuff about derivations is definable (so, again, our reasoning about sequences is preceding recursion).
One solution to this is to explicitly bake in the desired $\#$-operation. But we could also add definition to recursion to our underlying logic. The simplest approach to this yields least fixed point logic (LFP). The idea of LFP is to allow definitions which "build up in stages." Consider a formula $\varphi$ of a single variable $x$ in our language together with a new unary predicate symbol $U$. From $\varphi$ we get a map $m_\varphi$ on subsets of our structure $\mathcal{M}$: $$A\mapsto\{x:\mathcal{M}_A\models\varphi(x)\},$$ where $\mathcal{M}_A$ is the expansion of $\mathcal{M}$ gotten by interpreting $U$ as $A$.
Now if $U$ occurs only positively in $\varphi$, the map $m_\varphi$ is monotonic in that $A\subseteq B\implies m_\varphi(A)\subseteq m_\varphi(B)$. This means that there is a least fixed point of $m_\varphi$ - that is, a set $LFP_\varphi$ such that
$m_\varphi(LFP_\varphi)=LFP_\varphi$, and
whenever $m_\varphi(X)=X$ we have $X\supseteq LFP_\varphi$.
Essentially, we think of $\varphi$ as telling us how to go from a partial approximation to an object we're trying to build to a better approximation, and $LFP_\varphi$ is the set of all things we ever throw in. Least fixed point logic basically lets us build $LFP_\varphi$.
Of course, least fixed points don't always look how we might want them to. For instance, consider the linear order $\mathbb{N}+\mathbb{Z}$. There is a natural way to try to define the "even" elements by LFP: $$\varphi(x,U)\equiv [x=0]\vee [\exists y\in U(x=SSy)]$$ (where $S$ is the successor operation, which is of course definable). But $LFP_\varphi$ doesn't extend into the $\mathbb{Z}$-part at all: the least fixed point of $\varphi$ is the set of all standard even numbers only.
I don't know too much about least fixed point logic. It is a hot topic in computer science, but there the focus is especially on finite models. I believe its general model theory is well-studied but is quite complicated.