First off, the distinction you're attempting to between "counting numbers" and "natural numbers" doesn't match how the words is usually used in mathematics. To a mathematician, "the natural numbers" in general means the pre-formal concept you're calling "counting numbers". (And the phrasing "counting numbers" generally isn't used at all). In particular, being a model of this or that formal system doesn't qualify anything as being the natural numbers; only the true Platonic natural numbers are natural numbers.
(That's except for people who have learned enough set theory to know how natural numbers are usually modeled there, but haven't yet gained a sufficiently wide horizon on the matter to know that the model is not the actual thing).
Whether the natural numbers are a human invention or not is something of a philosophical morass that I won't go into -- but the mere fact that they were not created by Peano is not a controversial statement.
Now, since we understand the naturals intuitively, at least to a pretty good extent, what do we need axioms for? Well, most primitively, to see how well we can do by reasoning purely mechanically from axioms. For example, it led to a significant improvement in geometrical reasoning when the ancient Greeks started demanding rigorous proofs from axioms rather than just geometrical intuition; it is natural to wish to know whether something similar could be done for arithmetic.
As it happens, it took many centuries after the Greeks before formal mathematical reasoning had progressed enough that it was thinkable that it might be possible to reason about integer arithmetic without depending on intuition. Peano's axioms were not aimed at telling anything new about the integers -- he was simply one of the first to have a sufficiently well developed notion of formal logic that he could hope to make an axiomatic treatment of the integers work.
The goal of the investigation, then, is not primarily to discover something new about the integers themselves, but to see whether a formal system can be made to support a "mechanical" reasoning about them that is rich and strong to conclude what we already know about them intuitively.
The Peano axioms (and their modern first-order successor, Peano Arithmetic) were somewhat successful in that, but nonetheless the crowning achievement of the whole program turned out to be a negative result: Gödel's Incompleteness Theorem tells us that every reasonable formal system for proving things about the integers will be incomplete: there are truths about the natural numbers that it cannot prove.
Nevertheless, it is still interesting in itself to investigate the strength and properties of various such formal systems.
Also, of course, it is pragmatically useful that when something can be proved in a formal system with few axioms to it, then it is very certain to be true about the actual naturals ... there's a much smaller risk of having overlooked something during the proof than if we base our reasoning purely on an intuitive understanding of how the naturals ought to behave. In this way it's kind of a gold standard for a proof that it can (in principle) be reduced to Peano Arithmetic or a similarly bare-bones system. We recognize that there will be things that can't, but these exceptions will be subject to a lot more scrutiny before one considers oneself convinced that they are in fact true.
Here is a proof of the weak claim. Let $X$ be any nonstandard model of PA with an initial segment $I$ containing all standard numbers which is closed under addition and multiplication but not exponentiation. Let $C$ be the set of $x\in X$ such that $x\leq n^i$ for some standard $n$ and some $i\in I$. Note that $C$ is closed under addition and multiplication: if $x\leq n^i$ and $y\leq m^j$, then $xy$ and $x+y$ are both at most $(m+n)^{i+j}$. So, we can take $X$ as a model of PA' with this $C$, and your set $M$ will be $C$.
Note also that $C$ is closed under exponentiation to elements of $I$, since if $x\leq n^i$ then $x^j\leq (n^i)^j=n^{ij}$. It follows that your set $E$ contains $I$. On the other hand, if $e\in E$, then in particular $2^e\in C$ so $2^e\leq n^i$ for some standard $n$ and some $i\in I$. But we have $(2^m)\geq n$ for some standard $m$, and so $n^i\leq 2^{mi}$ and so $e\leq mi$. Thus $e\in I$.
Thus your set $E$ for this model is just $I$. Since $I$ was chosen to not be closed under exponentiation, this proves the weak claim.
Best Answer
The bare bones answer is something like what Hagen has said. The idea is this: Exponentiation is understood to be a function defined recursively: $y=2^x$ iff there is a sequence $t_0,t_1,\dots,t_x$ such that
In this respect, exponentiation is hardly unique: $y=x!$ is defined similarly, for example. Now you'd say that there is a sequence $z_0,z_1,\dots,z_x$, such that
(That $t_0=z_0=1$ is coincidence. In one case it is because $2^0=1$; in the other, because $0!=1$.)
So, to define a formula saying that $y=2^x$, you'd like to say that there is a sequence $t_0,\dots,t_x$ as above.
The problem, of course, is that in Peano Arithmetic one talks about numbers rather than sequences. Gödel solved this problem when working on his proof of the incompleteness theorem: He explained how to code finite sequences by numbers, by using the Chinese remainder theorem. Recall that this result states that, given any numbers $n_1,\dots,n_k$, pairwise relatively prime, and any numbers $m_1,\dots,m_k$, there is a number $x$ that simultaneously satisfies all congruences $$ x\equiv m_i\pmod {n_i} $$ for $1\le i\le k$.
In particular, given $m_1,\dots,m_k$, let $n=t!$ where $t=\max(m_1,\dots,m_k,k)$. Letting $n_1=n+1$, $n_2=2n+1,\dots$, $n_k=kn+1$, we see that the $n_i$ are relatively prime, and we can find an $x$ that satisfies $x\equiv m_i\pmod{n_i}$ for all $i$. We can then say that the pair $\langle x,n\rangle$ codes the sequence $(m_1,\dots,m_k)$. In fact, given $x,n$, it is rather easy to "decode" the $m_i$: Just note that $m_i$ is the remainder of dividing $x$ by $in+1$.
Accordingly, we can define $y=2^x$ by saying that there is a pair $\langle a,b\rangle$ that, in the sense just described, codes a sequence $(t_0,t_1,\dots,t_x)$ such that $t_0=1$, $t_x=y$, and $t_{n+1}=2t_n$ for all $n<x$. (Again, "in the sense just described" ends up meaning simply that "the remainder of dividing $a$ by $ib+1$ is $t_i$ for all $i\le x$". Note that we are not requiring $b$ to be the particular number we exhibited above using factorials.)
Of course, one needs to prove that any two pairs coding such a sequence agree on the value of $t_x$, but this is easy to establish. And we can code a pair by a number using, for example, Cantor's enumeration of $\mathbb N\times\mathbb N$, so that $\langle a,b\rangle$ is coded as the number $$ c=\frac{(a+b)(a+b+1)}2+b. $$ This is a bijection, and has the additional advantages that it is definable and satisfies $a,b\le c$ (so it is given by a bounded formula).
An issue that appears now is that we need to formalize the discussion of the Chinese remainder theorem and the subsequent coding within Peano arithmetic. This presents new difficulties, as again, we cannot (in the language of arithmetic) talk about sequences, and cannot talk about factorials, until we do all the above, so it is not clear how to prove or even how to formulate these results.
This problem can be solved by noting that we can use induction within Peano arithmetic. One then proceeds to show, essentially, that given any finite sequence, there is a pair that codes it, and that if a pair codes a sequence $\vec s$, and a number $t$ is given, then there is a pair that codes the sequence $\vec s{}^\frown(t)$. That is, one can write down a formula $\phi(x,y,z)$, "$y$ codes a sequence, the $z$-th member of which is $x$", such that PA proves:
In fact, we can let $\phi$ be a bounded formula: Take $\phi(x,y,z)$ to be
Once we have in PA the existence of coded sequences like this, implementing recursive definitions such as exponential functions is straightforward.
There are two excellent references for these coding issues and the subtleties surrounding them: