There are many flavors of "meta" in logic. Most make very minimal use of the metatheory. For example, the Montague Reflection Principle in Set Theory says the following:
Metatheorem. For every formula $\phi(x)$ of the language of Set Theory, the following is provable: If $\phi(a)$ is true then there is an ordinal $\alpha$ such that $a \in V_\alpha$ and $V_\alpha \vDash \phi(a)$.
This is in fact an infinite collection of theorems, one for each formula $\phi(x)$. Gödel's theorem prevents ZFC from proving all of these instances at the same time. The proof of this metatheorem is by induction on the length of $\phi(x)$. The requirements on the metatheory are very minimal, all you need is enough combinatorics to talk about formulas, proofs, and enough induction to make sense of it all. This is by far the most common flavor of "meta" in logic. There really is not much to it. Most logicians don't worry about the metatheory in this context. Indeed, it is often irrelevant and, like most mathematicians, logicians usually believe in the natural numbers with full induction.
However, sometimes there is need for a much stronger metatheory. For example, the metatheory of Model Theory is usually taken to be ZFC. Gödel's Completeness Theorem is a nice contrasting example.
Completeness Theorem. Every consistent theory is satisfiable.
Recall that a theory is consistent if one cannot derive a contradiction form it, while a theory is satisfiable if it has a model. The consistency of a theory is a purely syntactic notion. When the theory is effective, consistency can be expressed in simple arithmetical terms in a manner similar to the one discussed above. However, satisfiability is not at all of this form since the models of a theory are typically infinite structures. This is why this theorem is usually stated in ZFC. For the Completeness Theorem, one can often get by with substantially less. For example, the system WKL0 of second-order arithmetic suffices to prove the Completeness Theorem for countable theories (that exist in the metaworld in question), but such a weak metatheory would be nothing more than a major inconvenience for model theorists.
Sometimes there are multiple choices of metatheory and no consensus on which is most appropriate. This happens in the case of Forcing in Set Theory, which has three common points of view:
Forcing is a way to extend a countable transitive model of ZFC' to a larger model of ZFC' with different properties. Here ZFC' is a finite approximation to ZFC, which makes this statement non-vacuous in ZFC because of the Reflection Principle above.
Forcing is a way to talk about truth in an alternate universe of sets. Here, the alternate universe is often taken to be a Boolean-valued model, formalized within the original universe.
Forcing is an effective way to transform a contradiction some type of extensions of ZFC into contradictions within ZFC itself. Here, ZFC could be replaced by an extension of ZFC.
The last point of view, which is the least common, is essentially arithmetical since it only talks about proofs. The second point of view is well suited for hard-core platonists who believe in one true universe of sets. The first point of view, which is probably most popular, essentially takes ZFC as a metatheory much like model theorists do.
Since there is no consensus, set theorists will often talk as if both $V$ and its forcing extension $V[G]$ are absolutely real universes of sets. Although this point of view is hard to justify formally, it is possible to makes sense of it using any one of the three viewpoints above and it has the advantage that it makes it easier to express ideas that go into forcing constructions, which is what set theorists really want to talk about.
Finally, the idea of analyzing which systems prove the consistency of other systems is very common in logic. In Set Theory, these systems often take the form of large cardinal axioms. In Second-Order Arithmetic these systems often take the form of comprehension principles. In First-Order Arithmetic these systems often take the form of induction principles. Together these form an incredibly long consistency strength hierarchy stretching from extremely weak basic arithmetical facts to incredibly deep large cardinal axioms. A very significant part of logic deals with studying this hierarchy and, as Andrej Bauer commented, logicians are usually very aware of where they are sitting in this hierarchy when proving metatheorems.
Raymond Smullyan gave a very general formulation in terms of representation systems. They appear in his "Theory of Formal Systems", and in the first and last chapters of "Godel's Incompleteness Theorems". They generalise first- and higher-order systems of logic, type theories, and Post production systems.
A representation system consists of:
A countably infinite set $E$ of expressions.
A subset $S \subseteq E$, the set of sentences.
A subset $T \subseteq S$, the set of provable sentences.
A subset $R \subseteq S$, the set of refutable sentences.
A subset $P \subseteq E$, the set of (unary) predicates.
A function $\Phi : E \times \mathbb{N} \rightarrow E$ such that, whenever $H$ is a predicate, then $\Phi(H,n)$ is a sentence.
The system is complete iff every sentence is either provable or refutable. It is inconsistent iff some sentence is both provable and refutable.
We say a predicate $H$ represents the set $A \subseteq \mathbb{N}$ iff $A = \{ n : \Phi(H,n) \in T \}$.
Let $g$ be a bijection from $E$ to $\mathbb{N}$. We call $g(X)$ the Godel number of $X$.
We write $E_n$ for the expression with Godel number $n$.
Let $\overline{A} = \mathbb{N} \setminus A$ and $Q^* = \{ n : \Phi(E_n,n) \in Q \}$.
We have:
(Generalised Tarski Theorem) The set $\overline{T^*}$ is not representable.
(Generalised Godel Theorem) If $R^*$ is representable, then the system is either inconsistent or incomplete.
(Generalised Rosser Theorem) If some superset of $R^* $ disjoint from $T^*$ is representable, then the system is incomplete.
In case it's not clear: in a first-order system, we can take $P$ to be the set of formulas whose only free variable is $x_1$, and $\Phi(H,n) = [\overline{n}/x_1]H$.
Best Answer
As Andres Caicedo points out in his comment (to the Question), the modest fragment $\sf{PRA}$ (Primitive Recursive Arithmetic) of $\sf{PA}$ (Peano arithmetic) is already is able to verify the incompleteness theorems.
It is worth noting that in the above $I\Delta_0 + exp$ can be even reduced to $I\Delta_0 + \Omega_1$, where $\Omega_1$ is the axiom asserting the totality of the function $2^{\left| x\right|^2 }$, where $\left| x\right|$ denotes the length of the binary expansion of $x$. The theory $I\Delta_0 + \Omega_1$ is commonly viewed as the weakest fragment of $\sf{PA}$ in which one can develop a workable "theory of syntax".
PS. As pointed out by Jeřábek, the incompleteness theorems can be implemented in even weaker systems.