A few days ago I was recalling some facts about the p-adic numbers, for example the fact that the p-adic metric is an ultrametric implies very strongly that there is no order on $\mathbb{Q}_p$, as any number in the interior of an open ball is in fact its center.
This argument is not correct. For instance why does it not apply to $\mathbb{Q}$ with the $p$-adic metric? In fact any field which admits an ordering also admits a nontrivial non-Archimedean metric.
It is true though that $\mathbb{Q}_p$ cannot be ordered. By the Artin-Schreier theorem, this is equivalent to the fact that $-1$ is a sum of squares. Using Hensel's Lemma and a little quadratic form theory it is not hard to show that $-1$ is a sum of four squares in $\mathbb{Q}_p$.
I know that if you take the completion of the algebraic close of the p-adic completion you get something which is isomorphic to $\mathbb{C}$ (this result was very surprising until I studied model theory, then it became obvious).
I don't mean to pick, but I am familiar with basic model theory and I don't see how it helps to establish this result. Rather it is basic field theory: any two algebraically closed fields of equal characteristic and absolute transcendence degree are isomorphic. (From this the completeness of the theory of algebraically closed fields of any given characteristic follows easily, by Vaught's test.)
So I was thinking, is there a $p$-adic number whose square equals 2? 3? 2011? For which prime numbers $p$?
All of these answers depend on $p$. The general situation is as follows: for any odd $p$, the group of square classes $\mathbb{Q}_p^{\times}/\mathbb{Q}_p^{\times 2}$ -- which parameterizes quadratic extensions -- has order $4$, meaning there are exactly three quadratic extensions of $\mathbb{Q}_p$ inside any algebraic closure. If $u$ is any integer which is not a square modulo $p$, then these three extensions are given by adjoinging $\sqrt{p}$, $\sqrt{u}$ and $\sqrt{up}$. When $p = 2$ the group of square classes has cardinality $8$, meaning there are $7$ quadratic extensions.
How far down the rabbit hole of algebraic numbers can you go inside the p-adic numbers? Are there general results connecting the choice (or rather properties) of $p$ to the "amount" of algebraic closure it gives?
I don't know exactly what you are looking for as an answer here. The absolute Galois group of $\mathbb{Q}_p$ is in some sense rather well understood: it is an infinite profinite group but it is "small" in the technical sense that there are only finitely many open subgroups of any given index. Also every finite extension of $\mathbb{Q}_p$ is solvable. All in all it is vague -- but fair -- to say that the fields $\mathbb{Q}_p$ are "much closer to being algebraically closed" than the field $\mathbb{Q}$ but "not as close to being algebraically closed" as the finite field $\mathbb{F}_p$. This can be made precise in various ways.
If you are interested in the $p$-adic numbers you should read intermediate level number theory texts on local fields. For instance this page collects notes from a course on (in part) local fields that I taught last spring. I also highly recommend books called Local Fields: one by Cassels and one by Serre.
Added: see in particular Sections 5.4 and 5.5 of this set of notes for information about the number of $n$th power classes and the number of field extensions of a given degree.
No, the different $p$-adic number systems are not in any way compatible with one another.
A $p$-adic number is a not a number that is $p$-adic; it is a $p$-adic number. Similarly, a real number is not a number that is real, it is a real number. There is not some unified notion of "number" that these are all subsets of; they are entirely separate things, though there may be ways of identifying bits of them in some cases (e.g., all of them contain a copy of the rational numbers).
Now, someone here is bound to point out that if we take the algebraic closure of some $\mathbb{Q}_p$, the result will be algebraically isomorphic to $\mathbb{C}$. But when we talk about $p$-adic numbers we are not just talking about their algebra, but also their absolute value, or at least their topology; and once you account for this they are truly different. (And even if you just want algebraic isomorphism, this requires the axiom of choice; you can't actually identify a specific isomorphism, and there's certainly not any natural way to do so.)
How can we see that they are truly different? Well, first let's look at the algebra. The $5$-adics, for instance, contain a square root of $-1$, while the $3$-adics do not. So if you write down a $5$-adic number which squares to $-1$, there cannot be any corresponding $3$-adic number.
But above I claimed something stronger -- that once you account for the topology, there is no way to piece the various $p$-adic number systems together, which the above does not rule out. How can we see this? Well, let's look at the topology when we look at the rational numbers, the various $p$-adic topologies on $\mathbb{Q}$. These topologies are not only distinct -- any finite set of them is independent, meaning that if we let $\mathbb{Q}_i$ be $\mathbb{Q}$ with the $i$'th topology we're considering, then the diagonal is dense in $\mathbb{Q}_1 \times \ldots \times \mathbb{Q}_n$.
Put another way -- since these topologies all come from metrics -- this means that for any $c_1,\ldots,c_n\in\mathbb{Q}$, there exists a sequence of rational numbers $a_1,a_2,\ldots$ such that in topology number 1, this converges to $c_1$, but in topology number two, it converges to $c_2$, and so forth. (In fact, more generally, given any finite set of inequivalent absolute values on a field, the resulting topologies will be independent.)
So even on $\mathbb{Q}$, the different topologies utterly fail to match up, so there is no way they can be pieced together by passing to some larger setting.
Best Answer
One of my favourite classical results using $p$-adic methods in elementary number theory is the theorem of Skolem-Mahler-Lech:
This is a theorem about linear recurrence sequences, which are sequences of integers where each term is a fixed linear combination of $n$ previous ones. So fixing $n$ the sequence $s_i$ is defined by choosing the first $n$ terms $$s_0,\ldots, s_{n-1}\in \mathbf Z$$ and a relation for all $k$ $$s_{k + n} = \sum_{i=0}^{n-1} a_i s_{k+i}$$ for fixed $a_i$.
Some examples are the Fibonacci sequence ($n = 2$,$s_0 = 0, s_1 = 1$, $a_0=a_1= 1$), and simpler things like any eventually periodic sequence, or the sequence $s_k = k$ (here $n=2$, $s_0 = 0, s_1=1$, $a_0 = -1, a_1= 2$). We can make other such sequences easily by noting that that the sum of any two linear recurrence sequences is also a linear recurrence sequence.
An important fact about such sequences are that their generating functions $$f_s = \sum_{k= 0}^\infty s_k x^k$$ are always rational functions of the variable $x$ (one polynomial divided by another), where the numerator defines the initial terms $s_0, \ldots, s_{n-1}$ and the denominator defines the recurrence relation.
Of the examples I mentioned above, the fibonacci sequence grows (exponentially), eventually periodic sequences are bounded, and the sequence $s_k=k$ also grows, just less quickly than fibonacci.
One question that one might then ask is:
from these examples (and others) we might conjecture that this set is periodic, except for finitely many exceptions (after all we can always change finitely many terms of any linear recurrence sequence to make a sequence with the same behaviour eventually but with zeroes wherever we want at the start).
How might one go about proving this? The first step of the proof is to use the rational generating function $f_s$ and write out its partial fraction decomposition over an algebraically closed field (like $\overline {\mathbf Q}$), this will be of the form
$$f_s = \sum_{i=1}^{\ell} \sum_{j = 1}^{r_i} \frac{\alpha_{ij}}{(x - \beta_{i})^j} $$
for some fixed roots $\beta_j$ of the original denominator of $f_s$.
Now using this decomposition we have $$f_s = \sum_{i=1}^{\ell} \sum_{j = 1}^{r_i} \alpha_{ij}{\left(\sum_{n=0}^\infty \beta_i^n x^n\right)^j} $$
what this gives is that $$s_n = \text{some polynomial expression involving terms }\beta^n $$
For example for the fibonacci sequence this recovers Binet's formula $$s_n = \frac{1}{\sqrt 5} \left(\frac{1+ \sqrt 5}2\right)^n-\frac{1}{\sqrt 5} \left(\frac{1- \sqrt 5}2\right)^n.$$ Or for the periodic sequence $0,1,0,1,0,1,\ldots$ this is $$ s_n = 1^ n - (-1)^n$$
So we've written $s_n$ as a sum of exponential-type functions in $n$ with different bases, that we want to describe the zeroes of this function for $n \in \mathbf N$.
Now the magic part: the function $e^x$ is an analytic function, and on a bounded domain analytic functions have only finitely many zeroes (unless they are zero everywhere). This would give us a lot of control over the zeroes of $s_n$ if the naturals were bounded. Which leads to the slightly strange question:
Of course using the usual absolute value and metric on $\mathbf Q$ and $\mathbf C$ this is totally false.
But in the $p$-adic numbers this is true! The integers are all bounded ($p$-adically) by norm $\le 1$. So lets treat these functions as $p$-adic functions and control the zero sets in some way.
How does this prove the result? The functions $\beta^n$ are not $p$-adic analytic functions of $n$ on their own, but they are so on small enough $p$-adic disks though, what ends up happening is we get distinction between congruence classes of $n$ mod $p-1$ for some well-chosen $p$ such that in each congruence class there are only finitely many zeroes of $s_n$ or the function $s_n$ is identically zero on that congruence class. This gives us the theorem mentioned above, that the zeroes of $s_n$ are periodic, except for finitely many exceptions.