Definitions used:
Given some set $X$, denote by $\beta X$ the set of ultrafilters on $X$. We can view $X$ as a subset of $\beta X$ by identifying each point $x \in X$ with the principal ultrafilter associated to $x$. Further, the collection of all sets of the form $\widehat{A} = \{ \mathcal{U} \in \beta X\ |\ A \in \mathcal{U} \}$ supplies us with a basis for a topology on $\beta X$.
If we view $X$ as a discrete space and equip $\beta X$ with the topology generated by the aforementioned basis, then $\beta X$ is termed the Stone-Čech compactification of $X$. The space $\beta X$ is both compact and Hausdorff; the proofs of both of these facts are routine.
My question:
Purely formally, I feel that I understand all of the above, in the sense that I can solve a problem of the kind "given the set $X$, define the set $\beta X$ as above; verify that what we claim to be a basis is indeed a basis, and show that under the topology generated by this basis $\beta X$ is both compact and Hausdorff" painlessly.
My issue is that I have no sense of why we might expect $\beta X$ to be compact Hausdorff in the first place. Is this something that could be anticipated, or is it just a happy accident that we find when asking ourselves (quite reasonably) whether or not $\beta X$ has these properties?
My intuition:
In general, I think of compactifications as "adding points at infinity" so that anything that wants to "escape to infinity" gets "captured". This is pretty vague, but works reasonably well for (e.g.) the Alexandroff compactification. The Alexandroff compactification of $\mathbb{R}$ is just $S^1$, so that anything that can run away to infinity in $\mathbb{R}$ ends up "looping back around" in $S^1$.
I have less intuition for (ultra)filters, and tend to think of filters as being prescriptions for saying when certain subsets of a set $X$ are "large enough", and of an ultrafilter extending a given filter as being a (not, in general, unique) way of consistently extending that prescription to apply to all subsets of $X$. This basically came from the answers of Henning Makholm and Asaf Karagila to this math.SE question.
Unfortunately, I'm struggling to use this understanding of ultrafilters and compactifications to see why we would suspect that $\beta X$ is compact Hausdorff before outright proving it.
(I apologise for such a vague question being so long: given how what I ask is in some senses nebulous and non-concrete, I wanted to be completely explicit about both the definitions I use and the intuition that I already have).
Best Answer
Since you already know about the Alexandroff one-point compactification, let me begin by saying that the Stone-Cech compactification is at the other extreme, adding as many points at infinity as possible. To see what that could mean, let's consider some other compactifications of $\mathbb R$, starting with the most familiar, the extended real line, obtained by adjoining the two points $+\infty$ and $-\infty$ at the two ends of the line. Compared with the Alexandroff compactification, we're now distinguishing two different ways of "going to infinity". Some sequences that converged to $\infty$ in the Alexandroff compactification fail to converge in the extended real line because part of the sequence goes to the left and part to the right (e.g., $(-1)^nn$).
This idea can be extended, to produce "bigger" compactifications. If you visualize $\mathbb R$ as embedded in the plane as the graph of the sine function and then take its closure in the extended plane $(\mathbb R\cup\{+\infty,-\infty\})^2$, you get a compactification with a whole line segment at $+\infty$ (and another at $-\infty$). Similarly, embedding $\mathbb R$ in $3$-dimensional space as a helix, by $x\mapsto (x,\cos x,\sin x)$, we get a compactification with circles at the ends. Proceeding analogously with all bounded continuous functions $\mathbb R\to\mathbb R$ (in place of $\cos$ and $\sin$), in a very high-dimensional space (in fact, $2^{\aleph_0}$ dimensions), you get one of the standard constructions of the Stone-Cech compactification of $\mathbb R$. Roughly speaking, it separates, into different points at infinity, all of the possible "ways to go to $\infty$" in $\mathbb R$.
The analogous story works for discrete spaces $X$ in place of $\mathbb R$ (except that I don't need to say "continuous" because all functions on a discrete space are continuous). A point in the Stone-Cech remainder of a discrete space $X$ should be thought of as a "way to go to $\infty$" in $X$. But how can such "ways" be described?
Well, in any compactification, each point $p$ at infinity is in the closure of the original space $X$, and so we can describe its location relative to $X$ by the trace on $X$ of its neighborhood filter, i.e. by $\mathcal F=\{U\cap X: p\in\text{interior}(U)\}$ (where $U$ refers to subsets of the compactification). Note that $\mathcal F$ can't contain any finite subsets of $X$, because such subsets are closed in the compactification (as in any $T_1$ space) and thus disjoint from suitable neighborhoods of any point $p$ at infinity.
For the Stone-Cech compactification of a discrete space $X$, this filter $\mathcal F$ must have one additional property, namely that we cannot have two disjoint subsets $A,B$ of $X$ both meeting all the sets in $\mathcal F$. The reason is that then "going to infinity" in $A$ and in $B$ would be two different ways to go to infinity, both leading to the same point $p$.
This additional property of the filter $\mathcal F$ is equivalent to saying that $\mathcal F$ is an ultrafilter. So this is how ultrafilters enter the picture of Stone-Cech compactifications of discrete spaces.
One then ordinarily continues by saying that, (1) since the filter $\mathcal F$ associated with any $p$ at infinity is an ultrafilter, and different $p$'s must correspond to different ultrafilters (in order for the compactification to be Hausdorff), we might as well identify the points $p$ with the ultrafilters $\mathcal F$, and (2) for the sake of uniformity, we might as well identify the principal ultrafilters (which haven't been used yet) with the points of $X$. With these conventions, one can prove (using compactness) that every ultrafilter on $X$ gets identified with a point in the Stone-Cech compactification of $X$. So the Stone-Cech compactification of a discrete space can be identified with the set of ultrafilters on $X$. Finally, one should verify that the topology is necessarily the one you described.