Searching for materials on discrete probability I find "first course in probability" type stuff and some seemingly rather specific things. Is there a principled, step-by-step introduction to advanced discrete probabilistic topics (e.g. in a book)? Are there any advanced general theorems in this area at all?
Book on advanced discrete probability
book-recommendationdiscrete mathematicsprobabilityreference-request
Related Solutions
You could start with Bartle - The Elements of Integration and Lebesgue Measure. You could also check here and here.
I like the book Probability, Statistics, and Random Processes for Electrical Engineering by Leon-Garcia (I used it to teach probability last semester). Of course, it is Electrical Engineering focused. It supresses the esoteric measure theory material.
The title of your book is funny because my first reaction is "Who but a mathematician would ever need measure theory anyway?" Of course I am being a bit snarky with that comment. I do think it is important for advanced students to at least know what measure theory is about. However, I feel it is more important for students to know the difference between a countably infinite set and an uncountably infinite set. Knowing the difference is essential for measure theory, but unfortunately some courses skip this, assume you already know it, and cover the less essential topics of sigma algebras.
In my humble opinion, you don't really need measure theory for probability or stochastic processes, although measure theory is certainly the foundation for those things. Similarly, Russell's Principia Mathematica is a foundation for basic arithmetic, but most people who use arithmetic (including mathematicians) have never read it (and certainly arithmetic existed before it). So, it is possible for you to learn and use something without going into detail on foundations. It is also good to know those foundations exist, especially if you eventually want answers to lingering questions.
The term "almost surely" is the most important one that you listed. It is synonymous with "with probability 1." For example, if you have a random variable $X$ that is uniformly distributed over the interval $[0,1]$, then $Pr[X \neq 1/2]=1$ and so almost surely $X$ is not $1/2$. Similarly, since the rational numbers in $[0,1]$ can be listed as $\{q_1, q_2, q_3, \ldots\}$ we have:
$$ Pr[\mbox{$X$ is rational}] = \sum_{i=1}^{\infty}Pr[X=q_i]=0 $$
and so $Pr[\mbox{$X$ is irrational}]=1$, which means $X$ is almost surely irrational.
Some sets are so complicated that they cannot have probabilities assigned to them. The term measurable describes a set that has a valid probability. A theorem that says "assuming the set is measurable" is just being precise, and you can ignore that phrase without worry. It just means they are restricting to the case where probabilities are defined (you cannot prove theorems otherwise). All the crazy theorems about measurability are designed to basically show that all practical sets of interest are measurable. In that sense, measure theory is self-defeating, since its most important results ensure you can safely ignore them.
Perhaps the most practical topics in measure theory are the convergence theorems, such as the Lebesgue dominated convergence theorem. These theorems tell you when you are allowed to pass a limit through an integral or through an expectation.
A "sigma algebra" is a class of sets defined so that all sets in the class can be measured. The "standard Borel sigma algebra" is a very large class of sets of real numbers. It is always possible to define the probability that a random variable falls in one of those sets. Since those sets are so extensive, every practical set you will ever work with will indeed be measurable (unless you end up working on foundational mathematical subjects or axiom-of-choice related set theory subjects).
Best Answer
Discrete Probability Models and Methods, by Pierre Brémaud (Springer). Quoting from the introduction to the book: