$\def\Spec{\mathop{\rm Spec}}
\def\R{{\bf R}}
\def\Ep{{\rm E}^+}
\def\L{{\rm L}}
\def\EpL{\Ep\L}$
One can argue that an object of the right category of spaces in measure theory is not a set equipped with a σ-algebra of measurable sets,
but rather a set $S$ equipped with a σ-algebra $M$ of measurable sets and a σ-ideal $N$ of negligible sets, i.e., sets of measure 0.
The reason for this is that you can hardly state any theorem of measure theory or probability theory without referring to sets of measure 0.
However, objects of this category contain less data than the usual measured spaces, because they are not equipped with a measure.
Therefore I prefer to call them enhanced measurable spaces, since they are measurable spaces enhanced with a σ-ideal of negligible sets.
A morphism of enhanced measurable spaces $(S,M,N)→(T,P,Q)$ is a map $S\to T$ such that
the preimage of every element of $P$ is a union of an element of $M$ and a subset of an element of $N$
and the preimage of every element of $Q$ is a subset of an element of $N$.
Irving Segal proved in “Equivalences of measure spaces” (see also Kelley's “Decomposition and representation theorems in measure theory”)
that for an enhanced measurable space $(S,M,N)$ that admits a faithful measure (meaning $μ(A)=0$ if and only if $A∈N$) the following properties are equivalent.
- The Boolean algebra $M/N$ of equivalence classes of measurable sets is complete;
- The space of equivalence classes of all bounded (or unbounded) real-valued functions on $S$ modulo equality almost everywhere is Dedekind-complete;
- The Radon-Nikodym theorem is true for $(S,M,N)$;
- The Riesz representation theorem is true for $(S,M,N)$ (the dual of $\L^1$ is isomorphic to $\L^∞$);
- Equivalence classes of bounded functions on $S$ form a von Neumann algebra (alias W*-algebra).
An enhanced measurable space that satisfies these conditions (including the existence of a faithful measure) is called localizable.
This theorem tells us that if we want to prove anything nontrivial about measurable spaces, we better restrict ourselves to localizable enhanced measurable spaces.
We also have a nice illustration of the claim I made in the first paragraph:
none of these statements would be true without identifying objects that differ on a set of measure 0.
For example, take a nonmeasurable set $G$ and a family of singleton subsets of $G$ indexed by themselves.
This family of measurable sets does not have a supremum in the Boolean algebra of measurable sets, thus disproving a naive version of (1).
But restricting to localizable enhanced measurable spaces does not eliminate all the pathologies:
one must further restrict to the so-called compact and strictly localizable enhanced measurable spaces,
and use a coarser equivalence relation on measurable maps: $f$ and $g$ are weakly equal almost everywhere
if for any measurable subset $B$ of the codomain the symmetric difference $f^*B⊕g^*B$ of preimages of $B$ under $f$ and $g$ is a negligible subset of the domain.
(For codomains like real numbers this equivalence relation coincides with equality almost everywhere.)
An enhanced measurable space is strictly localizable if it splits as a coproduct (disjoint union) of σ-finite (meaning there is a faithful finite measure)
enhanced measurable spaces.
An enhanced measurable space $(X,M,N)$ is (Marczewski) compact if there is a compact class $K⊂M$
such that for any $m∈M∖N$ there is $k∈K∖N$ such that $k⊂m$.
Here a compact class is a collection $K⊂2^X$ of subsets of $X$ such that for any $K'⊂K$ the following finite intersection property holds:
if for any finite $K''⊂K'$ we have $⋂K''≠∅$, then also $⋂K'≠∅$.
The best argument for such restrictions is the following Gelfand-type duality theorem for commutative von Neumann algebras.
Theorem.
The following 5 categories are equivalent.
- The category of compact strictly localizable enhanced measurable spaces with measurable maps modulo weak equality almost everywhere.
- The category of hyperstonean topological spaces and open continuous maps.
- The category of hyperstonean locales and open maps.
- The category of measurable locales (and arbitrary maps of locales).
- The opposite category of commutative von Neumann algebras and normal (alias ultraweakly continuous) unital *-homomorphisms.
I actually prefer to work with the opposite category of the category of commutative von Neumann algebras,
or with the category of measurable locales.
The reason for this is that the point-set definition of a measurable space
exhibits immediate connections only (perhaps) to descriptive set theory, and with additional effort to Boolean algebras,
whereas the description in terms of operator algebras or locales immediately connects measure theory to other areas of the central core of mathematics
(noncommutative geometry, algebraic geometry, complex geometry, differential geometry, topos theory, etc.).
Additionally, note how the fourth category (measurable locales) is a full subcategory of the category of locales.
Roughly, the latter can be seen as a slight enlargement of the usual category of topological spaces,
for which all the usual theorems of general topology continue to hold (e.g., Tychonoff, Urysohn, Tietze, various results about paracompact and uniform spaces, etc.).
In particular, there is a fully faithful functor from sober topological spaces (which includes all Hausdorff spaces) to locales.
This functor is not surjective, i.e., there are nonspatial locales that do not come from topological spaces.
As it turns out, all measurable locales (excluding discrete ones) are nonspatial.
Thus, measure theory is part of (pointfree) general topology, in the strictest sense possible.
The non-point-set languages (2–5) are also easier to use in practice.
Let me illustrate this statement with just one example: when we try to define measurable bundles of Hilbert spaces
on a compact strictly localizable enhanced measurable space in a point-set way, we run into all sorts of problems
if the fibers can be nonseparable, and I do not know how to fix this problem in the point-set framework.
On the other hand, in the algebraic framework we can simply say that a bundle of Hilbert spaces is a Hilbert W*-module over the corresponding von Neumann algebra.
Categorical properties of von Neumann algebras (hence of compact strictly localizable enhanced measurable spaces)
were investigated by Guichardet in “Sur la catégorie des algèbres de von Neumann”.
Let me mention some of his results, translated in the language of enhanced measurable spaces.
The category of compact strictly localizable enhanced measurable spaces admits equalizers and coequalizers, arbitrary coproducts, hence also arbitrary colimits.
It also admits products (and hence arbitrary limits), although they are quite different from what one might think.
For example, the product of two real lines is not $\R^2$ with the two obvious projections.
The product contains $\R^2$, but it also has a lot of other stuff, for example, the diagonal of $\R^2$,
which is needed to satisfy the universal property for the two identity maps on $\R$.
The more intuitive product of measurable spaces ($\R\times\R=\R^2$) corresponds to the spatial
tensor product of von Neumann algebras and forms a part of a symmetric monoidal structure on the category of measurable spaces.
See Guichardet's paper for other categorical properties (monoidal structures on measurable spaces, flatness, existence of filtered limits, etc.).
Another property worthy of mentioning is that the category of commutative von Neumann algebras
is a locally presentable category, which immediately allows one to use the adjoint functor theorem to construct commutative
von Neumann algebras (hence enhanced measurable spaces) via their representable functors.
Finally let me mention pushforward and pullback properties of measures on enhanced measurable spaces.
I will talk about more general case of $\L^p$-spaces instead of just measures (i.e., $\L^1$-spaces).
For the sake of convenience, denote $\L_p(M)=\L^{1/p}(M)$, where $M$ is an enhanced measurable space.
Here $p$ can be an arbitrary complex number with a nonnegative real part.
We do not need a measure on $M$ to define $\L_p(M)$.
For instance, $\L_0$ is the space of all bounded functions (i.e., the commutative von Neumann algebra corresponding to $M$),
$\L_1$ is the space of finite complex-valued measures (the dual of $\L_0$ in the ultraweak topology),
and $\L_{1/2}$ is the Hilbert space of half-densities.
I will also talk about extended positive part $\EpL_p$ of $\L_p$ for real $p$.
In particular, $\EpL_1$ is the space of all (not necessarily finite) positive measures on $M$.
Pushforward for $\L_p$-spaces.
Suppose we have a morphism of enhanced measurable spaces $M\to N$.
If $p=1$, then we have a canonical map $\L_1(M)\to\L_1(N)$, which just the dual of $\L_0(N)→\L_0(M)$ in the ultraweak topology.
Geometrically, this is the fiberwise integration map.
If $p≠1$, then we only have a pushforward map of the extended positive parts, namely, $\EpL_p(M)→\EpL_p(N)$, which is nonadditive unless $p=1$.
Geometrically, this is the fiberwise $\L_p$-norm.
Thus $\L_1$ is a functor from the category of enhanced measurable spaces to the category of Banach spaces
and $\EpL_p$ is a functor to the category of “positive homogeneous $p$-cones”.
The pushforward map preserves the trace on $\L_1$ and hence sends a probability measure to a probability measure.
To define pullbacks of $\L_p$-spaces (in particular, $\L_1$-spaces) one needs to pass to a different category of enhanced measurable spaces.
In the algebraic language, if we have two commutative von Neumann algebras $A$ and $B$,
then a morphism from $A$ to $B$ is a usual morphism of commutative von Neumann algebras $f\colon A\to B$
together with an operator valued weight $T\colon\Ep(B)\to\Ep(A)$ associated to $f$.
Here $\Ep(A)$ denotes the extended positive part of $A$.
(Think of positive functions on $\Spec A$ that can take infinite values.)
Geometrically, this is a morphism $\Spec f\colon\Spec B\to\Spec A$
between the corresponding enhanced measurable spaces and a choice of measure on each fiber of $\Spec f$.
Now we have a canonical additive map $\EpL_p(\Spec A)\to\EpL_p(\Spec B)$,
which makes $\EpL_p$ into a contravariant functor from the category of enhanced measurable spaces
and measurable maps equipped with a fiberwise measure to the category of “positive homogeneous additive cones”.
If we want to have a pullback of $\L_p$-spaces themselves and not just their extended positive parts,
we need to replace operator valued weights in the above definition
by finite complex-valued operator valued weights $T\colon B\to A$ (think of a fiberwise finite complex-valued measure).
Then $\L_p$ becomes a functor from the category of enhanced measurable spaces to the category of Banach spaces (if the real part of $p$ is at most $1$)
or quasi-Banach spaces (if the real part of $p$ is greater than $1$).
Here $p$ is an arbitrary complex number with a nonnegative real part.
Notice that for $p=0$ we get the original map $f\colon A\to B$ and in this (and only this) case we do not need $T$.
Finally, if we restrict ourselves to an even smaller subcategory defined by the additional condition $T(1)=1$
(i.e., $T$ is a conditional expectation; think of a fiberwise probability measure),
then the pullback map preserves the trace on $\L_1$ and in this case the pullback of a probability measure is a probability measure.
There is also a smooth analog of the theory described above.
The category of enhanced measurable spaces and their morphisms is replaced by the category of smooth manifolds and submersions,
$\L_p$-spaces are replaced by bundles of $p$-densities,
operator valued weights are replaced by sections of the bundle of relative 1-densities,
the integration map on 1-densities is defined via Poincaré duality (to avoid any dependence on measure theory) etc.
There is a forgetful functor that sends a smooth manifold to its underlying enhanced measurable space.
Of course, the story does not end here, there are many
other interesting topics to consider: products of measurable spaces,
the difference between Borel and Lebesgue measurability, conditional expectations, etc.
An index of my writings on this topic is available.
Best Answer
A different answer from the ones so far: Quantum randomness is another kind of randomness that is a generalization of traditional randomness, i.e. classical or non-quantum probability. I think that it fits the question because you could likewise say that non-Euclidean geometry, interpreted as not-necessarily-Euclidean geometry, is a generalization of Euclidean geometry.
A classical probability space is usually defined as a $\sigma$-algebra $\Omega$ with a normalized measure. From the Bayesian viewpoint the measure could equally well be called a "state". Now, a $\sigma$-algebra is the algebra of Boolean random variables with a certain set of axioms. But you can just as well write down axioms for $L^\infty(\Omega)$, the algebra of bounded complex random variables. In favorable cases, it is a commutative von Neumann algebra. In quantum probability you instead allow a non-commutative von Neumann algebra $\mathcal{M}$. Also, in standard quantum probability you keep the usual completed tensor product $\mathcal{M} \otimes \mathcal{N}$ as the model of a joint system. (Free probability theory is still quantum probability, but with a certain free product instead of a tensor product.) You also still have states, conditional states, joint states, correlations, generalized stochastic maps, etc.
Some of the variant models mentioned so far lead to different theorems, but generally give the same answers in combinatorial probability, questions like the birthday paradox or modeling games of chance. Quantum probability leads to a significantly different picture of combinatorial probability, generalizing the old one, but also allowing new answers such as violation of Bell's inequalities, covariance matrices that are Hermitian rather than real symmetric, new complexity classes such as BQP, etc.
Other variant models mentioned so far no longer give any answers for combinatorial probability, for instance models of forcing. But, part of the interest in probability is that it models real life. Amazingly, so does quantum probability; that was the central discovery of quantum mechanics when it was defined in the 1920s and 1930s.