Total probability as the sum of conditional probabilities

probabilityprobability theory

Suppose I have 2 events $C_1$ and $C_2$ so that $C_1$, $C_2$, and their complements ($C_1'$ and $C_2'$) partition the probability space. Let the event I'm interested in be denoted by $X$. $C_1$ and $C_2$ and their complements are conditionally independent given $X$. The probability of $X$ is given by,

Best Answer

Indeed, if you do have that conditional independence, then that will be okay.

PS: It is the non-empty pairwise intersections of the sets and their complements that partition the space.

Related Solutions

[Math] Understanding conditional independence of two random variables given a third one

The first definition is the informal one, but at the same time seems rather convoluted to me.

I'd prefer: X and Y are conditionally independent with respect to a given Z iff

$P(X \; Y | Z) = P(X | Z ) P(Y | Z)$

Recall that conditioning one (or several) variables on the value of another, is (informally) the same as restricting the whole universe to a part of it. Then, if you are given the value of $Z$, you can think as if you are defining new variables that are the same as the unconditioned but that are restricted to our new (smaller universe) $X' \equiv X | Z$ $Y' \equiv Y | Z$ The above formula simply states that $X'$ and $Y'$ are independent.

The first definition says the same, but applying (in words) the property that two variables are independent iff their conditioned probabilities are the same as the unconditioned : $A$ indep $B$ iff $P(A | B ) = P (A)$

Did Kolmogorov’s probability “experiments” survive in the modern theory

It seems that the "experiments" of Kolmogorov have been replaced by a more general notion of "independence between classes." Let $(\Omega, \mathscr{F}, \mathbf{P})$ be a probability space and let $\mathscr{C}_\alpha \subset \mathscr{F}$ be "classes of events." We say these clases are mutually independent if any choice of events $\mathrm{C}_\alpha \in \mathscr{C}_\alpha$ results in the events $\mathrm{C}_\alpha$ to be independent in the usual sense (that means that for any finite choice of different indices $\alpha_i,$ $1 \leq i \leq p,$ we have $\mathbf{P}\left(\bigcap\limits_{i = 1}^p \mathrm{C}_{\alpha_i} \right) = \prod\limits_{i = 1}^p \mathbf{P}(\mathrm{C}_{\alpha_i})$). A well-known result (do it, is not too hard!) shows that we can enlarge each $\mathscr{C}_\alpha$ by adding:

Proper differences in $\mathscr{C}_\alpha$ (i.e. add $\mathrm{A} - \mathrm{B}$ where $\mathrm{B} \subset \mathrm{A}$ both belong to $\mathscr{C}_\alpha$).
The sets $\varnothing, \Omega.$
Countable disjoint unions of sets in $\mathscr{C}_\alpha.$
Limits of monotone sequences in $\mathscr{C}_\alpha$ (i.e. add $\lim\limits_{n \to\infty} \mathrm{A}_n$ where each $\mathrm{A}_n \in \mathscr{C}_\alpha$ and either $\mathrm{A}_n \subset \mathrm{A}_{n+1}$ for all $n$ or $\mathrm{A}_{n+1} \subset \mathrm{A}_n$ for all $n$ ).
Intersections cannot be added (look for a simple counter-example). However, if we further assume that the classes are $\pi$-systems (each one is closed under intersections; of course, you can enlarge your classess using 1-4 to get a $\pi$-system if possible and if the original classes really aren't $\pi$-systems by say, lacking $\varnothing$ in them and having two disjoint events in them), then intersections can be added and therefore (by monotone class theorem), the sigma-fields $\mathscr{F}_\alpha = \sigma(\mathscr{C}_\alpha)$ are independent!

Kolmogorov is using $\mathscr{C}_j = \mathfrak{A}^{(j)}$ ($1 \leq j \leq n$) as finite partitions of $\Omega$ and all 5 aformentioned enlargements apply.

You also asked about what is the importance of independence and conditional probability. To begin with, independence is the cornerstone of probability theory and it is independence that distinguishes probability theory from measure theory. Independence is intimately linked to conditional probability as the definition of independence is the same as saying that conditioning has no probabilistic effect (i.e. the measure remains the same, even when we are taking a subset of the space). For instance, if $E \subset \Omega$ is an event, then we can induce a canonical probability measure on $E$ by stating $P|_E(A) = c P(A)$ for $A \subset E$ measurable and where $c$ is a proportionality constant guaranteeing that we have a real probability measure on $E.$ Clearly, $c = 1/P(E).$ The effect of this canonical construction is that we have changed the underlying measurable space from $(\Omega, \mathscr{F})$ to $(E, \mathscr{F}_E := \{A \cap E \mid A \in \mathscr{F}\}).$ This is okay but an analytically better approach is to fix the measurable space $(\Omega, \mathscr{F})$ and then define the induced measure conditional on the occurence of $E$ as $P(\cdot \mid E) = P(A \cap E) / P(E)$ for $A \in \mathscr{F}.$ In this way, we can consider many induced measures simultaneously and everything is good as each of them will be defined on the same measurable space. Anyway, independence connects with conditional probability in that $P(A \mid E) = P(A)$ is equivalent to independence providede $P(E) > 0$ (if $P(E) = 0,$ $E$ is already independent of every event). Unless you study probability theory, the effects of independence are hard to grasp. Perhaps the best known set of examples are those of Markov processes (these are the probabilistic analog of classical mechanics where knowing the state of the process at some time allows to drop or ignore the past as it is not useful to predicting the future) or of Renewal Theory (processes such that the occurrence of an event makes the process start afresh, the classical example is random walk returning to zero). An important aspect of independence is that the process start afresh in a probabilistic sense, not in a $\omega$-by-$\omega$ sense.

I would like to add that sometimes statisticians and physicists miscontrue the definition of conditional probabilities in a rather fundamental way. (The Bayesian statisticians are particularly guilty here.) They think of probability as a measure of belif in something (whatever this means) and of conditioning is making a hypothesis. In this sense, conditional probability is (mis)construe as "updating one's beliefs in the occurrence of an experiment knowing a hypothesis is true." The problem with this interpretation is that probability theorem is based on measure theory and the theorems of probability are those of frequency over a long run of independent repetitions. In other words, probability measure really are modelling the notion that if we repeat an experiment $n$ times, then a fraction $p$ of those time the experiment will be success (e.g. The Law of Large Number or the Ergodic Type Theorems). The statistical thinking of belief is undefined as far as I am concerned and if it is defined but is not fully equivalent to this frequentist notion, then they are using the wrong mathematical tool. (I have been unfortunate enough to read PhD thesis of Statisticians or Physicists who go on with deep philosphical debates over whether or not Kolmogorov axioms "are correct." Of course they are, mathematics is concerned only with correctness within itself. Whether or not the person who wants to apply mathematical tools to real-world problems is miscontruing definitions or interpretation is of no mathematical concern.) Anyway, this is already too long...

Best Answer

Related Solutions

[Math] Understanding conditional independence of two random variables given a third one

Did Kolmogorov’s probability “experiments” survive in the modern theory

Related Question