It is well known that the set of density operators $\{\rho\}$ for a quantum theory form a convex set. As I have seen them defined, we simply say that a state corresponds to some linear operator $\rho$ which is self-adjoint, positive semidefinite, and of unit trace.
Now it is often taken in books as obvious that, for a situation (I'm struggling to make this precise which is part of the question I suppose) in which there is some "classical" uncertainty about the state of some quantum system which may be, with probability $p_n$, in some pure state $|\psi_n\rangle \langle \psi_n|$, we must represent the state of the system via a weighted sum of these pure states. More precisely, we take
$$\rho = \sum_n p_n |\psi_n\rangle \langle \psi_n|, \sum p_n = 1, p_n \in [0,1] \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (A)$$
Is there any reason other than "experiment confirms this choice" that we might take this? Obviously this also depends on exactly which physical contexts imply this choice for $\rho$. For instance, if I enter a laboratory and am unsure if a spin has been prepared up or down but I know it is in one or the other, is it correct for me to assign $\rho = \frac{1}{2}|+\rangle \langle +|+\frac{1}{2}|-\rangle \langle -|$? It seems absurd since this cannot reproduce the statistics of measurements of the actual pure state.
Thus to rephrase and try to be more precise, my questions are:
(1) When is (A) the correct choice for $\rho$?
(2) Can we prove that it is the correct choice without appealing to experiment? NB that I understand that this $\rho$ is a valid state since $\{\rho\}$ forms a convex set; thus my question here is not "why is it a valid state" but rather "why is it the valid state for this physical situation"?
Best Answer
The heuristic I've seen before is to stipulate that there should be two kinds of averages,
a coherent (quantum) one in which we take expectation values as $\langle \psi | \hat{A} | \psi \rangle$, and
an incoherent (classical) one in which we take averages over a probability distribution in the usual way, i.e., $\bar{a} = \sum_n p_n a_n$.
Then, if you have a system that could be in multiple quantum states $|\psi_n\rangle$---and you don't know which one but you can assign a probability distribution $p_n$ to them---the average is given by the "double" average $$ \overline{\langle \hat{A} \rangle} = \sum_n p_n \langle\psi_n|\hat{A}|\psi_n\rangle\,. $$ From here, it's a matter of rearranging things. First, choose any orthonormal basis $|m\rangle$, and resolve the identity in this expression as $$ \overline{\langle \hat{A} \rangle} = \sum_n p_n \langle\psi_n|\hat{A}|\psi_n\rangle = \sum_n p_n \langle\psi_n|\hat{A}\left( \sum_m |m\rangle\langle m | \right)|\psi_n\rangle\,. $$ Rearranging, we get $$ \overline{\langle \hat{A} \rangle} = \sum_m\sum_n p_n \langle\psi_n|\hat{A} |m\rangle \langle m |\psi_n\rangle = \sum_m\sum_n p_n \langle m |\psi_n\rangle\langle\psi_n|\hat{A} |m\rangle \,. $$ Finally, push the sum over $n$ through to get $$ \overline{\langle \hat{A} \rangle} = \sum_m \langle m |\left( \sum_n p_n|\psi_n\rangle\langle\psi_n|\right)\hat{A} |m\rangle =\operatorname{Tr}\left(\rho\hat{A}\right)\,, $$ where $$ \rho = \sum_n p_n|\psi_n\rangle\langle\psi_n|\,. $$