In quantum mechanics textbooks which are a little more careful it is common to see it noted that a (pure) quantum state is not a vector $|\psi\rangle$ but rather a ray in Hilbert space, $c|\psi\rangle$ (for all $c \in \mathbb{C}$). Obviously, the axiom giving the probabilities for different measurements of different observables must be generalized to include a denominator which normalizes things.
In other (even more careful?) quantum mechanics textbooks (see e.g. the start of Chapter 2 of Ballentine's Quantum Mechanics: A Modern Development) it's noted that a state is to be represented by an operator $\rho$ which is a valid state as long as $\rho$ is nonnegative, of unit trace, and self-adjoint. This generalized definition works since the identification of pure states $\psi$ with $\rho = | \psi \rangle \langle \psi|$ allows for an isomorphic development of the theory. My question is: Is it valid, as in the pure state case noted above, to generalize the statement about $\rho$ above and to say that all operators $\rho'$ which are nonnegative and self-adjoint represent valid states, with the observation that every such $\rho'$ is related to a unique $\rho$ of unit trace by $\rho' = c\rho$ for some $c \in \mathbb{C}$. The axiom for averages of observables (which is enough to in turn obtain probability distributions as does Ballentine in Chap 2, if we make assumptions about functions of operators) is modified to something like, for an observable $R$, $\langle R \rangle = \textrm{Tr}(R\rho')/\textrm{Tr}(\rho')$ (and that we would recover the same results since $\textrm{Tr}(R\rho')/\textrm{Tr}(\rho') = \textrm{Tr}(Rc\rho)/\textrm{Tr}(c\rho) = c\textrm{Tr}(R\rho)/c\textrm{Tr}(\rho) = \textrm{Tr}(R\rho) $ for that corresponding unit trace operator $\rho$)?
Edit: As noted by J. Murray, the $c$ I mention should be positive (and real) so that all $\rho'$ remain nonnegative.
Best Answer
I don't really see any motivation for this. The standard model of quantum mechanics loosely works by modeling the state of a quantum mechanical system as an element of the underlying Hilbert space, and then computing dynamics via the Schrodinger equation.
As you say, a more careful treatment would note that in terms of physical content, $\psi$ and $c\psi$ for any nonzero $c\in \mathbb C$ represent precisely the same physical state. This identification is important, because otherwise there would be no possible way to decide how to model a system. Say a spin-1/2 system is prepared in the spin-up state. Is the initial state vector of the system given by $\pmatrix{1\\0}$ or $\pmatrix{-1\\0}$? Or maybe $\pmatrix{i\\0}$? The identification of all these vectors as representatives of the same physical state means that it doesn't matter - we can freely choose any of them.
As you say, though, this is insufficiently general because we'd like to allow "classical" statistical mixtures of such states. In an ensemble of pure states $|\psi_i\rangle$ with probabilites $p_i$, the appropriate choice is to use a density operator $$\rho = \sum_i p_i |\psi_i\rangle\langle\psi_i|$$ and the generalized rule $\langle A\rangle = \mathrm{Tr}(A\rho)$. We can define the set of all such $\rho$'s without reference to pure states by requiring that they be positive, trace-class operators with unit trace.
Your proposal is to enlarge the space of possible states by allowing them to be scaled by some nonnegative real number, but I don't see a reason to do so. The identification of physical states with rays in the Hilbert space is natural because vectors which are multiples of one another have the same physical content. The generalization to mixed states allows for statistical ensembles of pure states. The generalization to scaled density operators isn't physically necessary (in some sense it is the opposite of the identification $\psi \sim c\psi$) and doesn't actually get us anything useful.
Lastly, there is actually a further generalization if one adopts the algebraic formulation of quantum mechanics; in this picture, a state is given by a positive, normalized linear functional on a $C^*$ algebra, but this is much more technical and not particularly common.