Given a quantum system with associated Hilbert space $\mathcal H$, the set of all self-adjoint bounded operators is $\newcommand{\bh}{\mathcal B(\mathcal H)_\text{sa}}\bh$. In general, only a small subset of $\bh$ will represent physically observable operators. For infinite-dimensional systems, $\bh$ is huge and there's no hope ever finding experiments for all its members; even in finite-dimensional systems it is very challenging to find experimental schemes sensitive to even a vector-space basis for $\bh$.
The physical approach to this is to begin with a finite set of operators which you know you can measure. For a single free particle, for example, you'd take position and momentum; for a finite set of spins you'd take all their Pauli matrices. You then form the set $\mathcal A$ of all operators that can be formed from them via products and linear combinations, which has the structure of a $\mathcal C^\ast$ algebra, and that is your set of physical observables. The $\mathcal C^\ast$algebra itself is the really fundamental description of the system; the Hilbert space is simply one possible representation.
In this formalism, states are functionals on $\mathcal A$: they are functions
$$\rho:\mathcal A\rightarrow \mathbb C $$
that take an observable and give its measured value (or probable measured value, etc.) in that state. (In a Hilbert space representation, each such functional is associated with a density matrix $\hat\rho$, a trace-class positive operator such that $\rho(A)=\text{Tr}(\hat\rho\hat A)$ for $\hat A$ the Hilbert space operator associated with an arbitrary $A\in\mathcal A$.
Edit: As joshphysics and WetSavannaAnimal rightly point out, this works as stated only for bounded operators and not for unbounded ones like position or energy. I'm afraid I don't know well enough how this extends to that class of operators - that needs someone with much stronger functional analysis chops than mine.
A first remark: the term "Hermitian", even if very popular in physics is in my opinion quite misleading (because someone uses it for symmetric operators, others for self-adjoint ones).
A second remark: the self-adjoint operators of a given Hilbert space $\mathscr{H}$ are in one-to-one correspondence with the strongly continuous groups of unitary operators; not with any group of unitary operators. So it is not possible to associate observables with "unitary operators", but it is possible to associate them with strongly continuous (abelian, locally compact) groups of unitary operators.
These distinctions, even if in some sense subtle, may be important. In fact there are representations of unitary groups that does not admit a self-adjoint generator; for example the canonical commutation relations (in the exponentiated Weyl form) have such "non-regular" representations for fields, and are physically related to infrared problems (see e.g this link).
Concerning observables, the point is that it is quite difficult to give a satisfactory algebraic setting in order to collect together observables that are unbounded (as they actually are the majority of physically relevant quantities: e.g. energy, momentum...). One option is to construct an algebra of unbounded operators, but there are all kinds of domain "nightmares" to be taken into account. Another is to consider an algebra of bounded operators (a $C^*$ or von Neumann algebra), and "affiliate" unbounded self-adjoint operators to it in a suitable fashion. Both procedures are not, in my opinion, completely satisfactory; anyways the algebraic approach gives a very nice framework to understand some of the aspects of quantum theories, especially representations of groups of operators.
My personal point of view is to consider any self-adjoint operator on a given Hilbert space (usually a suitable representation of a $C^*$ algebra, or its bicommutant) as an observable. This choice is justified from the fact that any real-valued physically measurable quantity that is actually measured by physicists behaves like a self-adjoint operator (and not a symmetric one); in particular it has (mathematically speaking) an associated spectral family, as it is the case for self-adjoint operators but not for symmetric ones.
A last mathematical comment: Of course you can associate to a given self-adjoint operator $A$ a strongly continuous unitary group $e^{itA}$; and for example construct the $C^*$ algebra $\{e^{itA},t\in\mathbb{R}\}\overline{\phantom{ii}}$; where the bar stands for the closure (in the operator norm). That algebra may be very interesting to study, and be related to a certain symmetry group of transformations and so on. However, there are other algebras that could be even more interesting, for example the resolvent algebra $\{(A-i\lambda)^{-1}, \lambda\in\mathbb{R}\}\overline{\phantom{ii}}$.
In the case of CCR, the resolvent algebra has a "richer" structure of affiliated self-adjoint operators, and more importantly of automorphisms. That means that more types of quantum dynamics can be defined on the resolvent algebra, preserving it, than for the Weyl (unitary exponential) algebra. In the viewpoint of observables being only the operators affiliated to a given algebra, this means that the resolvent algebra contains more observables, and a less trivial structure of possible evolutions than the Weyl algebra.
Best Answer
Not every Hermitian operator is an observable. The identity on a space is certainly Hermitian, but it is no observable in any physical sense - its eigenvalue is $1$, and that's it. The eigenvalue of scalar operators gives you precisely zero information about the state considered, and usually, we would like an observable to at least give us some information about the state we measure.
This is a subtlety that is usually glossed over in most courses, but if you think about canonical quantization, it is quite clear that not every Hermitian operator on the space of states will be induced by an observable on the classical phase space.
The density matrix is similar - yuggib's interpretation is correct, but this is quite unlike our usual observables, where the eigenvalues of the observable would correspond to some (classical) property of the state being measured. I'm reluctant to unequivocally pronounce it an "observable" or not on these grounds.