As a preliminary remark, note that the Tate-Shafarevich group also measures a certain defect, just like the class group. Its elements correspond to homogeneous spaces that have points everywhere locally but no global points. This is explained e.g. in Silverman.
First, let us agree that the Birch and Swinnerton-Dyer conjecture is the elliptic curves analogue of (the square of!) the analytic class number formula. Just like the latter, the former gives an algebraic interpretation of the central value of the $L$-function associated with a Galois representation (in fact a compatible family of $l$-adic Galois representations). See this MO post for more on the comparison of these two formulae.
In both formulae, a central role is played by a certain finitely generated abelian group. In the class number formula, it's the unit group of the ring of integers, in the BSD formula it's the Mordell-Weil group. Both formulae contain the size of the torsion subgroup in the denominator, and the volume (to be precise the covolume under a suitable map) in the numerator. Also, both contain some Tamagawa numbers. In the class number formula, we can only see the Tamagawa numbers at the infinite places, in the guise of some powers of 2 and $\pi$. Finally, both contain the discriminants of the number fields involved. The only other ingredient is the class number in the one case and the size of sha in the other. It is therefore natural to conclude that these "correspond to each other" in these two situations.
There is in fact a more precise correspondence, but one that is much harder to explain. Both formulae have a common generalisation, the Bloch-Kato conjecture. Under this generalisation, sha and the class number literally become the same object attached to a motive. In particular, the class number can, just like sha, be expressed in terms of Galois cohomology. This is explained in some surveys on the Bloch-Kato conjecture and on its equivariant refinement, but even the surveys have quite a lot of prerequisites in order to be able to read and to understand them.
There are various ways to interpret how class groups measure (non)unique factorization. For example, Carlitz (1960) showed that the class group has order at most $2$ iff all factorizations of a nonzero nonunit into irreducibles have the same number of factors. Narkiewicz posed the problem of generalizing this, i.e. devising arithmetical characterizations of class groups. Following is one such characterization, due to J. Kaczorowski, Colloq. Math. 48 (1984), no. 2, 265-267.
Let $\,\cal O\,$ denotes the ring of integers of an algebraic number field. An algebraic integer $\rm\,a\in \cal O\,$ is said to be completely irreducible if it is irreducible and $\rm\,a^n\,$ has a unique factorization for all $\rm\,n\in \Bbb N.\,$ Let $\rm\ {\rm ord}\, a\ $ be the least $\rm\,n\in \Bbb N\,$ such that the length of any factorization of $\rm\,ab\,$ is $\rm\,\le n\,$ for any completely irreducible $\rm\,b\in \cal O.\:$ A sequence of nonassociate algebraic integers $\rm\,a_1,\ldots, a_k\,$ is said to be good if each $\rm\,a_i\,$ is completely irreducible but not prime, and their product $\rm\, a_1\cdots a_k\,$ factors uniquely. Suppose that $\rm\,a_1,\ldots,a_k\,$ is a good sequence having maximal $\rm\,\prod {\rm ord}\,a_i.\,$ Then $\cal O$ has class group $\,\rm\cong C({\rm ord}\, a_1\!) \oplus \cdots \oplus C({\rm ord}\,a_k\!),\:$ where $\rm \,C(n) \cong $ cyclic group of order $\rm\,n.\,$ A proof can be found in Chapter $9$ of Narkiewicz's book Elementary and Analytic Theory of Algebraic Numbers.
Similar results were also published by F. Halter-Koch, and D.E. Rush around the same time. Since then these results have been generalized and abstracted into a powerful theory of nonunique factorization in Krull monoids. Search on said authors and Geroldinger to learn more.
Below is Geroldinger's summary of this line of research, from a paper in Jnl. Algebra 1990
Almost $20$ years ago, W. Narkiewicz posed the problem to give an arithmetical characterization of the ideal class group of an algebraic number field ([13, problem 32]). In the meantime there are various answers to this question if the ideal class group has a special form. (cf. [4], [5], [12] and the literature cited there).
The general case was treated by J. Koczorowski [11], F. Halter-Koch [8], [9, §5] and D. E. Rush [16]. In principle they proceed in the following way: they considera finite sequence $(a_i)_{i=1\ldots r}$ of algebraic integers, requiring a condition of independence and a condition of maximality. Thereby the condition of independence guarantees that the ideal classes $g_i$ of one respectively all prime ideals $g_i$ appearing in the prime ideal decomposition of $a_i$ are independent in a group theoretical sense. The invariants of the class group are extracted from arithmetical properties of the $a_i$’s, and the condition of maximality ensures that one arrives at the full class group but not at a subgroup.
Best Answer
Everyone says the class group "measures" the failure of unique factorization, but the only sense of measuring that failure is exactly what you noticed: class number 1 vs. class number greater than 1. That is the only justification for such terms as "measuring the failure". Edit: By "only sense" I mean that the people who teach about ideal class groups in algebraic number theory classes have no grander meaning in mind than the distinction $h=1$ and $h > 1$ when they speak about class groups measuring the failure of unique factorization. There are descriptions of the structure of the ideal class group in terms of how elements factors into other elements (see the link in Bill Dubuque's comment above), but that is not what people have in mind when they speak about ideal class groups "measuring the failure" of unique factorization.
This is typical in math: you construct a group (or vector space, etc.) for each object in some family and the group is nontrivial iff some nice property doesn't hold. Let's call the nice property "wakalixes". Then you say to the world "this group measures the failure of wakalixes" and that leads generations of students to ask "What do you mean it measures the failure? What does having one nontrivial wakalixes group or some other nonisomorphic nontrivial wakalixes group actually mean?" And the answer is "All that means is that wakalixes fails in both cases." There is no other meaning intended in geneal. In number theory, topology, etc., when you try to do something and run into a gadget that is trivial when you can do what you want and nontrivial when you can't do what you want, you call the gadget "a measure of the failure" to do what you want or "an obstruction" to do what you want.
Ideal class groups have interpretations and applications that are not directly about the failure or not of unique factorization, and such applications are much more important for the role of ideal class groups in number theory than trying to intuit some down to earth meaning about a class group being cyclic of order $4$.
Maybe the following point will interest you. If you replace $\mathcal O_K$ with $\mathcal O_K[1/\alpha]$ where $\alpha$ is a nonzero element of $\mathcal O_K$ such that the ideal class group of $K$ is generated by ideal classes of the prime ideals dividing $(\alpha)$, then $\mathcal O_K[1/\alpha]$ is a PID. For instance, $\mathbf Z[\sqrt{-5}]$ has class number 2 generated by the ideal class of the prime ideal $\mathfrak p = (2,1+\sqrt{-5})$. We have $\mathfrak p^2 = (2)$ and $\mathbf Z[\sqrt{-5},1/2]$ is a PID. More generally, for a ring of $S$-integers $\mathcal O_{K,S}$, where $S$ is a finite set of places of $K$ containing all the archimedean places, the ideal class group is a quotient group of the ideal class group of $\mathcal O_K$ (details are in the answer here), so by putting into $S$ a suitable set of primes you can gradually kill off the whole class group and you're left with a PID. In this way, the class group tells you how to enlarge $\mathcal O_K$ in a mild way to recover unique factorization while maintaining other nice properties (like a finitely generated unit group, which would not happen if you did something extreme and just replaced $\mathcal O_K$ by $K$).
When I was a grad student I was really bothered by encountering the same slogan ("it measures the failure...") in algebraic topology with homology groups. I asked a postdoc "If I told you $H_{37}(X)$ has a particular value, does that automatically mean something to you?" And the postdoc said "Nope."