While trying to get some perspective on the extensive literature about highest weight modules for affine Lie algebras relative to "level" (work by Feigin, E. Frenkel, Gaitsgory, Kac, ….), I run into the notion of dual Coxeter number but am uncertain about the extent of its influence in Lie theory. The term was probably introduced by Victor Kac and is often denoted by $h^\vee$ (sometimes by $g$ or another symbol). It occurs for example in the 1990 third edition of his book Infinite Dimensional Lie Algebras in Section 6.1. (The first edition goes back to 1983.) It also occurs a lot in the mathematical physics literature related to representations of affine Lie algebras. And it occurs in a 2009 paper by D. Panyushev in Advances which studies the structure of complex simple Lie algebras.
Where in Lie theory does the dual Coxeter number play a natural role (and why)?
A further question is whether it would be more accurate historically to refer instead to the Kac number of a root system, since the definition of $h^\vee$ is not directly related to the work of Coxeter in group theory.
BACKGROUND: To recall briefly where the Coxeter number $h$ comes from, it was introduced by Coxeter and later given its current name (by Bourbaki?). Coxeter was studying a finite reflection group $W$ acting irreducibly on a real Euclidean space of dimension $n$: Weyl groups of root systems belonging to simple complex Lie algebras (types $A–G$), these being crystallographic, together with the remaining dihedral groups and two others. The product of the $n$ canonical generators of $W$ has order $h$, well-defined because the Coxeter graph is a tree. Its eigenvalues are powers of a primitive $h$th root of 1 (the "exponents"): $1=m_1 \leq \dots \leq m_n = h-1$. Moreover, the $d_i = m_i+1$ are the degrees of fundamental polynomial invariants of $W$ and have product $|W|$.
In the Weyl group case, where there is an irreducible root system (but types $B_n, C_n$ yield the same $W$), work of several people including Kostant led to the fact that $h$ is 1 plus the sum of coefficients of the highest root relative to a basis of simple roots. On the other hand, the dual Coxeter number is 1 plus the sum of coefficients of the highest short root of the dual root system. For respective types $B_n, C_n, F_4, G_2$, the resulting values of $h, h^\vee$ are then $2n, 2n, 12, 6$ and $2n-1, n+1, 9,4$. This gets pretty far from Coxeter's framework.
One place where $h^\vee$ clearly plays an essential role is in the study of a highest weight module for an affine Lie algebra, where the canonical central element $c$ acts by a scalar (the level or central charge). The "critical" level $-h^\vee$ has been especially challenging, since here the theory seems to resemble the characteristic $p$ situation rather than the classical one.
Best Answer
The dual Coxeter number comes up naturally as a normalization factor for invariant bilinear forms on the Lie algebra: according to Kac's book which you quote, $2h^{\vee}$ is the ratio between the Killing form and the "minimal" bilnear form (the trace form for $sl_n$), which has the property that the square of the length of the maximal root is 2.
This minimal form corresponds to the minimal affine Kac-Moody group corresponding to the Lie algebra, or equivalently to the minimal line bundle on the affine Grassmannian or the moduli spaces of G-bundles on curves (the generator of the Picard group). As a result, the $-2h^\vee$-th power of the basic ample line bundle on the Grassmannian or moduli space of bundles (which is associated to the level given by the Killing form) ends up being identified with the canonical line bundle, and in particular the $h^\vee$th power is a square-root of the canonical bundle, or spin structure. (This is analogous to the role of $\rho$ for the finite flag variety.) Thus the critical level arises naturally geometrically -- it corresponds to half-forms on the Grassmannian/moduli spaces. The basic yoga of quantization (or of unitary/normalized induction of representations) tells us that classical symmetries are "shifted" by half-forms - cf $\rho$-shifts in representation theory. Likewise the critical shift for affine algebras.. for example the Feigin-Frenkel theorem is the analogue of the Harish-Chandra isomorphism: the center of the enveloping algebra at critical level (rather than level 0 as one might naively guess, ignoring half-form twists) is isomorphic to the algebra of invariant polynomials on the (dual of the) Lie algebra. (This can be said more canonically keeping track of symmetries of change of variable, magic word being "opers", but let's ignore that).
One can say all this very naturally algebraically (without resorting to geometry) -- $\rho$ can be described as the square root of the modular character of the Borel subalgebra (up to sign or something, not being very careful here). The critical level has a similar description in terms of the positive half (Taylor series part) of the Kac-Moody algebra - if you try to define the modular character of this half you are quickly led to semiinfinite determinants etc, ie to the previous geometric story, and so one can assert that the critical level "is" half the modular character of the positive loop subalgebra.