Concerning point c), on how complex numbers come into quantum theory:
This has a beautiful conceptual explanation, I think, by applying Lie theory to classical mechanics. The following is taken from what I have written on the nLab at quantization -- Motivation from classical mechanics and Lie theory. See there for more pointers and details:
Quantization of course was and is motivated by experiment, hence by observation of the observable universe: it just so happens that quantum mechanics and quantum field theory correctly account for experimental observations where classical mechanics and classical field theory gives no answer or incorrect answers. A historically important example is the phenomenon called the “ultraviolet catastrophe”, a paradox predicted by classical statistical mechanics which is not observed in nature, and which is corrected by quantum mechanics.
But one may also ask, independently of experimental input, if there are good formal mathematical reasons and motivations to pass from classical mechanics to quantum mechanics. Could one have been led to quantum mechanics by just pondering the mathematical formalism of classical mechanics? (Hence more precisely: is there a natural Synthetic Quantum Field Theory?)
The following spells out an argument to this effect. It will work for readers with a background in modern mathematics, notably in Lie theory, and with an understanding of the formalization of classical/prequantum mechanics in terms of symplectic geometry.
So to briefly recall, a system of classical mechanics/prequantum mechanics is a phase space, formalized as a symplectic manifold (X,ω). A symplectic manifold is in particular a Poisson manifold, which means that the algebra of functions on phase space X, hence the algebra of classical observables, is canonically equipped with a compatible Lie bracket: the Poisson bracket. This Lie bracket is what controls dynamics in classical mechanics. For instance if H∈C ∞(X) is the function on phase space which is interpreted as assigning to each configuration of the system its energy – the Hamiltonian function – then the Poisson bracket with H yields the infinitesimal time evolution of the system: the differential equation famous as Hamilton's equations.
Something to take notice of here is the infinitesimal nature of the Poisson bracket. Generally, whenever one has a Lie algebra 𝔤, then it is to be regarded as the infinitesimal approximation to a globally defined object, the corresponding Lie group (or generally smooth group) G. One also says that G is a Lie integration of 𝔤 and that 𝔤 is the Lie differentiation of G.
Therefore a natural question to ask is: Since the observables in classical mechanics form a Lie algebra under Poisson bracket, what then is the corresponding Lie group?
The answer to this is of course “well known” in the literature, in the sense that there are relevant monographs which state the answer. But, maybe surprisingly, the answer to this question is not (at time of this writing) a widely advertized fact that has found its way into the basic educational textbooks. The answer is that this Lie group which integrates the Poisson bracket is the “quantomorphism group”, an object that seamlessly leads to the quantum mechanics of the system.
Before we spell this out in more detail, we need a brief technical aside: of course Lie integration is not quite unique. There may be different global Lie group objects with the same Lie algebra.
The simplest example of this is already one of central importance for the issue of quantization, namely, the Lie integration of the abelian line Lie algebra ℝ. This has essentially two different Lie groups associated with it: the simply connected translation group, which is just ℝ itself again, equipped with its canonical additive abelian group structure, and the discrete quotient of this by the group of integers, which is the circle group
U(1)=ℝ/ℤ.
Notice that it is the discrete and hence “quantized” nature of the integers that makes the real line become a circle here. This is not entirely a coincidence of terminology, but can be traced back to the heart of what is “quantized” about quantum mechanics.
Namely, one finds that the Poisson bracket Lie algebra 𝔭𝔬𝔦𝔰𝔰(X,ω) of the classical observables on phase space is (for X a connected manifold) a Lie algebra extension of the Lie algebra 𝔥𝔞𝔪(X) of Hamiltonian vector fields on X by the line Lie algebra:
ℝ⟶𝔭𝔬𝔦𝔰𝔰(X,ω)⟶𝔥𝔞𝔪(X).
This means that under Lie integration the Poisson bracket turns into an central extension of the group of Hamiltonian symplectomorphisms of (X,ω). And either it is the fairly trivial non-compact extension by ℝ, or it is the interesting central extension by the circle group U(1). For this non-trivial Lie integration to exist, (X,ω) needs to satisfy a quantization condition which says that it admits a prequantum line bundle. If so, then this U(1)-central extension of the group Ham(X,ω) of Hamiltonian symplectomorphisms exists and is called… the quantomorphism group QuantMorph(X,ω):
U(1)⟶QuantMorph(X,ω)⟶Ham(X,ω).
While important, for some reason this group is not very well known, which is striking because it contains a small subgroup which is famous in quantum mechanics: the Heisenberg group.
More precisely, whenever (X,ω) itself has a compatible group structure, notably if (X,ω) is just a symplectic vector space (regarded as a group under addition of vectors), then we may ask for the subgroup of the quantomorphism group which covers the (left) action of phase space (X,ω) on itself. This is the corresponding Heisenberg group Heis(X,ω), which in turn is a U(1)-central extension of the group X itself:
U(1)⟶Heis(X,ω)⟶X.
At this point it is worth pausing for a second to note how the hallmark of quantum mechanics has appeared as if out of nowhere simply by applying Lie integration to the Lie algebraic structures in classical mechanics:
if we think of Lie integrating ℝ to the interesting circle group U(1) instead of to the uninteresting translation group ℝ, then the name of its canonical basis element 1∈ℝ is canonically ”i”, the imaginary unit. Therefore one often writes the above central extension instead as follows:
iℝ⟶𝔭𝔬𝔦𝔰𝔰(X,ω)⟶𝔥𝔞𝔪(X,ω)
in order to amplify this. But now consider the simple special case where (X,ω)=(ℝ 2,dp∧dq) is the 2-dimensional symplectic vector space which is for instance the phase space of the particle propagating on the line. Then a canonical set of generators for the corresponding Poisson bracket Lie algebra consists of the linear functions p and q of classical mechanics textbook fame, together with the constant function. Under the above Lie theoretic identification, this constant function is the canonical basis element of iℝ, hence purely Lie theoretically it is to be called ”i”.
With this notation then the Poisson bracket, written in the form that makes its Lie integration manifest, indeed reads
[q,p]=i.
Since the choice of basis element of iℝ is arbitrary, we may rescale here the i by any non-vanishing real number without changing this statement. If we write ”ℏ” for this element, then the Poisson bracket instead reads
[q,p]=iℏ.
This is of course the hallmark equation for quantum physics, if we interpret ℏ here indeed as Planck's constant. We see it arises here merely by considering the non-trivial (the interesting, the non-simply connected) Lie integration of the Poisson bracket.
This is only the beginning of the story of quantization, naturally understood and indeed “derived” from applying Lie theory to classical mechanics. From here the story continues. It is called the story of geometric quantization. We close this motivation section here by some brief outlook.
The quantomorphism group which is the non-trivial Lie integration of the Poisson bracket is naturally constructed as follows: given the symplectic form ω, it is natural to ask if it is the curvature 2-form of a U(1)-principal connection ∇ on complex line bundle L over X (this is directly analogous to Dirac charge quantization when instead of a symplectic form on phase space we consider the the field strength 2-form of electromagnetism on spacetime). If so, such a connection (L,∇) is called a prequantum line bundle of the phase space (X,ω). The quantomorphism group is simply the automorphism group of the prequantum line bundle, covering diffeomorphisms of the phase space (the Hamiltonian symplectomorphisms mentioned above).
As such, the quantomorphism group naturally acts on the space of sections of L. Such a section is like a wavefunction, except that it depends on all of phase space, instead of just on the “canonical coordinates”. For purely abstract mathematical reasons (which we won’t discuss here, but see at motivic quantization for more) it is indeed natural to choose a “polarization” of phase space into canonical coordinates and canonical momenta and consider only those sections of the prequantum line bundle which depend only on the former. These are the actual wavefunctions of quantum mechanics, hence the quantum states. And the subgroup of the quantomorphism group which preserves these polarized sections is the group of exponentiated quantum observables. For instance in the simple case mentioned before where (X,ω) is the 2-dimensional symplectic vector space, this is the Heisenberg group with its famous action by multiplication and differentiation operators on the space of complex-valued functions on the real line.
Best Answer
Indeed,
canonical quantization works just when it works.
It is in my view wrong and dangerous to think that this is the way to construct quantum theories even if it sometimes works: it produced astonishing results as the theoretical explanation of the hydrogen spectrum.
However, after all the world is quantum and classical physics is an approximation: the quantization procedures go along the wrong direction! There are in fact several no-go results against a naive validity of such procedures cumulatively known as Groenewold -Van Hove's theorem.
However, the question remains: why does that weird relation between Poisson brackets and commutators exist?
In fact, this relation motivates the naive quantization procedures.
In my view, the deepest answer relies upon the existence of some symmetry groups in common with classical and quantum theory.
These groups $G$ of transformations are Lie groups and they are therefore characterized by their so called Lie algebras $\mathfrak{g}$, which are vectors spaces equipped with a commutator structure $[a,b] \in \mathfrak{g}$ if $a,b\in \mathfrak{g}$. We can think of $a\in \mathfrak{g}$ as the generator of a one-parameter subgroup of $G$ usally denoted by $\mathbb{R} \ni t \mapsto \exp(ta) \in G$. If $a_1, \ldots, a_n \in \mathfrak{g}$ form a vector basis, it must hold $$[a_i,a_j] = \sum_k C^k_{ij}a_k\tag{1}\:,$$ for some real constants $C_k^{ij}$. These constants (almost) completely determine $G$. For instance, if $G=SO(3)$ the group of 3D rotations, the one-parameters subgroups are rotations around fixed axes and it is always possible to choose $C_k^{ij}= \epsilon_{ijk}$ (the so-called Ricci symbol).
In classical physics, one represents the theory in the Hamiltonian formulation. States are points of a $2n$ smooth dimensional manifold $F$ called the space of phases, with prefereed classes of coordinates, said canonical, denoted by $q^1,\ldots, q^n, p_1,\ldots, p_n$.
If $G$ is a symmetry group of the system, then there is a faithful representation $G \ni g \mapsto \tau_g$ of it in terms of (canonical) transformations $\tau_g : F \to F$ which move the classical states according to the transformation $g$. The representation $G \ni g \mapsto \tau_g$ admits an infinitesimal description in terms of infinitesimal canonical transformations strictly analogous to the infinitesimal description of $G$ in terms of its Lie algebra $\mathfrak{g}$. In this case the corresponding of the Lie algebra is a linear space of smooth functions, $A \in C^\infty(F, \mathbb{R})$ representing classical observables, and the Poission bracket $\{A,B\} \in C^\infty(F, \mathbb{R})$.
An (actually central) isomorphism takes place between the Lie algebra $(\mathfrak{g}, [\:,\:])$ and the similar Lie algebra $(C^\infty(F, \mathbb{R}), \{\:\:\})$ made of physical quantities where the commutator $\{\:\:\})$ is just the famous Poisson bracket.
If $a_k\in \mathfrak{g}$ corresponds to $A_k\in C^\infty(F, \mathbb{R})$ and (1) is valid for $G$, then $$\{A_i,A_j\} = \sum_k C^k_{ij}A_k + c_{ij}1 \tag{2}$$ where the further constants $c_{ij}$, called central charges, depend on the representation. $$a \mapsto A\tag{2'}$$ defines a (projective or central) isomorphism of Lie algebras.
When passing to the quantum description, if $G$ is still a symmetry group a similar mathematical structure exists. Here, the space of (pure) states is a complex Hilbert space $H$ and the (pure) states are normalized vectors $\psi\in H$ up to phases.
If $G$ is a symmetry group there is a (projective/central) unitary representation $G \ni g \mapsto U_g$ in terms of unitary operators $U_g : H\to H$. The one-parameter subgroups of $G$ are now represented by unitary groups of exponental form (I will systematically ignore a factor $1/\hbar$ in front of the exponent) $$\mathbb{R} \ni t \mapsto e^{-it \hat{A}}\:,$$ where $\hat{A}$ is a (uniquely determined) selfadjoint operator.
Again, if (1) is valid and $\hat{A}_k$ corresponds to $a_k\in \mathfrak{g}$, we have that $$[-i\hat{A}_i,-i\hat{A}_j]= -i\sum_k C^k_{ij}\hat{A}_k -i c'_{ij}I \tag{3}$$ where $[\:,\:]$ is the commutator of operators. In other words $$a \mapsto -i\hat{A} \tag{3'}$$ defines a (projective) isomorphism of Lie algebras.
I stress that the isomorphisms (2') and (3') independently exist and they are just due to the assumption that $G$ is a symmetry group of the system and the nature of the representation theory machinery.
Using these two isomprphisms, we can construct a third isomorphism (assuming $c_{ij}=c'_{ij}$) that interpolates between the classical and the quantum realm.
In this way, if $A \in C^\infty(F, \mathbb{R})$ corresponds to $\hat{A} : H \to H$ (actually one should restrict to a suitable dense domain), then $$\{A,B\} \quad \mbox{corresponds to} \quad i[\hat{A},\hat{B}]\tag{4}$$ when comparing (2) and (3). (I again ignored a factor $\hbar$ since I have assumed $\hbar=1$ in the exponential expression of the one-parameter unitary groups.)
It is now clear that (4) is the reason of the correspondence principle of canonical quantization when the same symmetry group exists both in classical and in quantum physics.
In non relativistic physics, the relevant symmetry group is the Galileo group. This plays a crucial role both in classical and in non-relativistic quantum physics.
So we must have a (central) representation of its Lie algebra both in classical Hamiltonian and in Quantum physics.
Relying upon the above discussion, we conclude that the isomorphism relating the isomorphic classical and quantum representations of the Galileo group -- the map associating classical quantities to corresponding operators preserving the commutation relations -- includes the so called canonical quantisation procedure
Let us illustrate this fact in details. The Lie algebra $\mathfrak{g}$ includes a generator $p$ which, in classical Hamiltonian theory, describes the momentum (generator of the subgroups of translations) and another generator $k$ (generator of the subgroup of classical boost) corresponding to the position up to a constant corresponding to the mass of the system $m$.
Let us focus on the three levels.
Geometrically $$[k,p]=0\:.$$ In the Hamiltonian formulation, a central charge show up $$\{k,p\}= m 1$$ so that, defining $x:= k/m$, we have $$\{x,p\}= 1\:.$$ In Quantum physics, in view of the discussion above, we should find for the corresponding generators/observables $$[-i\hat{K},-i\hat{P}]= -im \hat{I}$$ hence, defining $\hat{X}:= \frac{1}{m}\hat{K}$, $$[\hat{X},\hat{P}]= i \hat{I}$$
This correspondence, which preserves the commutation relation, can be next extended from the initial few observables describing the Lie algebra to a larger algebra of observables said the universal enveloping algebra. It is constructed out of the Lie algebra of the Galileo group. It includes for instance polynomials of observables.
Summing up: there are some fundamental symmetry groups in common with classical and quantum physics. These groups are the building blocks used to construct the theory, since they are deeply connected to basic notions as the concept of reference frame and basic physical principles as the relativity principle. The existence of these groups creates a link between classical and quantum physics. This link passes through the commutator structure of (projective) representations of the said group which is (projective) isomorphic to the Lie algebra of the symmetry group. Quantization procedures just reflect this fundamental relationship. Next the two theories evolve along disjoint directions and, for instance, in quantum theory, further symmetry groups arise with no classical corresponding.