I have this very bad habit of going to the scratch, discarding all the developments of a theory and worldly knowledge, and ask some fundamental (mostly stupid and naive, as some may say) questions as to why why we needed so and so assumption, why we had to consider this way, could we assume $X$ instead of $Y$ and get a different theory and so on and so forth. As a part of this, is the following question :
Way back in the early twentieth century, physicists struggled to explain certain phenomenon like photo electric effect which needed light waves to behave a particles (photons), and the interference effect of electrons (diffraction) which needed them to behave as waves.
So they said, hey, consider a wave (don't ask me know what it is, but just consider it)… ok
$$\Psi(x,t) = e^{i(kx-\omega t)},$$ now without asking what is $\Psi$, we can explain the interference of electrons and also the photo electric effect, basically the wave/particle duality, if we make an analogy between wave and particle nature as $p = \hbar k$, $E = \hbar \omega$.
My seemingly blunt question is, if you want to explain wave nature and interference by considering a wave function $\Psi$, why the heck do we need complex numbers, why not just real function? Cant we consider something like $\Psi(x,t) : \mathbb{R} \times \mathbb{R} \to \mathbb{R}$ or something like that. Hold on, Water waves are successfully explained by real wave function for example $\Psi(x,t) = \cos(kx-\omega t)$, so why the heck we need complex waves for making an anology for wave-particle duality? What if we just unlearn all the QM and start with a real function for waves, what is going to happen?
Sorry for being a bit presumptuous, but some gentlemen will start talking about probability amplitude, uncertainty principle, and so on and so forth, but Gentlemen, wait, I haven't gone that far! I don't have Born interpretation and I don't have uncertainty principle or for that matter the Schrodinger equation yet, So your logic will lead to circular arguments!
After all our goal is to explain physical phenomenon…what if we venture into this jungle of real functions and come up with a totally different theory which explains physical phenomenon.
((If you ask me to do research in theoretical physics, I'll throw all the QM books in garbage (no disregard though) and start thinking from this point of view…Thats my style of working!)
My expected answer is in this spirit, "Hey if you go in that direction, you are bound to end up in a quick sand, for so and so reason"
Best Answer
Good luck to you. First of all you are wrong that classical physics did not use imaginary functions. The solutions of Maxwell's equations expressed as imaginary functions are more general and universal than sines and cosines.
$$\mathbf{E}(\mathbf{r}, t)=\mathrm{Re}\{\mathbf{E}(\mathbf{r} e^{i\omega t}\}$$
Imaginary functions are a useful tool in integrations and descriptions of real data.
With such blind spots I am sure nobody will ask you to do research in theoretical physics.
The difference between the classical use of imaginary functions from the solutions of the wave equations and the quantum mechanical one is the postulate the posits that the square of the wavefunction is real and gives the probability of an elementary particle (or nuclear) interaction to be observed. When in the microcosm quantum mechanics reigns. There one cannot take a ruler, mark it and measure, it was found that the theories and data agreed when the probability postulate was imposed. One has to make many measurements and get the probability distribution for a particularly desired value.
The above link discusses the postulates of quantum mechanics which were not imposed out of a freak imagination, but were necessary to be able to calculate and fit known observations, like the hydrogen atom, and predict the outcome of experiments and observations.
EDIT to address the last part of the question:
That works for art, art is much less dependent on data bases of observations and the tools that can be used.
The fact that for two thousand years people have been creating models of physical observations, and particularly the last 300 a data base of mathematical tools too, constrains creativity in science. The mathematical tools have been used to model all observations up to now. These models are in a way a shorthand description of nature that could be used in many ways instead of going back to the data itself. There exists a frontier of experimental research where the models have not been validated , and that is where new thinking can come in.
If you go into the direction of throwing everything away you will end up with vague models like the Democritus atomic model, or the phlogiston theory, in your own words. The mathematical models used now are validated, some of them to great accuracy. New mathematical tools to model the already modeled data would only be worth the attention if something new and unexpected is predicted and found in the experiments.
There are people working off the beaten track theories, trying to explain quantum mechanics by underlying deterministic theories. These people have a thorough knowledge of existing mathematical tools and the physics models that have been validated. They just want to work at the frontier by ignoring that mainstream physics considers their effort contradictory or impossible/prohibited by the postulates of quantum mechanics and special relativity. An example is the current research interests of G.'t Hooft who has also participated here a while ago .
So if you go in that direction you will end in quick sand surely if you do not have a thorough knowledge of the data and mathematical tools used by physics up to now. If you make the effort to acquire them, then of course you are free to prove mainstream physics "wrong" , as long as your new theory can accommodate the data shorthand of the models up to now . All new theories as they appeared in physics joined smoothly with the old ones, as limiting cases.