In reading Weinstock's Calculus of Variations, on pages 261 – 262 he explains how Schrodinger apparently first derived the Schrodinger equation from variational principles.
Unfortunately I don't think page 262 is showing so I'll explain the gist of it:
"In his initial paper" he considers the reduced Hamilton-Jacobi equation
$$\frac{1}{2m}\left[\left(\frac{\partial S}{\partial x}\right)^2 \ + \ \left(\frac{\partial S}{\partial y}\right)^2 \ + \ \left(\frac{\partial S}{\partial z}\right)^2\right] \ + \ V(x,y,z) \ – \ E \ = \ 0$$
for a single particle of mass $m$ in an arbitrary force field described by a potential $V = V(x,y,z)$.
With a change of variables $S \ = \ K\log(\Psi)$, (where $K$ will turn out to be $\hbar=h/2\pi$) it reduces to
$$\frac{K^2}{2m}\left[\left(\frac{\partial \Psi}{\partial x}\right)^2 \ + \ \left(\frac{\partial \Psi}{\partial y}\right)^2 \ + \ \left(\frac{\partial \Psi}{\partial z}\right)^2\right] \ + \ (V \ – \ E)\Psi^2 \ = 0.$$
Now instead of solving this he, randomly from my point of view, chose to integrate over space
$$I = \iiint_\mathcal{V}\left(\frac{K^2}{2m}\left[\left(\frac{\partial \Psi}{\partial x}\right)^2 \ + \ \left(\frac{\partial \Psi}{\partial y}\right)^2 \ + \ \left(\frac{\partial \Psi}{\partial z}\right)^2\right] + (V – E)\Psi^2\right)\,\mathrm{d}x\,\mathrm{d}y\,\mathrm{d}z$$
then Weinstock extremizes this integral which gives us the Schrodinger equation.
Apparently as the book then claims on page 264 it is only after this derivation that he sought to connect his idea's to deBroglie's wave-particle duality.
Thus I have three questions,
1) What is the justification for Feynman's famous quote:
Where did we get that [Schrödinger's equation] from? It's not possible to derive it from anything you know. It came out of the mind of Schrödinger.
The Feynman Lectures on Physics
in light of the above derivation. I note that all the derivations I've seen of the Schrodinger equation doing something like using operators such as $i\hbar\partial/\partial t = E$ to derive it always mention it's merely heuristic, yet what Schrodinger apparently originally did seems like a roundabout way of solving the Hamilton-Jacobi equation with no heuristic-ness in sight. What subtleties am I missing here? Why would I be a fool to arrogantly 'correct' someone who says Schrodinger is not derivable from anything you know?
Note I've read a ton of the threads on this topic on this forum & none even go near the calculus of variations, in fact the above contradicts this explanation here that even refers to Schrodinger's Nobel prize lecture, so hopefully it's not a duplicate.
2) Is the mathematical trick Schrodinger has used something you can use to solve problems?
3) Why can't you use this exact derivation in the relativistic case?
Edit: On the topic of complex numbers:
On page 276, 14 pages after he explains what I posted, & after he goes through how Schrodinger linked his work with DeBroglie's work, it is only then that Weinstock says:
In a more complete study of quantum mechanics than the present one the
admissibility of complex eigenfunctions $\Psi$ is generally shown to be
necessary. If $\Psi$ is complex, the quantity $|\Psi|^2$ is employed as the
position probability-density function inasmuch as $\Psi^2$ is not restricted
to real nonnegative values.
Apparently Schrodinger was able to do what I have posted using real-valued functions & have K as I have defined it, without i. If you're following what Weinstock is saying he shows how the hydrogen atom energy levels are explainable without complex numbers, i.e. he is able to derive a physical interpretation of the eigenvalues (discrete energy levels) of the Schrodinger equation that were in accord with experiment (see Section 11.3 Page 279 on). As far as I understand it it is in trying to find a physical interpretation of the eigenfunctions that one is forced into complex numbers, though apparently, according to the book, it can be shown to be necessary. That is something to take into account!
Best Answer
I'm answering only part of your question, please bear in mind that some of the topics that you pointed out are doubts I have myself.
Concerning the comment from Feynman, I usually don't like the authority arguments. If your only reason to believe something is that someone said it, it's not a good reason. Just as some examples, Newton believed that light had no wave-like behaviour, which is simply wrong.
About the variational formulation of Schrödinger Equation. If by variational you mean, having a Lagrangian, the following Lagrangian does the job:
$ \mathcal L = \frac{i\hbar}{2}\left(\psi^*\partial_t \psi - \psi \partial_t \psi^*\right) - \frac{\hbar^2}{2m}\left(\nabla \psi\right)\cdot \left(\nabla \psi^*\right) - V\psi^*\psi $
With the assumption that you have a complex classical scalar field, $\psi$, and that you can calculate the Euler-Lagrange equation separately for $\psi$ and $\psi^*$. As with anything using the extremum action principle, you really have to guess which your Lagrangian is based in Symmetry principles + some guideline for your problem, here is no different.
I don' t know how did Schrödinger made the original derivation, but if you make a slight change of variables in the above lagrangian, you get very close to what you have written above.
The trick is writing $\psi = \sqrt n e^{iS/\hbar}$, where $n$ is the probability density and $S$ is,
essentiallythe phase of the wave-function. If you have trouble with the derivation, I can help you latter.Again about Feynman's quote. It's possible to arrive at Schrödinger equation without passing through an arbitrary heuristic procedure, that way is developed by Ballentine's book. You still have to postulate where you live, if you consider that arbitrary or not, it is completely up to you.
The other point that it is possible to consistently and rigorously construct a quantum theory from any classical mechanical theory, using quantization by deformation. That's why I think it's a bit false that "It's not possible to derive it from anything you know. It came out of the mind of Schrödinger", as Feynman, and many other great names, said.