I've been trying to get a better grasp of the vacuum catastrophe problem, that is vacuum energy as calculated in QFT vs the observed value in cosmology differs by 120 orders of magnitude. The issue fundamentally arises from the fact that if we try to calculate the absolute value of the zero point energy of any quantum field the result is necessarily divergent (infinity) as we add all the possible modes of vibrations (frequencies) of the field, which are infinite. It is then speculated that there exist a cutoff frequency which would prevent adding infinite quantities, but to me it seems rather arbitrary this cutoff corresponds to the Planck lenght, which is simply a number where seemingly unphysical stuff happens when we cram two electrons too close together. Using this reasoning we get the wildly wrong (according to measurements) 10^110 value.
If we instead try to fit the measured value in the equations (from astronomical observations of the Hubble constant), we get a corresponding cut off lenght of around 1 mm.
Now this result has mostly been deemed unphysical as well, but for reasons that are not really convincing to me, like the cutoff corresponding to the natural quantization of spacetime can't be 1 mm, but it doesn't seem to be necessarily related to that. It could stem from a totally different property of fields. To my understanding this cutoff doesn't prevent lower wavelength vibrations, it just prevents them at the fundamental level of the field (unexcited fields).
I'd be glad if anyone could link or just explain this issue better, and why they think 1 mm cutoff is/is not a valid solution.
I tried looking this stuff up online, but i couldn't really find many papers that tackled this issue (explaining a possible 1 mm cutoff), and many of those i found were rather hard to digest.
Something I will link though is these two studies, that hopefully will make it more clear where I'm trying to point the discussion: one that refutes first low energies cutoffs, then tries to generalize to any possible cutoff, using the common explanation of the Casimir effect as stemming from a quantum vacuum effect.
The second study elegantly explains how a quantum vacuum interpretation of this effect is not legitimate mathematically, hence invalidating many assumptions of the first paper.
This paper discusses why a cut off of any kind in vacuum energy would produce different effects from those observed in the Casimir experiment (IF vacuum energy is responsible for the effect)
https://doi.org/10.1016/j.physletb.2006.08.026
This paper explains why the Casimir effect is better understood with Van Der Waals forces rather than vacuum fluctuations:
Best Answer
There is a strong argument that zero point energy in field (at least in free EM field) is really just an artifact of dubious glorification of a particular Hamiltonian (the usual one carrying zero point energy terms $\hbar \omega_m/2$) into EM field energy.
Most(all?) measurable things (frequencies of transitions...) are predicted by the theory the same if instead of the problematic Hamiltonian
$$ \sum_{m} \hbar\omega_m(a_m^+a_m + \frac{1}{2}) \tag{*} $$ with the zero point terms, the normally ordered Hamiltonian $$ \sum_{m} \hbar \omega_m a_m^+a_m\tag{**} $$ is used. In fact if we want the quantum energy to correspond to classical Poynting energy in some limit of strong coherent states (a quantum EM field described well by classical light theory) , we can't define quantum energy by $(*)$, because there is no zero-point energy in e.g. classical electrostatic field or plane wave.
Already in classical mechanics we encounter many Hamiltonians that differ in form and value, but are equivalent in their equations of motion. But despite that infinite plethora of Hamiltonians for any given system, we use only one expression to define standard energy, often sum of kinetic and some standard form of potential energy. It seems natural to do similar in QT of field, i.e. we may work with many Hamiltonian operators with different constant shifts, but we should settle on some unique definition of energy in relation to rest of physics. Using (**) seems much better than (*) for this.
Trying to explain the value of the cosmological term via zero point energy of field is like the old idea of explaining mass of electron in terms of electromagnetic mass of some interacting system of charged elements - originally, a cool new idea, but after many trials it didn't really work out, the calculated value was either way too big (charge concentrated to a ball of size limited by experiments to be less than 1e-18 m) or zero (if electron has no parts), and we got back to regarding the electron mass as independent constant of nature, having nothing to do with EM mass.