Brief introduction:
Thermodynamic integration is a neat computational method used mainly for computing free energy differences between target and reference states of classical many-body systems, such as gases and liquids.
The key idea is the following: free energy is a thermal quantity (i.e. not expressible as averages of phase space coordinates) and thus not measurable as such in any experimental or numerical way. But free energy derivatives can of course be measured, e.g., in the canonical ensemble the derivative of the Helmholtz free energy with respect to the volume gives us the pressure, which is measurable both experimentally and numerically. Being able to compute such derivatives, one then uses thermodynamic integration methods to compute free energy differences along reversible paths (in the plane of any tuple of natural variables) that connect a reference state of the system (i.e., one for which the free energy is actually known) to a target desired state (whose free energy we want to compare with the reference state).
All this sounds so far rather natural, but the trick in thermodynamic integration and maybe its strength lies in the fact that as long as we want to compute things numerically and thus not bound by experimental limitations, one is not limited to physical paths, instead any parameter $\lambda$ in the free energy can be used (as a dynamic variable) to perform the thermodynamic integration, as long as the function (potential energy or free energy) admits a derivative with respect to the chosen variable. One generally expresses this method as follows:
We parametrize the potential energy of the system w.r.t any parameter $\lambda$ be it a physical one or not, then havign two states in mind, state (1) being the reference (obtained when $\lambda = 0$) and state (2) the target state (obtaied when $\lambda = 1$) whose free energy we are interested in, we write:
$$
U(\lambda) = (1-\lambda)U_1 + \lambda U_2 \tag{(1)}
$$
then for example if we take the parametrized Helmholtz free energy $F(\lambda),$ the free energy difference can be shown to be:
$$
F_{\lambda=1} – F_{\lambda=0} = \int_{\lambda=0}^{\lambda=1} d\lambda \left\langle \frac{\partial U(\lambda)}{\partial \lambda} \right\rangle_{\lambda} \tag{(2)}
$$
where $\langle \rangle_{\lambda}$ is an ensemble average over the system with the potential energy function $U(\lambda).$ The claim often made in literature is that, such thermodynamic integrations are valid using any function $U(\lambda)$ as long as it is differentiable and satisfies the boundary conditions (for the reference and target states).
Question:
Purely from a conceptual point of view, I have no idea what is going on here. How is it possible that we can just parametrize the free energy/potential energy by non-physical parameters and still manage to correctly gauge the free energy difference correctly between two physical states of a system?
Intuitively, I would have expected that if the thermodynamic integration method is performed using non-physical parameters, then one would get nonsense, meaning predicting the wrong equilibrium configurations as an example. But somehow all this is possible and commonly used in computational physics. I'm just trying to understand why this method can work so flexibly.
Best Answer
Some mathematical statements then an intuition statement:
Mathematical
We often use parameterizations like this to mathematically represent a continuous motion from one point to the next, for instance when discussing convex spaces, in which we say a space is convex if all points between any two given points are also in the set (in the language of the wiki page, convex if the vector $\lambda u_i+(1-\lambda) u_j$ is also in the space, for any vectors $u_i$ and $u_j$ in the space; satisfied by a sphere, but not by a donut).
Similarly, the parameterization here is simply a way to represent traveling through a space, in this case a space of possible states. The "nonphysical" parameter $\lambda$ isn't introducing new physics any more than my above analogy means the space of a sphere is in any way physically changed by our mathematical wandering through the space of a sphere. The reason this works is that you're wandering through a continuously changing energy landscape, which brings me to...
Intuition
For state variables like energy, as long as you wander from one state to another, keeping track of your progress as you go (more on that below), you can find your new energy from your old, just like with the traveling analogy already discussed.
The nuance is that if you, say, travel from one city to another city directly North, you might take a path that veers a bit east before coming back west to arrive at its destination. You might say, doesn't that increase my total distance traveled? Yes, but any travel east cancels any travel west. Similarly, going through this space of states with varying energies, any increases along the path cancel any decreases along the path, and you can arrive at the total energy difference between your initial and final states. Now you can see why we need this path to be differentiable with respect to $\lambda$: we can't allow discontinuous jumps.
So in conclusion, we don't really care about the nature of the path, since any funny deviations from point A to point B will cancel. $\lambda$ is just the mathematical parameter that allows us to continuously follow our path and ensure we end where we want to go.