It appears that they both describe the upper boundary mass that a cloud in space may inhibit before gravitational collapse. From Wikipedia, on Jeans mass/Jeans instability:
"…Jeans instability causes the collapse of interstellar gas clouds and subsequent star formation. It occurs when the internal gas pressure is not strong enough to prevent gravitational collapse of a region filled with matter. For stability, the cloud must be in hydrostatic equilibrium […] The Jeans instability likely determines when star formation occurs in molecular clouds."
Wikipedia on Bonnor-Ebert:
"…the Bonnor–Ebert mass is the largest mass that an isothermal gas sphere embedded in a pressurized medium can have while still remaining in hydrostatic equilibrium. Clouds of gas with masses greater than the Bonnor–Ebert mass must inevitably undergo gravitational collapse to form much smaller and denser objects. As the gravitational collapse of an interstellar gas cloud is the first stage in the formation of a protostar, the Bonnor–Ebert mass is an important quantity in the study of star formation."
The phrasing varies but I feel that, unless I've misunderstood, they're surely referring to the same phenomenon? That after a certain mass is reached, the gas cloud/sphere is not in hydrostatic equilibrium, therefore it collapses due to gravity acting inwards overcomes the gas' pressure acting outwards (in simplified terms), and that this in turn is seen in the study of star formation. What is the difference between the two, and when is one more appropriate to study than the other? Planet formation vs star formation? Is it to do with the medium (BE mass mentions pressurised medium and J mass does not)?
I suspect that the answer is glaringly obvious but Google is only turning out notes or papers on one or the other, but no one comparing the two (as far as I've been able to find).
Best Answer
Your suspicion is correct, the Bonnor-Ebert mass is indeed describing the same instability condition as the Jeans instability. This is the gravitational instability, which is tied to the competition between self-gravity and internal pressure. The only reason they are given different names is that they represent two different ways to arrive at this instability condition, which end up giving compatible results.
To derive the Bonnor-Ebert mass, one considers equilibrium solutions to a spherically symmetric configuration of self-gravitating gas in hydrostatic equilibrium. These are solutions to the Lane-Emden equation. One can then consider the stability of normal modes in these solutions to perturbations. For an isothermal equation of state, the fundamental ("breathing") mode is unstable whenever the mass of the sphere exceeds
$$ M_{BE} = 1.18 \frac {c_s^3}{\rho_0^{1/2} G^{3/2}}$$
where $c_s$ is the isothermal sound speed and $\rho_0$ is the central density of the sphere. In other words, spherical configurations of gas with isothermal equations of state that are initially in equilibrium are unstable to collapse when perturbed if they are more massive than $M_{BE}$. This analysis can be extended to gas with non-isothermal equations of state.
To derive the instability from the Jeans point of view, one can instead consider traveling-wave perturbations (sound waves) traveling through a self-gravitating, homogeneous medium of density $\rho_0$ and isothermal sound speed $c_s$. Through linear analysis of small amplitude waves one can derive the dispersion relation for these waves and conclude that when the wavelength exceeds a critical length,
$$\lambda_J = \sqrt{\frac{\pi c_s^2}{G \rho_0}}\, ,$$
the amplitude grows exponentially in time. So structures that have lengths larger than this are susceptible to collapse when subjected to perturbations. Again, this was all done in the isothermal case for simplicity, but can be generalized to other equations of state.
To see how these two analyses relate to each other, consider the mass enclosed within a sphere of diameter $\lambda_J$ and uniform density $\rho_0$. You'll see that it is the same as $M_{BE}$ to within a factor of 2 or so, which shouldn't be too worrisome given the differences in the initial configurations (an equilibrium sphere for $M_{BE}$, and a uniform medium for the Jeans analysis).
For a detailed discussion, see e.g. chapter 9 of this textbook. The primary application there is star formation, but the physics is general and may be applied to other situations.