The shell theorem applies to spherically-symmetric mass distributions. If you have a compressible fluid so that the density varies with depth, you can still apply the shell theorem (unless there is something else going on, for example net angular momentum that breaks the spherical symmetry). Specifically, the gravitational field is the same as that of a point mass at the center of the distribution.
Here is a demonstration:
Newton's law for the gravitational field of a point mass is
$$F_g/m = GM/r^2$$
with $F_g/m$ the acceleration of a test particle and M the mass of the gravitating body. This is linear in M, meaning that we can take two mass distributions and superimpose them, then find the gravitational field of this new mass distribution by superimposing the two associated individual gravitational fields.
Assume the shell theorem holds for constant density spheres. Take two spheres, one slightly larger than the other, so their radii are $r$ and $r+dr$. Give the larger sphere density $\rho$ and the smaller one density $-\rho$. Superimpose them. You get a thin shell of radius $r$ and thickness $dr$. The shell is empty in the middle because the positive and negative density distributions are canceling there. (It doesn't matter that there is no such thing as negative mass - the shell theorem is purely mathematical and so holds for negative numbers, and the final distribution we're dealing with has no negative mass; it was just a calculation trick.)
By superimposing the two gravitational fields of the positive and negative density distributions, we see that the gravitational field of this shell is the same as that of a point mass at the center of the shell with the mass $4\pi r^2 dr \rho$.
The shell theorem thus holds for these thin shells. A distribution with varying density at different radii is simply a lot of such shells, so the shell theorem holds for arbitrary $\rho(r)$ as long as $\rho$ is a function of $r$ alone.
As for the change in density as a function of radius, we need to balance the hydrostatic pressure with the gravitational attraction.
Take a little piece of the mass distribution with volume $dV$. It is pulled down by a gravitational force
$$F_g = \frac{GM(r) dm}{r^2}$$
$dm$ is $\rho dV$ and $M(r)$ is the mass located closer to the center of the distribution than the point in question, so
$$M(r) = \int_0^r 4\pi x^2 \rho(x) dx$$
This is balanced by the force from pressure
$$F_p = \nabla P dV$$
We assume there is a relationship between pressure and density so
$$\nabla P = \frac{\partial P}{\partial r} = \frac{d P}{d\rho}\frac{d \rho}{dr}$$
Equating the gravitational and pressure forces gives, after some algebra
$$4\pi G r^2\rho = \frac{d}{dr} \left( \frac{r^2}{\rho} \frac{dP}{d\rho} \frac{d\rho}{dr} \right)$$
The solution depends on $\rho(P)$, and is subject to the boundary condition $P(R) = 0$ with $R$ the radius of the distribution, as well as the condition $M(R) = M$, the total mass. We will probably find $R = \infty$.
Thus, once we specify a relationship between $\rho$ and $P$ based on the thermodynamics of the system involved, we can integrate the differential equation to find $\rho(r)$.
Best Answer
If I go to a shop and buy $5$ apples and $10$ bananas then I can usually take the price of one apple $a$ and the price of one banana $b$ and add these together to get a total cost of $5a+10b$. And I pay the same total amount if I buy apples and bananas at the same time or I buy apples, then go back to the shop later and buy bananas - my purchases do not interact with one another. This is a linear system.
But if there is an offer of "$5$ apples for the price of $3$" or "one free banana with every $5$ apples" or "$10\%$ off if you spend more than $\$5$" then the cost of $5$ apples and $10$ bananas will no longer be $5a+10b$. This is a non-linear system, and there is an interaction between my different purchases.