Your overall method for determining the equilibrium for a star is correct. We use a combination of an assumed 'polytropic relation':
$$P = K \rho^{1+\frac{1}{n}}$$
and the requirement that this balances the gravitational pressure at a radius r (for a spherical shell of thickness $dr$):
$$4\pi r^2\frac{dP(r)}{dr} = -\frac{GM(r)}{r^2}4\pi r^2\rho(r)$$
where $M(r)$ is the mass enclosed by the shell. Using the polytropic equation together with the fact that $\frac{dM(r)}{dr} = 4\pi r^2\rho(r)$ gives a second order differential equation for $\rho(r)$, called the Lane-Emden equation. You can use the ideal gas law
$$P = \rho\frac{k_BT}{m}$$
where $m$ is the atomic mass of hydrogen, to determine the temperature $T(r)$. Some issues are that firstly, you will need to know $n$, and secondly, solutions are known only for some values of $n$.
For some solutions, there is a finite radius $R$, the radius of the star, for which the density goes to zero. It turns out that, for stars like our sun, $n=3$ is a good model, and there is a finite radius. However, an exact (analytic) solution is unknown.
As a special case, if you take $n\rightarrow\infty$ in the polytropic equation, essentially giving the ideal gas equation itself, with the assumption of constant temperature throughout the star, the solution is called an "isothermal sphere" - but this does not model the sun very well.
For the general case, knowing this radius $R$ (assuming you want the experimental value), you can plug in the integration constants to get $\rho(r)$, and the mass is simply
$$M = \int_{0}^{R}4\pi r^2\rho(r)dr$$
Also note that you will need one of the total mass or the radius of the star to find the other.
The correct formula is actually
$$M = \frac{4\pi^2 a^3}{GP^2}$$
and is a form of Kepler's third law. $M$ in this formula is the central mass which must be much larger than the mass of the orbiting body in order to apply the law.
In reality the formula that should be used is
$$M_1 + M_2 = \frac{4\pi^2 a^3}{GP^2},$$
where $M_1+ M_2$ is the sum of the masses of the two objects and $a$ is the semi-major axis. This obviously reduces to Kepler's third law if $M_1 \gg M_2$.
So with a measurement of $P$ and $a$ alone you can only constrain the sum of the two masses. In order to get the individual masses you need additional information that could be for instance: the relative orbital speeds of the two objects or the relative sizes of the two orbits, both of which would give you the mass ratio $M_1/M_2$.
Best Answer
The basic concept here is that of hydrostatic equilibrium.
If you consider a thin slab of material of density $\rho$ and thickness $\Delta r$ in the star. It has a pressure of $P$ below the slab and a pressure $P + \Delta P$ above the slab. The weight of the slab will be $\rho g A \Delta r $, where $A$ is the area covered by the slab and $g$ is the local value of gravity. To keep the slab in equilibrium you need to balance this weight with the force exerted upwards on the slab due to the pressure difference between the top and bottom. i.e. $$ \rho g\ A\ \Delta r = -\Delta P\ A$$ Thus $\rho g\ \Delta r = -\Delta P$ and as $\Delta r \rightarrow 0$, we can say $$ \frac{dP}{dr} = -\rho(r) g(r) = - \rho \frac{GM(<r)}{r^2},$$ where $\rho(r)$ and $g(r)$ are functions of radius within the star and $M(<r)$ is the mass contained within radius $r$.
To solve this differential equation requires a self-consistent solution of the equations of stellar structure (involving the energy generation and energy transport equations), since the pressure also depends on temperature and composition.
To make progress with this in a back-of-the-envelope kind of way then some vast simplifications are required, namely an assumption about how the density depends on radius. If we assume that the density is constant (awful, but it does give the right proportionalities) then $$\frac{dP}{dr} = - \frac{G \rho}{r^2} \frac{4\pi}{3} \rho r^3$$ $$ \int^{0}_{P(r)} dP = - \frac{4\pi G \rho^2}{3} \int^{R}_{r} r\ dr,$$ where $P=0$ at the surface of the star where $r=R$.
$$ P(r) = \frac{2\pi G \rho^2}{3} (R^2 - r^2)$$
Then we could put this in terms of the mass of the star $M$ by noting that $\rho =3M/4\pi R^3$ $$ P(r) = \frac{2\pi G}{3}\left(\frac{3M}{4\pi R^3}\right)^2 (R^2 - r^2) = \frac{3G}{8\pi}\left(\frac{M^2}{R^4}\right)\left(1 -\frac{r^2}{R^2}\right), $$ and the central pressure (at $r=0$) would be $$P(0) = \frac{3G}{8\pi} \left(\frac{M^2}{R^4}\right)$$
The proportionality is correct here, but comparison with a real star, like the Sun reveals that whilst the average pressure is reasonable, the central pressure is a couple of orders of magnitude too low, because the density of the Sun is not constant - the pressure and density are much higher in the centre.