I answered a more general form of this problem here. Think of the integral as the sum of all the angles you can have, and the division as averaging it out. Just as $\frac{1}{n}\sum_{i=1}^n f(i)$ is an average, so is $\frac{1}{\pi}\int_0^\pi f(\theta)\mathrm d\theta$. So, in this case, you want to take the average of all the $\theta$ you can have, but there are an infinite amount of these, so how do you take an average? By this integral, of course!
Once you have an average $\theta$, you need to recognize that the length of the arc the third point can occupy in order to form the triangle is exactly $\theta$, the length of the minor arc between the first two points. The probability of landing in this arc is $\theta/2\pi$, and thus your integral.
You are right to think of the probabilities as areas, but the set of points closer to the center is not a triangle. It's actually a weird shape with three curved edges, and the curves are parabolas.
The set of points equidistant from a line $D$ and a fixed point $F$ is a parabola. The point $F$ is called the focus of the parabola, and the line $D$ is called the directrix. You can read more about that here.
In your problem, if we think of the center of the triangle $T$ as the focus, then we can extend each of the three edges to give three lines that correspond to the directrices of three parabolas.
Any point inside the area enclosed by the three parabolas will be closer to the center of $T$ than to any of the edges of $T$. The answer to your question is therefore the area enclosed by the three parabolas, divided by the area of the triangle.
Let's call $F$ the center of $T$. Let $A$, $B$, $C$, $D$, $G$, and $H$ be points as labeled in this diagram:
The probability you're looking for is the same as the probability that a point chosen at random from $\triangle CFD$ is closer to $F$ than to edge $CD$. The green parabola is the set of points that are the same distance to $F$ as to edge $CD$.
Without loss of generality, we may assume that point $C$ is the origin $(0,0)$ and that the triangle has side length $1$. Let $f(x)$ be equation describing the parabola in green.
By similarity, we see that $$\overline{CG}=\overline{GH}=\overline{HD}=1/3$$
An equilateral triangle with side length $1$ has area $\sqrt{3}/4$, so that means $\triangle CFD$ has area $\sqrt{3}/12$. The sum of the areas of $\triangle CAG$ and $\triangle DBH$ must be four ninths of that, or $\sqrt{3}/27$.
$$P\left(\text{point is closer to center}\right) = \displaystyle\frac{\frac{\sqrt{3}}{12} - \frac{\sqrt{3}}{27} - \displaystyle\int_{1/3}^{2/3} f(x) \,\mathrm{d}x}{\sqrt{3}/12}$$
We know three points that the parabola $f(x)$ passes through. This lets us create a system of equations with three variables (the coefficients of $f(x)$) and three equations. This gives
$$f(x) = \sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}$$
The integral of this function from $1/3$ to $2/3$ is $$\int_{1/3}^{2/3} \left(\sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}\right) \,\mathrm{d}x = \frac{5}{54\sqrt{3}}$$
This gives our final answer of $$P\left(\text{point is closer to center}\right) = \boxed{\frac{5}{27}}$$
Best Answer
No. Generally speaking you need to calculate the quadrangle defined by B, the midpoint of side AB, the midpoint of side CB and the point where the two perpendicular bisectors of the sides AB and CB meet. See https://en.wikipedia.org/wiki/Voronoi_diagram . The probability is the area of that quadrangle divided through the area of the triangle.