You are right to think of the probabilities as areas, but the set of points closer to the center is not a triangle. It's actually a weird shape with three curved edges, and the curves are parabolas.
The set of points equidistant from a line $D$ and a fixed point $F$ is a parabola. The point $F$ is called the focus of the parabola, and the line $D$ is called the directrix. You can read more about that here.
In your problem, if we think of the center of the triangle $T$ as the focus, then we can extend each of the three edges to give three lines that correspond to the directrices of three parabolas.
Any point inside the area enclosed by the three parabolas will be closer to the center of $T$ than to any of the edges of $T$. The answer to your question is therefore the area enclosed by the three parabolas, divided by the area of the triangle.
Let's call $F$ the center of $T$. Let $A$, $B$, $C$, $D$, $G$, and $H$ be points as labeled in this diagram:
The probability you're looking for is the same as the probability that a point chosen at random from $\triangle CFD$ is closer to $F$ than to edge $CD$. The green parabola is the set of points that are the same distance to $F$ as to edge $CD$.
Without loss of generality, we may assume that point $C$ is the origin $(0,0)$ and that the triangle has side length $1$. Let $f(x)$ be equation describing the parabola in green.
By similarity, we see that $$\overline{CG}=\overline{GH}=\overline{HD}=1/3$$
An equilateral triangle with side length $1$ has area $\sqrt{3}/4$, so that means $\triangle CFD$ has area $\sqrt{3}/12$. The sum of the areas of $\triangle CAG$ and $\triangle DBH$ must be four ninths of that, or $\sqrt{3}/27$.
$$P\left(\text{point is closer to center}\right) = \displaystyle\frac{\frac{\sqrt{3}}{12} - \frac{\sqrt{3}}{27} - \displaystyle\int_{1/3}^{2/3} f(x) \,\mathrm{d}x}{\sqrt{3}/12}$$
We know three points that the parabola $f(x)$ passes through. This lets us create a system of equations with three variables (the coefficients of $f(x)$) and three equations. This gives
$$f(x) = \sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}$$
The integral of this function from $1/3$ to $2/3$ is $$\int_{1/3}^{2/3} \left(\sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}\right) \,\mathrm{d}x = \frac{5}{54\sqrt{3}}$$
This gives our final answer of $$P\left(\text{point is closer to center}\right) = \boxed{\frac{5}{27}}$$
Best Answer
Let the figure be a square with each side = s and the point be (a, b).
More than half of the co-ordinates are then known (in terms of s, a, b) except h, k, p and m but they can all be expressed in terms of s, a, and b.
For example, (m, 0) is a point on the line formed by (0, s) and (a, b). The corresponding equation is $\frac {y – b}{x – a} = \frac {s – b}{0 – a}$. (m, 0) lies on this line means $\frac {[0] – b}{[m] – a} = \frac {s – b}{0 – a}$.
Note that it is not necessary to calculate all the triangles area. This is because the sum of the areas all the light green triangles (originally was dark green) is equal to half of the area of the square. Thus, we only need to find the areas of triangles XHK, XAM and XBP.