We know that $f_{XY} (x,y)$ is constant inside the triangle since $(X,Y)$ is uniformly chosen. Then the integral of the p.d.f. over the triangle $T$ with base $b$ and height $h$ reduces like this:
$$\iint \limits_Tf_{XY} dA = bh/2 \cdot f_{XY} = 1$$
And since $b = h =1$, $f_{XY}=2$ inside the triangle.
You are right to think of the probabilities as areas, but the set of points closer to the center is not a triangle. It's actually a weird shape with three curved edges, and the curves are parabolas.
The set of points equidistant from a line $D$ and a fixed point $F$ is a parabola. The point $F$ is called the focus of the parabola, and the line $D$ is called the directrix. You can read more about that here.
In your problem, if we think of the center of the triangle $T$ as the focus, then we can extend each of the three edges to give three lines that correspond to the directrices of three parabolas.
Any point inside the area enclosed by the three parabolas will be closer to the center of $T$ than to any of the edges of $T$. The answer to your question is therefore the area enclosed by the three parabolas, divided by the area of the triangle.
Let's call $F$ the center of $T$. Let $A$, $B$, $C$, $D$, $G$, and $H$ be points as labeled in this diagram:
The probability you're looking for is the same as the probability that a point chosen at random from $\triangle CFD$ is closer to $F$ than to edge $CD$. The green parabola is the set of points that are the same distance to $F$ as to edge $CD$.
Without loss of generality, we may assume that point $C$ is the origin $(0,0)$ and that the triangle has side length $1$. Let $f(x)$ be equation describing the parabola in green.
By similarity, we see that $$\overline{CG}=\overline{GH}=\overline{HD}=1/3$$
An equilateral triangle with side length $1$ has area $\sqrt{3}/4$, so that means $\triangle CFD$ has area $\sqrt{3}/12$. The sum of the areas of $\triangle CAG$ and $\triangle DBH$ must be four ninths of that, or $\sqrt{3}/27$.
$$P\left(\text{point is closer to center}\right) = \displaystyle\frac{\frac{\sqrt{3}}{12} - \frac{\sqrt{3}}{27} - \displaystyle\int_{1/3}^{2/3} f(x) \,\mathrm{d}x}{\sqrt{3}/12}$$
We know three points that the parabola $f(x)$ passes through. This lets us create a system of equations with three variables (the coefficients of $f(x)$) and three equations. This gives
$$f(x) = \sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}$$
The integral of this function from $1/3$ to $2/3$ is $$\int_{1/3}^{2/3} \left(\sqrt{3}x^2 - \sqrt{3}x + \frac{\sqrt{3}}{3}\right) \,\mathrm{d}x = \frac{5}{54\sqrt{3}}$$
This gives our final answer of $$P\left(\text{point is closer to center}\right) = \boxed{\frac{5}{27}}$$
Best Answer
One would think this would be the ratio of areas. The area within a distance $r$ from a corner is just: $$A= \frac{60^\circ (\pi r^2)}{360^\circ}=\frac{\pi r^2}{6}$$ So the area of three corners is $\pi r^2 / 3$, assuming $2r$ is smaller than the side length, so that the circles don't overlap. In your case this condition is satisfied, since $2r=1/2 < 1$. Now, if you know the area of a triangle, can you finish the calculation?