Area ABF = Area DEC, and for each of these triangles, the area = $\frac{1}{2}x^2sin 120^{\circ}$, or $\frac{1}{2}(4\sqrt{\frac{7}{3}})^2 \times \frac{\sqrt{3}}{2}$. The area of both of these triangles, then, is $56\frac{\sqrt{3}}{3}$.
Remaining is the parallelogram BCEF. The base is the length of FB, or $4\sqrt{7}$. The height is the perpendicular distance between the lines, given by $\frac{\vert{b_2 - b_1}\vert}{\sqrt{m^2 + 1}}$, or $\frac{22\sqrt{3}}{3\sqrt{7}}$. The parallelogram's area, then, is $88\frac{\sqrt{3}}{3}$. Adding it all together yields an area of $48\sqrt{3}$.
Note, I think it's much simpler to compute the area of ABDE and the other two triangles. The x-coordinate of B is $\sqrt{x^2 - 4}$, or $10\frac{\sqrt{3}}{3}$. Then the area of ABDE is 8 $\times$ $10\frac{\sqrt{3}}{3}$. The total area of the other two triangles BCD and EFA is 8 $ \times$ $8\frac{\sqrt{3}}{3}$, yielding a total area of $48\sqrt{3}$.
A formula will not be able to deal directly with the edge cases. To convert from $(X,Y)$ to $(H_q,H_r)$ you will need an algorithm. To begin with, imagine vertical lines draw through the top left points of the hexagons. This divides the grid into columns, and you need to know which column the point is in. This is simply $\lfloor\frac{x}{18}\rfloor$. We'll call this value $a$. We then divide that column into cells using the horizontal lines of the hexagons in that column so that each cell is an $18\times14$ rectangle with its top left corner concurrent with that of a hexagon, then find a value $b$ for the cell containing the given point.
When $a$ is even we calculate $b=\lfloor\frac{-y}{14}\rfloor$, and when $a$ is odd $b=\lfloor\frac{-y-7}{14}\rfloor$.
Using these values we find $H_r=b-\lfloor\frac{a}{2}\rfloor$ and $H_q=H_r+a$. Also use these values to find the $(x,y)$ values for the top left corner $P$ of the cell.
When $a$ is even $P_{x,y}=(18a,-14b)$, when $a$ is odd $P_{x,y}=(18a,-14b-7)$.
Now check the location of the given point within the cell to see if it falls in one of the corner regions.
First let $T_x=X-P_x-11$. If this value $T_x\le0$ then the point is in the main region and our values for $H_q$ and $H_r$ are correct. Otherwise, let $T_y=P_y-Y$.
Finally, if $T_y<7$ and $T_x>T_y$ we need subtract $1$ from $H_r$, or if $T_y>7$ and $T_x>14-T_y$ we need to add $1$ to $H_q$.
C code reference (tested and confirmed):
a=floor(x/18);
if(((int)a%2)==0){
b=floor(-y/14);
px=18*a;
py=-14*b;
}else{
b=floor((-y-7)/14);
px=18*a;
py=-14*b-7;
}
hr=b-floor(a/2);
hq=hr+a;
tx=x-px-11;
if(tx>0){
ty=py-y;
if((ty<7)&&(tx>ty))hr-=1;
if((ty>7)&&(tx>14-ty))hq+=1;
}
Best Answer
In $\mathbb R^2$, there is no regular hexagon whose vertices are on the lattice points.
Proof :
Suppose that there exists a regular hexagon $A_1A_2\cdots A_6$ whose vertices are on the lattice points.
Then, the triangle $A_1A_3A_5$ is an equilateral triangle. WLOG, we may set $A_1(0,0),A_3(a,b),A_5(c,d)$ where $(a,b)\not=(0,0)$. Then, considering the area, we get $$\frac 12|ad-bc|=\frac{\sqrt 3}{4}(a^2+b^2),$$ i.e. $$\sqrt 3=\frac{2|ad-bc|}{a^2+b^2}$$ The LHS is irrational, the RHS is rational, a contradiction.
By the way, in $\mathbb R^3$, there are regular hexagons whose vertices are on lattice points.
Here is an example :
$$(0,-1,-1),\quad (1,0,-1),\quad (1,1,0)$$ $$(0,1,1),\quad (-1,0,1),\quad (-1,-1,0)$$
These are the midpoints of the six sides of the cube whose vertices are $(\pm 1,\pm 1,\pm 1)$.