The key fact is that every simple polygon, not necessarily convex, can be decomposed into $n-2$ triangles by drawing $n-3$ diagonals. Then the sum of the interior angles of the polygon is equal to the sum of interior angles of all triangles, which is clearly $(n-2)\pi$.
The existence of triangulations for simple polygons follows by induction once we prove the existence of a diagonal.
For a proof, see Chapter 1 of Discrete and Computational Geometry by Devadoss and O'Rourke. This chapter is freely available.
The picture below from that chapter that captures the gist of the proof:
See also Diagonals: Feature Column from the AMS by Malkevitch.
Credits to @Siddhant. After reading his answer, another method came to my mind and I wish to share it here.
The coordinates of points A,B and D have been given in the question. So these points are fixed. Point C is the variable point on the given parabola. We need to choose the appropriate value of C so that the area of the quadrilateral ABCD is maximum.
Let us consider the following diagram.
Area of the quadrilateral ABCD is nothing but the sum of areas of the triangles ABD and BCD. Since points A,B and D are fixed, the area of triangle ABD is constant for the given question. Since C is a variable point and is not moving parallel to the line BD, the area of triangle BCD is not constant.
Our objective is to maximize the area of the quadrilateral ABCD. Since area of ABD is fixed. We need to maximise the are of triangle BCD.
We know that, area of a triangle is given by $\frac 1 2 bh$ where $b$ and $h$ are the base and hypotenuse of the triangle respectively. In triangle BCD, let us consider the base as the line segment BD and the height represented by the green coloured line in the above diagram. Since B and D are fixed, the length of base BD is fixed. Now, in order to maximise the area of triangle BCD, it is sufficient to find the coordinates for the point C when it is at the maximum distance from the line segment BD(base of BCD).
Now let us find the equation of the line BD using the coordinates of B and D.
$$\frac{y-1}{x+1}=\frac 6 3 = 2$$
$$y-1=2(x+1)$$
$$2x-y+3=0$$
Next, let us compute the distance (or height $h$) of the variable point C$(p,p^2+p+1)$ from the base BD as shown below:
$$h=\frac{2p-(p^2+p+1)+3}{\sqrt{2^2+1^2}}$$
$$h=\frac{-p^2+p+2}{\sqrt 3}$$
We have obtained $h$ as a function of $p$. Now we need to maximize $h$. It is easy to see that the quadratic in $p$ attains the highest value at $p=1/2$.
Hence the coordinate for point C such that the area of quadrilateral ABCD is maximum is $(1/2,7/4)$.
Best Answer
You're describing the famous shoelace formula, also known as Gauss's area formula: https://en.wikipedia.org/wiki/Shoelace_formula