Since you clearly state "We have to make use of the axioms for area as a set function," lets list them:
DEFINITION/AXIOMS
We assume that there exists a class $M$ of measurable sets in the plane and a set function a whose domain is $M$ with the following properties:
1) $A(S) \geq 0$ for each set $S\in M$.
2) if S and T are two sets in M their intersection and union is also in M and we have:
$A(S \cup T) = A(S) + A(T) - A(S \cap T)$.
3)If S and T are in M with $S\subseteq T$ then $T − S\subseteq M$ and $A(T − S) = A(T) − A(S)$.
4) If a set S is in M and S is congruent to T then T is also in M and $A(S) = A(T).$
5) Every rectangle R is in M. If the rectangle has length h and breadth k then $A(R) = hk$.
6) Let Q be a set enclosed between two step regions S and T. A step region is formed from a finite union of adjacent rectangles resting on a common base, i.e. $S\subseteq Q \subseteq T$. If there is a unique number $c$ such that $A(S) \le c \le A(T)$ for all such step regions $S$ and $T$, then $A(Q) = c$.
So we know by 2) that the intersection of two rectangles $R_1 \subseteq M$, $R_2 \subseteq M$ is in $M$. Use the axioms/properties listed above to determine what this means for the triangle formed by the intersection of two such rectangles.
Note also that Apostol defines a rectangle as follows: $R = \{ (x,y)| 0\le x\leq h, \text{ and}\; 0\le y\le k \}$.
Note that the union of two congruent right triangles can be positioned to form a rectangle, and alternatively, a rectangle's diagonal forms two congruent right triangles.
Refer to how Apostol defines congruence: to show two triangles (or two rectangles, or two sets of points, in general) are congruent, show that there exists a bijection between the two sets of points such that distances are preserved.
Note that this problem can be generalized to all triangles, not just right triangles. To see this, you need only note that every triangle is the union of two right triangles.
$A'(x)=0$ is equivalent to $x=\cot(x)$ or $x=1 / \tan(x)$.
There are several solutions, but the only one in $[0,\frac{\pi}{2})$ is about $0.860333589$
I suspect there is no closed form, but it is easy to find a close enough value by numerical approximation.
You then need to multiply this by its cosine, double it, and subtract the result from the total area under the curve
Best Answer
An upper bound estimate would be:
$$E \le h | f(b) - f(a) | = \dfrac{1}{6} |f(1) - f(0)| = \dfrac{1}{6}$$
If we draw the picture using six-rectangles and superimpose the parabola, we get:
Using the Right-Hand Riemann sum, we have:
$$I = h \sum_{n=1}^6 f(n/6) = \dfrac{1}{6} \left(f(1/6) + f(2/6) + f(3/6) + f(4/6) + f(5/6) + f(6/6) \right) = 0.421296$$
See if you can derive the two items above.
The actual integral result is:
$$\displaystyle \int_0^1 x^2 dx = \dfrac{1}{3}$$
Now compare the Riemann result to actual and see if that matches what the error estimate $E$ provided.