First Derivative Test:
Find all points where $f'(x)=0$ or is undefined, these are called critical points. Any of these could be an extrema. However that's not the only option. The point could also be an "inflection point".
An inflection point $(a, f(a))$, is a point where the derivative $f'(a)=0$, but where $f'(a-\delta)$ and $f'(a+\delta)$, where $\delta$ is a very small number, have the same sign (both $f'(a-\delta)$ & $f'(a+\delta)$ are positive, or both are negative). An inflection point, is neither a maximum or a minimum.
NOTE: This is not the technical definition of an inflection point, but it does provide a method that will work for any continuous function $f: D \subset \Bbb R \to \Bbb R$, assuming the $\delta$ you choose is "small enough".
Second Derivative Test:
Plug in the values you found for $f'(x)=0$ into the second derivative $f''(x)$. If:
$\begin{cases} f''(x)\gt 0 & \text{Then f(x) is a relative minimum} \\
f''(x)\lt 0 & \text{Then f(x) is a relative maximum} \\f''(x)=0 & \text{Then the test is inconclusive} \end{cases}$
So for your example:
$f'(x)=6x^2-4x^3=0$ when $x=0$ or $x=\frac 32$. And $f'(x)$ is a polynomial so it is defined over the entire domain of $f$.
That means that your potential relative maxima and minima are at $(0,f(0))$ and $(\frac 32, f(\frac 32))$.
Let's use the second derivative test:
$f''(x)=12x-12x^2=12x(1-x)=0$ when $x=0$ or $x=1$ and is again defined everywhere. So let's figure out where it's positive and negative by choosing three points $c_1, c_2, c_3$ such that $c_1 \lt 0 \lt c_2 \lt 1 \lt c_3$. How about $c_1=-1$, $c_2=\frac 12$, and $c_3=2$ -- those seem like pretty easy numbers.
Plugging those into $f''(x)$, we get
$f''(-1)=-12-12=-24\lt 0$
$f''(\frac 12)=6-3=3 \gt 0$ and
$f''(2) = 24-48=-24 \lt 0$
So we've found that $f(x)$ is concave up on the interval $(0,1)$ and concave down on the set $(-\infty, 0)\cup(1,\infty)$.
So now let's evaluate the points $x=0$ and $x=\frac 32$ that we found via the first derivative test. We see that $\frac 32 \in (-\infty, 0)\cup(1,\infty)$, so we know that the point $(\frac 32, f(\frac 32))$ is a relative maximum.
On the other hand $x=0$ is inconclusive with the second derivative test. So let's check if it's an inflection point via the method I described above. Let's make our $\delta=\frac 14$ and evaluate $f'(-\frac 14)$ and $f'(\frac 14)$.
$f'(-\frac 14)=6(\frac {1}{16})-4(\frac{-1}{64})=\frac {7}{16} \gt 0$ and
$f'(\frac 14)=6(\frac {1}{16})-4(\frac{1}{64})=\frac {5}{16} \gt 0$
So $(0,f(0))$ is an inflection point of $f(x)$ and thus not an extrema.
Hint
You have
$$f'(x)=4(\cos^2 x-\sin^2 x).$$ To get the critical points you have to solve $f'(x)=0.$ You have done it and you have obtained $x=\pi/4$ and $x=3\pi/4.$ Now, you have to study the sign of $f$ on the intervals $(0,\pi/4),$ $(\pi/4,3\pi/4)$ and $(3\pi/4,\pi).$ ($0$ and $\pi$ because they are the extremes of the interval where the function is defined and $x\pi/4$ and $3\pi/4$ because they are the critical points.)
Remember that if $f'(x)>0,\: x\in (a,b)$ then $f$ is strictly increasing in $(a,b)$ and if $f'(x)<0,\: x\in (a,b)$ then $f$ is strictly decreasing in $(a,b).$ A local minimum is obtained when you change from an interval where the function is decreasing to an interval where it is increasing. It is similar for a local maximum. (This is the first derivative test.)
Can you finish?
Best Answer
Note that $a=0$ is special. Apart from not being defined at $0$, our function is then the same as the function $g(x)=x$, and has no local max or min.
We deal now with $a\gt 0$ and $a\lt 0$. I would deal with these separately.
Case $a\gt 0$: The derivative is $0$ at $x=0$ and at $x=2a$. We examine each of these points in turn.
(i) The bottom of the derivative is safely positive. Consider the top $x(x-2a)$ very near $x=2a$. Note that $x$ is positive. For $x$ near $2a$ but smaller than $2a$, we see that $x-2a$ is negative. For $x\gt 2a$, it is positive. so the derivative is negative for $x$ smaller than $2a$, but near $2a$, and positive afterwards. Thus our function near $2a$ is going down and then up, we have a local minimum at $x=2a$.
(ii) We now look at the derivative near $x=0$. If $x\lt 0$, then $x$ is negative. Also, $x-2a$ is negative there, so the derivative is positive. For $x\gt 0$ but close to $0$, we have $x$ positive and $x-2a$ negative, so the derivative is negative. Thus near $x=0$ our function is going up and then down, we have a local max at $x=0$. But the problem only asks about the interval $(0,\infty)$, so we needn't have bothered.
Case $a\lt 0$: We are only asked for local max and min in $(0,\infty)$. In the case $a\lt 0$ there are no critical points in $(0,\infty)$, so no local max or min. without that restriction, the analysis would go along lines similar to the case $a\gt 0$.