Let $\displaystyle C = \sqrt{A^2 + B^2}$.
Then $\displaystyle A \cos t + B \sin t = C \left(\frac{A}{\sqrt{A^2+B^2}} \cos t + \frac{B}{\sqrt{A^2+B^2}} \sin t\right)$, and we can visualize a right triangle with opposite $A$ and adjacent $B$ to notice that we can find an angle $\phi$ such that
$\displaystyle \frac{A}{\sqrt{A^2+B^2}} = \sin \phi$
and
$\displaystyle \frac{B}{\sqrt{A^2+B^2}} = \cos \phi$.
This means that $\displaystyle A \cos t + B \sin t = C (\sin \phi \cos t + \cos \phi \sin t) = C \sin (\phi + t)$. Now we can easily see that the maximum is $C = \sqrt{A^2+B^2}$ and the minimum is $-C = -\sqrt{A^2+B^2}$.
And of course, the maximum is reached when $\displaystyle \phi + t = \frac{\pi}{2}$, and the minimum when $\displaystyle \phi + t = \frac{3\pi}{2}$.
Proceeding in the usual way, find critical points of the function by setting its partial derivatives to zero and solving. You’re using $f$ for both the function and one of the coefficients in its definition, so I’ll call the function $F$ here. To reduce clutter, let $G(x,y)=dx^2+ey^2+fxy+gx+hy+i$, so that $F(x,y) = (ax+by+c) G(x,y)^{-1/2}$. Differentiating, we have
$$\begin{align} F_x(x,y) &= (a G(x,y) - \frac12 (ax+by+c) (2dx+fy+g))G(x,y)^{-3/2} = 0 \\
F_y(x,y) &= (b G(x,y) - \frac12(ax+by+c)(fx+2ey+h))G(x,y)^{-3/2} = 0, \end{align}$$ so the problem becomes that of finding the intersections of a pair of conics. A general strategy for doing this is to find a degenerate linear combination of the two conics, split it into linear factors and then solve the simpler line-conic intersections. In this case, we’re already most of the way there: subtract $2a$ times the second equation from $2b$ times the first to get $$(ax+by+c)\left[ (af-2bd)x + (2ae-bf)y + (ah-bg)\right] = 0,$$ which is already split into linear terms. The intersection points of the two conics are thus the intersections of the two lines defined by these factors with either of the two original conics.
Turning to the first line, setting $ax+by+c=0$ reduces the partial derivative equations to $aG(x,y)=0$ and $bG(x,y)=0$. Unless $a=b=0$, this means that $G(x,y)=0$, but $F$ is undefined in this region, so there can be no critical points there. That leaves the intersections of the line $$(af-2bd)x + (2ae-bf)y + (ah-bg) = 0$$ with either of the other conics, which I’ll leave to you to work out.
If you’re interested in exploring the qualitative behavior of this family of functions, I’d suggest examining the simpler cases in which $G(x,y)=0$ is one of the canonical examples of the different types of conics. All of the functions in this family can be obtained via an affine coordinate transformation from one of these canonical equations, which will move the critical points in a well-defined way.
Best Answer
I found a solution. Since $$f(x) = \sin x \left( 1 + \int_{-\frac \pi 2}^{\frac \pi 2} f(t)dt\right) + \cos x \int_{-\frac \pi 2}^{\frac \pi 2} tf(t)dt$$ We can rewrite it as $$f(x) = A\sin x + B\cos x$$ This gives us the equations $$\begin{gather} A = 1 + \int_{-\frac \pi 2}^{\frac \pi 2}f(t)dt \tag{1} \\ B = \int_{-\frac \pi 2}^{\frac \pi 2} tf(t)dt \tag{2} \end{gather}$$
Using some integration properties, it is easy to see that $\int_{-\frac \pi 2}^{\frac \pi 2}f(t) = \int_{-\frac \pi 2}^{\frac \pi 2}B\cos t dt$ and $\int_{-\frac \pi 2}^{\frac \pi 2}tf(t)dt = \int_{-\frac \pi 2}^{\frac \pi 2}At\sin t dt$. Evaluating these integrals and substituting in the equations, they simplify to $$\begin{gather} A = 1 + 2B \tag{1} \\ B = 2A \tag{2} \end{gather}$$ Solving this system gives $A = -\frac 13$ and $B = -\frac 23$. Thus $$f(x) = -\frac 13 \sin x - \frac 23 \cos x$$
The maximum and minimum values of $f(x)$ are $\frac{\sqrt{5}}{3}$ and $-\frac{\sqrt{5}}{3}$ respectively.