Let be $f:[a,b]\to\mathbb{R}$ a continuous function.
Show that if $\int\limits_a^bf(x)g(x)dx\geq 0$ holds for all continuous functions $g:[a,b]\to\mathbb{R}$ with $g(a)=g(b)=0$ and $g(x)\geq 0$ for all $x\in]a,b[$, then $f(x)\geq 0$ for all $x\in[a,b]$.
My approach:
Let's assume there exists a $x_0\in]a,b[$ such that $f(x_0)<0$. If we set $\epsilon:=-\frac{f(x_0)}{2}>0$, then by continuity of $f$ there exists a $\delta>0$ such that for all $x\in[a,b]$ with $|x-x_0|<\delta\implies |f(x)-f(x_0)|<-\frac{f(x_0)}{2}$. This means that $f(x)<0$ for all $x\in]x_0-\delta,x_0+\delta[$. Next, we define the function $g:[a,b]\to\mathbb{R}$ where
$$
g(x):=\begin{cases}0,&x\in[a,x_0-\delta]\\x-(x_0-\delta),&x\in]x_0-\delta,x_0]\\ (x_0+\delta)-x,&x\in]x_0,x_0+\delta[\\0,&x\in[x_0+\delta,b].\end{cases}
$$
If we check the limits at the "critical" three points $x_0-\delta,x_0$ and $x_0+\delta$ we see immediately that $g$ is continuous. Further, we see that
\begin{align*}
&f(x)g(x)=0, x\in[a,x_0-\delta]\\
&f(x)g(x)\leq\frac{f(x_0)}{2}(x-x_0+\delta)\leq0, x\in[x_0-\delta,x_0]\\
&f(x)g(x)\leq\frac{f(x_0)}{2}(x_0+\delta-x)\leq0, x\in[x_0,x_0+\delta]\\
&f(x)g(x)=0, x\in[x_0+\delta,b].
\end{align*}
Using the well known operations of Riemann integrals we get:
\begin{align*}
&\int\limits_a^bf(x)g(x)dx\\
&=\underset{=0}{\underbrace{\int\limits_a^{x_0-\delta}f(x)g(x)dx}}+\underset{\leq0}{\underbrace{\int\limits_{x_0-\delta}^{x_0}f(x)g(x)dx}}+\underset{\leq0}{\underbrace{\int\limits_{x_0}^{x_0+\delta}f(x)g(x)dx}}+\underset{=0}{\underbrace{\int\limits_{x_0+\delta}^bf(x)g(x)dx}}\leq0.
\end{align*}
We know that if $h$ is a continuous function, then $\int\limits_a^bh(x)dx=0\implies h(x)=0$ for all $x\in[a,b]$. As $f(x)g(x)\neq 0$ for particular $x\in[a,b]$ it follows that $\int\limits_a^bf(x)g(x)dx\neq 0$. Hence, $\int\limits_a^bf(x)g(x)dx<0$. This is a contradiction so there mustn't be a $x_0$ such that $f(x_0)<0$.
Note that if we had assumed $x_0=a$ or $x_0=b$ then we could adjust $g$ accordingly and get the same result.
Is this correct? Do you have any suggestions or maybe a shorter proof?
Also: The function $g$ I have constructed satisfies $g(a)=g(b)=0$ but I don't see why it should be necessary?! Maybe there are other functions which we can use to achieve a contradiction which doesn't satisfy $g(a)=g(b)=0$?
EDIT
We know that if $h$ is a continuous function with $h(x)\geq0$ for all $x\in[a,b]$, then $\int\limits_a^bh(x)dx=0\implies h(x)=0$ for all $x\in[a,b]$. As $(-f(x)g(x))\geq 0$ for all $x\in[x_0-\delta,x_0+\delta]$ and $(-f(x)g(x))\neq 0$ for particular $x\in[x_0-\delta,x_0+\delta]$ it follows that $\int\limits_a^b-f(x)g(x)dx\neq 0$. Hence, $\int\limits_a^bf(x)g(x)dx\neq0$ which shows that $\int\limits_a^bf(x)g(x)dx<0$. This is a contradiction so there mustn't be a $x_0$ such that $f(x_0)<0$.
Best Answer
This is almost correct. The problem starts when you say "we know $\int h=0\implies h=0$". This is true only if we know that $h\ge0$ (what about $\int_{-1}^1xdx$ for example?).
But the reasoning is correct: assume that $f(x_0)<0$ for some $x_0$ and, by continuity, find some $\varepsilon>0$ and an interval $I=[s,t]\subset[a,b]$ around $x_0$ such that $f(x)<-\varepsilon$ for all $x\in I$. Then define $g:[a,b]\to\mathbb{R}$ by $g(x)=0$ outside $I$, $g(x_0)=1$ and extend linearly on $[s,t]$ so that $g$ is continuous (so $g$ will look like a small peak at $x_0$). By assumption $\int_a^bfg\ge0$. However, let's compute this integral: $$\int_a^bf(x)g(x)dx=\int_s^tf(x)g(x)dx\le-\varepsilon\int_s^tg(x)dx=-\varepsilon\frac{|s-t|}{2}<0$$ where to compute the integral of $g$ on $[s,t]$ we used the formula "area of isosceles triangle = length of base $\times$ $\frac{1}{2}$height" (note, the height is equal to $g(x_0)=1$ and the length of base is equal to the length of $I$, i.e. equal to $|s-t|$). This is a contradiction.