This is very close to a Padé approximant, and in this case the formula is simple enough that it's easy to derive. Firstly, we know that $\sin(x)$ is $0$ at $x=0, x=\pi$; this suggests recasting in terms of the variable $y=x(\pi-x)$. What we're after is a first-order rational approximation $\sin(x) = f(y) = \frac{ay+b}{cy+d}$; since we know that $f(y) = 0$ at $y=0$ (i.e., as $x$ approaches $0$ or $\pi$) then the constant term in the numerator is $0$, and after dividing out the approximation takes the form $\frac{y}{a+by}$.
Now, we certainly want our approximation to give $\sin(\pi/2) = f(\pi^2/4) = 1$; this means $\displaystyle{\frac{\pi^2/4}{a+b\pi^2/4}} = 1$, or $4a+b\pi^2 = \pi^2$, or $a=\frac{1-b}{4}\pi^2$. The other relation between $a$ and $b$ presumably comes from trying to match the derivative at $0$, $\left.d(\sin(x))/dx\right|_{x=0} = 1$; the condition for this this can easily be written out in terms of $df/dy$ at $y=0$. I'll spare the arithmetic (unless someone's really curious), but the result works out to be that $a=\pi$; this would give $b=(\pi-4)/\pi$ and the overall approximant $f(y) = \frac{\pi y}{\pi^2+(\pi-4)y}$, but instead the formula uses a second approximation by setting $a=5\pi^2/16$, which (thanks to the first relation) gives a rational value of $b$ (and in fact, the 'nice' value $1/4$). This approximation is equivalent to saying that $5\pi^2/16\approx\pi$, or in other words that $\pi\approx 16/5 = 3.2$; it means a slight error in the slope of the approximation at $x=0$, but that's a fair tradeoff for the ease of calculation gained.
Hint
By similarity with least square fit problems, I think that you need to minimize $$F=\int_0^{\pi}\Big(\sin(x)-(a x^2+bx+c)\Big)^2~dx$$ Compute the expression and say that $$\frac{dF}{da}=\frac{dF}{db}=\frac{dF}{dc}=0$$ This will provide three linear equations for the three unknowns $a,b,c$.
This seems to be very similar to what you posted (and the result is quite good).
Edited later
The answer you obtained is perfectly right but, if I may, I would like to add a comment about the problem itself. Fist, you notices that the function does not reproduce the values of $\sin(x)$ at the bounds.
As mentioned by Lucian, almost 1400 years ago, Mahabhaskariya of Bhaskara I, a seventh-century Indian mathematician and astronomer, proposed for the sine developements in terms of $x(\pi-x)$. So, let me consider the approximation $$\sin(x)\approx \sum_{i=1}^j a_i \big(x(\pi-x)\big)^i$$ (I excluded constant terms in order the values of $\sin(x)$ at the bounds be respected) and repeat the calculations as you made them (you could notice from your final result that the expression corresponds to $j=1$ plus a constant term).
For $j=1$, $F=2.27 \times 10^{-3}$ which is effectively worse than the $9.36 \times 10^{-4}$ you obtained (because I omitted the constant term).
For $j=2$, $F=7.89 \times 10^{-7}$ which is much better.
For $j=3$, $F=6.77 \times 10^{-10}$ which is much much better.
This was just to show that, depending on the base function we select, for the same number of adjustable parameters (or even less), we can make approximations of very different quality.
Just for your curiosity, the marvelous approximation given 1400 years ago $$\sin(x) \simeq \frac{16 (\pi -x) x}{5 \pi ^2-4 (\pi -x) x}$$ leads to $F=2.98 \times 10^{-6}$.
Best Answer
I assume you want something like this:
This is the line $y = x$ up to some maximum point $c$, which it maintains until it gets to $\pi - c$, where it drops by the line $y = \pi - x$, etc. We can control $c$ to determine the best fit.
The simplest criterion to find the best fit for is the minimum height difference between the two curves. The trapezoidal curve (the usual term of art for this sort of curve is "piecewise linear") is highest above the sine curve at $x = c$, where the distance is $c - \sin c$. The sine curve is highest above the trapezoidal curve at $x = \pi/2$, where the distance is $1 - c$. The distance will be minimized when these two distances are equal, so $c - \sin c = 1 - c$. Wolfram Alpha computes this to be when $c \approx 0.887862$.
If you want to minimize the area, that calculation is a lot more complex. By symmetry, it is sufficient to look only at the portion between $0$ and $\pi/2$, where the area is given by $$A = \left(\int_0^c x\,dx + \int_c^{\sin^{-1} c} c\,dx - \int_0^{\sin^{-1} c} \sin x\, dx\right) + \left(\int_{\sin^{-1} c}^{\pi/2} \sin x\, dx - \int_{\sin^{-1} c}^{\pi/2} c\,dx\right)\\=\frac{c^2}2 + c(\sin^{-1}c - c) - (1 - \cos(\sin^{-1}c)) + \cos(\sin^{-1}c) -c(\frac{\pi}2- \sin^{-1}c)\\=-1 - \frac\pi 2 c - \frac {c^2}2 +2\sqrt{1 - c^2} + 2c\sin^{-1}c$$
To minimize the area, we take the derivative with respect to $c$ and set it to 0: $$A' = -\frac \pi 2 - c -\frac{2c}{\sqrt{1 - c^2}} + 2\sin^{-1}c + \frac {2c}{\sqrt{1-c^2}}=2\sin^{-1}c-\frac \pi 2 - c = 0 $$ Which Wolfram Alpha says is $c \approx 0.95261$.
Of course, I just used $y = \sin x$, whereas your curve will need to be of the form $$y = A\sin(Bx)$$ but the same solution should work using lines of slope $\pm AB$, and a cutoff value of $C = Ac$ for your choice of the two $c$ values calculated above.