As suggested in comments by @Dahaka
Let $I(b)$ represent the given integral. Hence we have $I(0)=0$.
Now differentiating $I(b)$ with respect to $b$ we get $$I'(b) =\int_0^{\infty} \csc ^2x \cdot \frac {a-b\sin^2x}{a+b\sin^2x}\cdot\left( \frac {\sin^2x(a-b\sin^2x)+\sin^2x(a+b\sin^2x)}{(a-b\sin^2x)^2}\right) dx$$
Simplifying we get $$I'(b) =\int_0^{\infty} \left(\frac {1}{a-b\sin^2x} -\frac {1}{a+b\sin^2x}\right)dx$$
On dividing both the numerator and denominator by $\cos^2x$ and then letting $u$ substitution $u=\tan x$ we get $$I'(b) =\int_0^{\infty} \left(\frac {1}{a+(a-b)u^2}+ \frac {1}{a+(a+b)u^2}\right) du$$
And both of these simply evaluate using the fact that $$\int \frac {dx}{a^2+x^2}=\frac 1a \arctan \left(\frac xa\right)+ C$$
Hence $$I'(b) =\frac {\pi}{2\sqrt a}\left( \frac {1}{\sqrt {a-b}}+\frac {1}{\sqrt {a+b}}\right) $$
And now integrating again with respect to $b$ yields $$I(b)=\frac {\pi(\sqrt {a+b}-\sqrt {a-b})}{\sqrt a} +C$$
But using the fact that $I(0)=0$ we get $C=0$ and hence $$I(b)=\frac {\pi(\sqrt {a+b}-\sqrt {a-b})}{\sqrt a} $$
Q. E. D
\begin{align}I&=\int_{0}^{\frac{\pi}{2}}\frac{dx}{\left(\sqrt{\sin x}+\sqrt{\cos x}\right)^2}\\
&=\int_{0}^{\frac{\pi}{4}}\frac{dx}{\left(\sqrt{\sin x}+\sqrt{\cos x}\right)^2}
+\int_{\frac{\pi}{4}}^{\frac{\pi}{2}}\frac{dx}{\left(\sqrt{\sin x}+\sqrt{\cos x}\right)^2}\\\end{align}
In the second integral perform the change of variable $y=\frac{\pi}{2}-x$,
\begin{align}I&=2\int_{0}^{\frac{\pi}{4}}\frac{dx}{\left(\sqrt{\sin x}+\sqrt{\cos x}\right)^2}\\
\end{align}
Perform the change of variable $y=\tan x$,
\begin{align}
I&=2\int_{0}^{1}\frac{1}{(1+\sqrt{x})^2\sqrt{1+x^2}}\,dx\\\end{align}
Perform the change of variable $y=\dfrac{1-x}{1+x}$,
\begin{align}I&=\sqrt{2}\int_{0}^{1}\frac{1-\sqrt{1-x^2}}{x^2\sqrt{1+x^2}}\,dx\\
&=-\sqrt{2}\Big[\frac{1-\sqrt{1-x^2}}{x\sqrt{1+x^2}}\Big]_0^1-\sqrt{2}\int_0^1 \frac{\sqrt{1-x^2}-2}{\sqrt{1-x^2}(1+x^2)^{\frac{3}{2}}}\,dx\\
&=-1-\sqrt{2}\int_0^1 \frac{1}{(1+x^2)^{\frac{3}{2}}}\,dx+2\sqrt{2}\int_0^1 \frac{1}{\sqrt{1-x^2}(1+x^2)^{\frac{3}{2}}}\,dx\\
&=-1-\sqrt{2}\left[\frac{x}{\sqrt{1+x^2}}\right]_0^1+2\sqrt{2}\int_0^1 \frac{1}{\sqrt{1-x^2}(1+x^2)^{\frac{3}{2}}}\,dx\\
&=2\sqrt{2}\int_0^1 \frac{1}{\sqrt{1-x^2}(1+x^2)^{\frac{3}{2}}}\,dx-2\\
\end{align}
Perform the change of variable $y=\dfrac{1-x}{1+x}$,
\begin{align}I&=\int_0^1 \frac{x^2+1+2x}{\sqrt{x}(1+x^2)^{\frac32}}\,dx-2\\
&=\int_0^1 \frac{1}{\sqrt{x}\sqrt{1+x^2}}\,dx+2\int_0^1 \frac{\sqrt{x}}{(1+x^2)^{\frac32}}\,dx-2\\
\end{align}
Perform the change of variable $y=\sqrt{x}$ in both integrals,
\begin{align}I&=2\int_0^1 \frac{1}{\sqrt{1+x^4}}\,dx+4\int_0^1 \frac{x^2}{(1+x^4)^{\frac32}}\,dx-2\end{align}
\begin{align}A&=\int_0^1 \frac{1}{\sqrt{1+x^4}}\,dx\end{align}
Perform the change of variable $y=\frac{1}{x}$,
\begin{align}A&=\int_1^\infty \frac{1}{\sqrt{1+x^4}}\,dx=\int_0^\infty \frac{1}{\sqrt{1+x^4}}\,dx-\int_0^1 \frac{1}{\sqrt{1+x^4}}\,dx\\
&=\int_0^\infty \frac{1}{\sqrt{1+x^4}}\,dx-A
\end{align}
Therefore,
\begin{align}A&=\frac{1}{2}\int_0^\infty \frac{1}{\sqrt{1+x^4}}\,dx\end{align}
In the same manner one obtains,
\begin{align}\int_0^1 \frac{x^2}{(1+x^4)^{\frac32}}\,dx&=\frac{1}{2}\int_0^\infty \frac{x^2}{(1+x^4)^{\frac32}}\,dx\end{align}
Therefore,
\begin{align}I&=\int_0^\infty \frac{1}{\sqrt{1+x^4}}\,dx+2\int_0^\infty \frac{x^2}{(1+x^4)^{\frac32}}\,dx-2\end{align}
Perform the change of variable $y=x^4$,
\begin{align}I&=\frac{1}{4}\int_0^\infty \frac{x^{-\frac34}}{(1+x)^{\frac12}}\,dx+\frac{1}{2}\int_0^\infty \frac{x^{-\frac14}}{(1+x)^{\frac32}}\,dx-2\\
&=\frac{1}{4}\text{B}\left(\frac{1}{4},\frac{1}{4}\right)+\frac{1}{2}\text{B}\left(\frac{3}{4},\frac{3}{4}\right)-2\\
&=\frac{1}{4}\times \frac{\Gamma^2\left(\frac{1}{4}\right)}{\Gamma\left(\frac{1}{2}\right)}+\frac{1}{2}\times \frac{\Gamma^2\left(\frac{3}{4}\right)}{\Gamma\left(\frac{3}{2}\right)}-2\\
&=\frac{1}{4}\times \frac{\Gamma^2\left(\frac{1}{4}\right)}{\Gamma\left(\frac{1}{2}\right)}+\frac{1}{2}\times \frac{\Gamma^2\left(\frac{3}{4}\right)}{\frac{1}{2}\Gamma\left(\frac{1}{2}\right)}-2\\
&=\frac{1}{4}\times \frac{\Gamma^2\left(\frac{1}{4}\right)}{\Gamma\left(\frac{1}{2}\right)}+\frac{\Gamma^2\left(\frac{3}{4}\right)}{\Gamma\left(\frac{1}{2}\right)}-2\\
\end{align}
It is well known (Euler's reflection formula) that,
\begin{align}\Gamma\left(\frac{1}{2}\right)=\sqrt{\pi}\end{align}
Therefore,
\begin{align}\boxed{I=\frac{\Gamma^2\left(\frac{1}{4}\right)}{4\sqrt{\pi}}+\frac{\Gamma^2\left(\frac{3}{4}\right)}{\sqrt{\pi}}-2}\end{align}
NB:
$\text{B}$ is the Euler beta function.
Best Answer
$$I=\int_{0}^{\pi/3}\ln^4\left(\frac{\sin(x+\pi/3)}{\sin x}\right)dx=\int_0^\frac{\pi}{3}\ln^4 \left(\frac12 +\frac{\sqrt 3}{2}\cot x\right)dx=\frac{\sqrt 3}{2}\int_0^1 \frac{\ln^4 x}{x^2-x+1}dx$$ Above follows by the substitution $\frac12+\frac{\sqrt{3}}{2}\cot x\to x$. We also have for $t\in(0,\pi), x\in (-1,1)$: $$\frac{\sin t}{x^2-2x\cos t+1}=\sum_{n=1}^\infty x^{n-1}\sin(nt)$$ $$\Rightarrow I=\sum_{n=1}^\infty \sin\left(\frac{n\pi}{3} \right)\int_0^1 x^{n-1} \ln^4 x dx=24\sum_{n=1}^\infty \frac{\sin\left(\frac{n\pi}{3} \right)}{n^5}$$
Similarly to here, we have for $x\in(0,2\pi)$: $$\frac{\pi-x}{2}=\sum_{n=1}^\infty\frac{\sin(nx)}{n}$$ Integrating the above with respect to $x$ gives: $$\sum_{n=1}^\infty \frac{\cos(nx)}{n^2}=\frac{(\pi-x)^2}{4}+C_1$$ Setting $x=\pi$ gives $C_1=-\frac{\pi^2}{12}$ and integrating again produces: $$\sum_{n=1}^\infty \frac{\sin(nx)}{n^3}=-\frac{(\pi-x)^3}{12}-\frac{\pi^2}{12}x+C_2$$ Putting $x=\pi $ yields $C_2= \frac{\pi^3}{12}$. One more time:
$$\sum_{n=1}^\infty \frac{\cos(nx)}{n^4}=-\frac{(\pi-x)^4}{48}+\frac{\pi^2}{24}x^2-\frac{\pi^3}{12}x+C_3$$ For $x=\pi \Rightarrow C_3=\frac{23\pi^4}{720}$. And finally, one more similar step gives for $x \in(0,2\pi)$: $$S(x)=\sum_{n=1}^\infty \frac{\sin(nx)}{n^5}=\frac{(\pi-x)^5}{240}+\frac{\pi^2}{72}x^3-\frac{\pi^3}{24}x^2+\frac{23\pi^4}{720}x-\frac{\pi^5}{240}$$
And well, the value of the integral is just $24S\left(\frac{\pi}{3}\right)$. Doing the algebra yields: $$\boxed{\int_{0}^{\pi/3}\ln^4\left(\frac{\sin x}{\sin(x+\pi/3)}\right)dx=\frac{17\pi^5}{243}}$$