You've made great progress in deriving the Euler-Lagrange equation for the functional $S[y] = \int_{a}^{b} f(x) \sqrt{1 + y'^2} \, dx$ with the given boundary conditions $y(a) = A$ and $y(b) = B$. Let's continue from where you left off.
From the Euler-Lagrange equation, you have:
$$\frac{d}{dx} \left( \frac{f(x) y'}{\sqrt{1 + y'^2}} \right) = 0$$
This means that the quantity $\frac{f(x) y'}{\sqrt{1 + y'^2}}$ is constant along the stationary path. Let's denote this constant as $\beta$. So, we have:
$$\frac{f(x) y'}{\sqrt{1 + y'^2}} = \beta$$
Now, let's solve for $y'$. Rearranging the equation, we get:
$$y' = \beta \frac{\sqrt{1 + y'^2}}{f(x)}$$
Square both sides to eliminate the square root:
$$y'^2 = \beta^2 \frac{1 + y'^2}{f(x)^2}$$
Rearrange and solve for $y'^2$:
$$y'^2 = \frac{\beta^2}{f(x)^2 - \beta^2}$$
Taking the square root (and considering the positive root since $f(x) > 0$), we get:
$$y' = \frac{\beta}{\sqrt{f(x)^2 - \beta^2}}$$
Now, integrate this expression to find $y(x)$. Remember, $y(a) = A$, so we will integrate from $a$ to $x$:
$$y(x) = A + \int_{a}^{x} \frac{\beta}{\sqrt{f(w)^2 - \beta^2}} \, dw$$
The value of $\beta$ must be determined from the boundary condition $y(b) = B$. Substituting $x = b$ in the above equation, we get:
$$B = A + \int_{a}^{b} \frac{\beta}{\sqrt{f(w)^2 - \beta^2}} \, dw$$
Rearranging, we find:
$$B - A = \beta \int_{a}^{b} \frac{1}{\sqrt{f(w)^2 - \beta^2}} \, dw$$
This final equation gives us the condition that $\beta$ must satisfy. Thus, the stationary path $y(x)$ is indeed:
$$y(x) = A + \beta \int_{a}^{x} \frac{1}{\sqrt{f(w)^2 - \beta^2}} \, dw$$
with $\beta$ satisfying
$$B - A = \beta \int_{a}^{b} \frac{1}{\sqrt{f(w)^2 - \beta^2}} \, dw$$
This completes the derivation.
Best Answer
Given an explicit quantity, it is easy to verify whether or not it is conserved (as a function of time $x$), cf. Gonçalo's answer.
If we don't already know the explicit form of the conserved quantity/first integral, we can find it using Noether's theorem. This is what we will do in this answer.
It is easy to see that OP's functional $$ S[y;a,b] ~=~S[\tilde{y};\tilde{a},\tilde{b}]$$ has an exact symmetry $$\tilde{y}(\tilde{x})~=~\lambda^{\color{red}{2}}y(x), \qquad \tilde{x}~=~\lambda^{\color{red}{-1}}x, $$ where $\lambda> 0$ is a positive parameter.
Since OP's Lagrangian is $$L~=~x^5(\dot{y}^2-\frac{2}{3}y^3),$$ the corresponding momentum and energy are $$ p~:=~\frac{\partial L}{\partial \dot{q}}~=~2x^5\dot{y}$$ and $$h~:=~p\dot{y}-L~=~x^5(\dot{y}^2+\frac{2}{3}y^3),$$ respectively. The corresponding Noether charge $$ Q~=~\color{red}{2y}p - \color{red}{(-x)}h~=~4x^5y\dot{y}+x^6(\dot{y}^2+\frac{2}{3}y^3)$$ is OP's sought-for conserved quantity/first integral.
For more details, see e.g. this related Phys.SE post.