The sum $S(p)$ counts the lattice points with positive coordinates under $y=\sqrt{px}$ from $x=1$ to $x=\frac{p-1}{4}$. Instead of counting the points below the parabola, we can count the lattice points on the parabola and above the parabola, and subtract these from the total number of lattice points in a box. Stop here if you only want a hint.
Since $p$ is prime, there are no lattice points on that parabola (with that range of $x$ values).
The total number of lattice points in the box $1 \le x \le \frac{p-1}4, 1\le y \le \frac {p-1}2$ is $\frac{(p-1)^2}8$.
The lattice points above the parabola are to the left of the parabola. These are counted by
$T(p)= \lfloor 1^2/p \rfloor + \lfloor 2^2/p \rfloor + ... + \lfloor (\frac{p-1}2)^2/p \rfloor$.
$T(p)+S(p) = \frac{(p-1)^2}8$, so $S = \frac {p^2-1}{12}$ is equivalent to $T(p) = \frac{(p-1)(p-5)}{24}$.
Consider $T(p)$ without the floor function. This sum is elementary:
$$\sum_{i=1}^{(p-1)/2} \frac{i^2}p = \frac 1p \sum_{i=1}^{(p-1)/2} i^2 = \frac 1p \frac 16 (\frac{p-1}2)(\frac {p-1}2 + 1)(2\frac{p-1}2 +1) = (p^2-1)/24.$$
What is the difference between these? Abusing the mod notation, $\frac{i^2}p - \lfloor \frac{i^2}p \rfloor = 1/p \times (i^2 \mod p)$. So,
$$(p^2-1)/24 - T(p) = \sum_{i=1}^{(p-1)/2} \frac{i^2}p - \lfloor \frac{i^2}p \rfloor = \sum_{i=1}^{(p-1)/2} \frac 1p \times (i^2 \mod p) = \frac 1p \sum_{i=1}^{(p-1)/2} (i^2 \mod p).$$
Since $i^2 = (-i)^2$, this last sum is over the nonzero quadratic residues. Since $p$ is $1 \mod 4$, $-1$ is a quadratic residue, so if $a$ is a nonzero quadratic residue, then so is $p-a$. Thus, the nonzero quadratic residues have average value $p/2$ and the sum is $\frac{(p-1)}2 \frac p2$.
$$(p^2-1)/24 - T(p) = \frac 1p \frac{(p-1)}2 \frac p2 = \frac{p-1}4$$
$$T(p) = \frac{(p-1)(p-5)}{24}.$$
That was what we needed to show.
Best Answer
Letting $\sqrt{n} + \epsilon = \sqrt{n+1}$ for some $\epsilon > 0$, squaring gives $n + 2\epsilon \sqrt{n} + \epsilon^{2} = n + 1$ therefore the equation $\epsilon^{2} + 2\epsilon \sqrt{n} - 1 = 0$ has solution $\epsilon = \sqrt{n} - \sqrt{n - 1}$
For $n > 1$ this difference will always be less than 1, thus you arrive at the result that $\lfloor \sqrt{n} + \sqrt{n+1}\rfloor = 2k-1$ when $n+1 = k^2$