The sum $S(p)$ counts the lattice points with positive coordinates under $y=\sqrt{px}$ from $x=1$ to $x=\frac{p-1}{4}$. Instead of counting the points below the parabola, we can count the lattice points on the parabola and above the parabola, and subtract these from the total number of lattice points in a box. Stop here if you only want a hint.
Since $p$ is prime, there are no lattice points on that parabola (with that range of $x$ values).
The total number of lattice points in the box $1 \le x \le \frac{p-1}4, 1\le y \le \frac {p-1}2$ is $\frac{(p-1)^2}8$.
The lattice points above the parabola are to the left of the parabola. These are counted by
$T(p)= \lfloor 1^2/p \rfloor + \lfloor 2^2/p \rfloor + ... + \lfloor (\frac{p-1}2)^2/p \rfloor$.
$T(p)+S(p) = \frac{(p-1)^2}8$, so $S = \frac {p^2-1}{12}$ is equivalent to $T(p) = \frac{(p-1)(p-5)}{24}$.
Consider $T(p)$ without the floor function. This sum is elementary:
$$\sum_{i=1}^{(p-1)/2} \frac{i^2}p = \frac 1p \sum_{i=1}^{(p-1)/2} i^2 = \frac 1p \frac 16 (\frac{p-1}2)(\frac {p-1}2 + 1)(2\frac{p-1}2 +1) = (p^2-1)/24.$$
What is the difference between these? Abusing the mod notation, $\frac{i^2}p - \lfloor \frac{i^2}p \rfloor = 1/p \times (i^2 \mod p)$. So,
$$(p^2-1)/24 - T(p) = \sum_{i=1}^{(p-1)/2} \frac{i^2}p - \lfloor \frac{i^2}p \rfloor = \sum_{i=1}^{(p-1)/2} \frac 1p \times (i^2 \mod p) = \frac 1p \sum_{i=1}^{(p-1)/2} (i^2 \mod p).$$
Since $i^2 = (-i)^2$, this last sum is over the nonzero quadratic residues. Since $p$ is $1 \mod 4$, $-1$ is a quadratic residue, so if $a$ is a nonzero quadratic residue, then so is $p-a$. Thus, the nonzero quadratic residues have average value $p/2$ and the sum is $\frac{(p-1)}2 \frac p2$.
$$(p^2-1)/24 - T(p) = \frac 1p \frac{(p-1)}2 \frac p2 = \frac{p-1}4$$
$$T(p) = \frac{(p-1)(p-5)}{24}.$$
That was what we needed to show.
Assume there's a non-integral $x$ which satisfies the stated criteria. For $n = 1$, for some integer $m \ge 1$, we'd have
$$\lfloor x\rfloor = m^2 \;\;\to\;\; x = m^2 + r, \;\; 0 \lt r \lt 1 \tag{1}\label{eq1A}$$
With $n = 2$, we get
$$(m^2)^2 \lt x^2 = (m^2 + r)^2 \lt (m^2 + 1)^2 \tag{2}\label{eq2A}$$
Thus, for $\lfloor x^2\rfloor$ to be a perfect square requires that
$$\lfloor x^2\rfloor = (m^2)^2 \;\;\to\;\; (m^2+r)^2 \lt (m^2)^2 + 1 \tag{3}\label{eq3A}$$
Have $n$ double each time, so $n = 2^k$. Using the first $2$ terms of the binomial theorem expansion, and with large enough $k$, we have
$$\begin{equation}\begin{aligned}
& (m^2+r)^{2^{k}} - (m^{2^k}+1)^2 \\
& \gt (m^2)^{2^k} + 2^{k}(m^2)^{2^k-1}r - (m^{2^k})^2 - 2(m^{2^k}) - 1 \\
& = m^{2^{k+1}} + 2^{k}rm^{2^{k+1}-2} - m^{2^{k+1}} - 2m^{2^k} - 1 \\
& = 2m^{2^k}\left(2^{k-1}rm^{2^{k}-2}-1\right) - 1 \\
& \gt 0
\end{aligned}\end{equation}\tag{4}\label{eq4A}$$
This shows $\left\lfloor x^{2^k}\right\rfloor \ge (m^{2^k}+1)^2$ for all sufficiently large $k$. Note that, for any $k$, we have $x^{2^k}\gt \left(m^{2^{k}}\right)^2$, and $\left\lfloor x^{2^k}\right\rfloor$ can't be any value between $\left(m^{2^k}\right)^{2}+1$ and $\left(m^{2^k}+1\right)^{2}-1$, inclusive. Thus, for some $k$, there must be a switch between the floor value being $(m^{2^k})^2$, and then for $k+1$ it being at least $(m^{2^{k+1}}+1)^2$, i.e., that
$$(m^2+r)^{2^{k}} \lt (m^{2^k})^2+1, \;\; (m^2+r)^{2^{k+1}} \ge \left((m^{2^k})^2+1\right)^2 \tag{5}\label{eq5A}$$
However, squaring the left side above gives $(m^2+r)^{2^{k+1}} \lt \left((m^{2^k})^2+1\right)^2$, contradicting the right side above. This shows the original assumption that there's such a non-integral $x$ must be false.
Best Answer
Given an integer $n$, by long division there are uniquely determined integers $q_n$ and $r_n$ with $0\leq r_n<n$ such that $$2^n=q_n\cdot n+r_n.$$ Then $\lfloor\tfrac{2^n}{n}\rfloor=q_n$.
First, if $r_n=0$ then $q_n$ divides $2^n$ and so $q_n$ is a power of $2$. In this case $n$ also divides $2^n$, so $n$ is also a power of $2$. Conversely, if $n$ is a power of $2$, say $n=2^k$, then clearly $$\lfloor\frac{2^n}{n}\rfloor=2^{2^k-k}$$ is a power of $2$.
Next, if $r_n>0$ and $q_n$ is a power of $2$, say $q_n=2^m$ for some nonnegative integer $m$, then $$2^n=2^m\cdot n+r_n.$$ Clearly $2^m\leq2^n$ and so $2^m$ also divides $r_n$. Then from $r_n<n$ it follows that $2^m<n$ and hence $$2^n=2^m\cdot n+r_n<n^2+n.$$ Of course the left hand side grows faster than the right hand side, so this is only possible for small values of $n$; already for $n=5$ we see that $2^5>5^2+5$, so $n\leq4$. The values $n=1,2,4$ are powers of $2$ and so we just need to check for $n=3$ that $$\lfloor\frac{2^3}{3}\rfloor=\lfloor\frac{8}{3}\rfloor=2,$$ is also a power of $2$.