Begin a problem:
Prove that:
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{12}\right\rfloor = \left\lfloor\dfrac{(n^2-10)^2}{144}\right\rfloor$$
I try replace denominator by $m$, with the help a computer, i found a few identities
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{2}\right\rfloor = \left\lfloor\dfrac{(n^2-2)^2}{24}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{4}\right\rfloor = \left\lfloor\dfrac{(n^2-2)^2}{48}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{6}\right\rfloor = \left\lfloor\dfrac{(n^2-7)^2}{72}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{8}\right\rfloor = \left\lfloor\dfrac{(n^2-5)^2}{96}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{12}\right\rfloor = \left\lfloor\dfrac{(n^2-10)^2}{144}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{16}\right\rfloor = \left\lfloor\dfrac{(n^2-11)^2}{192}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{3}\right\rfloor = \left\lfloor\dfrac{(n^2-2)(n^2-3)}{36}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{5}\right\rfloor = \left\lfloor\dfrac{(n^2-6)(n^2-7)}{60}\right\rfloor$$
$$\sum_{k=1}^n\sum_{j=1}^k\left\lfloor\dfrac{(j-1)^2}{7}\right\rfloor = \left\lfloor\dfrac{(n^2-6)(n^2-7)}{84}\right\rfloor$$
…
$m=9,10,11,13,14,15$ that form is impossible!
There is something very mysterious that "prevents" the construction of a general formula!???
Best Answer
Exchanging the order of summation we get (notice the summand doesn't depend on $k$)
$$ S(n,m) = \sum_{k=1}^n \sum_{j=1}^k \left\lfloor \frac{(j-1)^2}{m} \right\rfloor = \sum_{j=1}^n \sum_{k=j}^n \left\lfloor \frac{(j-1)^2}{m} \right\rfloor = \sum_{j=0}^n (n-j) \left\lfloor \frac{j^2}{m} \right\rfloor $$
(We also changed the summation index $j \to j+1$ and then notice we can include the boundary $j=n$ because the summand is $0$ there)
UPDATED answer
Let's use $\lfloor y \rfloor = y - \{y\}$ where $\{y\}$ is the fractional part of $y$. We have
$$ S(n, m) = \sum_{j=0}^n (n-j)\left(\frac{j^2}{m} - \left\{\frac{j^2}{m} \right\} \right) = \frac{n^2(n^2-1)}{12m} - \sum_{j=0}^n (n-j)\left\{\frac{j^2}{m} \right\} $$
Let's study the sum $h(n) = \sum_{j=0}^n (n-j)\left\{\frac{j^2}{m} \right\}$. It is infact the convolution of the sequences $f(n)=n$ and $g(n)=\left\{\frac{n^2}{m} \right\}$.
The generating function of $h$ is the product of those of $f$ and $g$. Denote the generating functions with corresponding capital letters. We know $F(x)=\frac{x}{(1-x)^2}$. Let's calculate $G(x)$.
$$ G(x)=\sum_{n=0}^\infty g(n)x^n = \sum_{k=0}^\infty \sum_{r=0}^{m-1} g(km+r)x^{km+r} = \sum_{k=0}^\infty x^{km} \sum_{r=0}^{m-1} \left \{ \frac{r^2}{m} \right\} x^r = \frac{\sum_{r=0}^{m-1} \left \{ \frac{r^2}{m} \right\} x^r}{1-x^m} = \frac{\sum_{r=0}^{m-1} (r^2 \mod m) x^r}{m(1-x^m)} $$
So
$$ H(x) = \frac{\sum_{r=0}^{m-1} (r^2 \mod m) x^{r+1}}{m(1-x)^2(1-x^m)} $$
We can decompose $H$ into partial fractions. Notice that $1-x^m$ has roots $\zeta^j$ for $j=0,1,\dots, m-1$ where $\zeta = e^{2\pi i /m}$. We get
$$ H(x) = \frac{1}{m(1-x)^3} + \frac{m-1}{2m(1-x)^2} + \frac{m^2-1}{12m(1-x)} + \sum_{j=1}^{m-1} \frac{\zeta^j}{(1-\zeta^j)^2 (\zeta^j-x)} $$
So extracting the coefficient from these (use binomial theorem) and after simplifying we get
$$ h(n) = \frac{1}{m^2} \cdot \sum_{r=0}^{\min(m,n)-1} (r^2 \bmod m) \left[ \frac{n^2}{2} + \left(\frac{m}{2}-r \right)n + \frac{r(r-m)}{2} + \frac{m^2-1}{12} + \sum_{j=1}^{m-1} \frac{1}{(1-\zeta^j)^2 \zeta^{j(n-r-1)}} \right] $$
It's interesting that the inner sum involving the roots of unity seems to always be a rational number. That would need some further investigation. And one further hypothesis is that the coefficient of $n^1$ in the final formula will be $0$. Maybe that follows since the sum has quadratic residues as coefficients, there are $\frac{m}{2}$ of them and we have the coeffient $\frac{m}{2} - r$.