If there is a mathematical body of knowledge that allows you to manipulate formulas (13) and (14) into a better form, then great! The Riemann zeta function has already proved its worth in this field. It was instrumental in the proofs by Hadamard and de la Vallée-Poussin of the Prime Number Theorem. What allowed that to happen was that the zeta function is a meromorphic function, and so complex variable theory could be applied to its study. One never knows with certainty whether further study of the zeta function will pay off, but people are betting that it will, based on past successes.
I don't know whether anyone is promoting (13) and (14) as useful tools in the study of prime numbers, but if so, the burden is on those people to exhibit mathematical tools that allow one to manipulate (13) and (14) effectively.
Because of all the floor functions, (13) and (14) don't have very nice analytical properties. As other posters have pointed out, they encode an elementary algorithm for identifying primes. In particular, the expression
$$
\left\lfloor\frac{j}{s}\right\rfloor-\left\lfloor\frac{j-1}{s}\right\rfloor
$$
that appears in the numerator of (14) equals $1$ if $s$ evenly divides $j$ and equals $0$ if it does not. Summing this quantity for $s$ from $1$ to $j$ gives you the number of divisors of $j.$ Prime numbers have exactly two divisors; composite numbers all have at least three. The numerator in (14) is therefore $0$ when $j$ is prime, and positive when it is composite. Dividing by $j$ and multiplying by $-1$ results in the quantity $s(j),$ which is $0$ when $j$ is prime and lies in the interval $(-1,0)$ when $j$ is composite. The floor of $s(j$) therefore equals $0$ when $j$ is prime and $-1$ when $j$ is composite. Adding $1$ to $\lfloor s(j)\rfloor$ gives the characteristic function of the primes.
Notice that using the formula (14) requires one to do more divisions than even a naive brute-force algorithm to compute the number of divisors would require. That price might be worth paying if the formula had nice mathematical properties that allowed one to extract additional information from it, but I don't see that this has been demonstrated. To me, it looks like a cumbersome arithmetic encoding of an inefficient algorithm. I would be happy to be proved wrong about this!
Formula (13) likewise looks like a cumbersome arithmetic encoding of a brute-force method for computing the $n^\text{th}$ prime, and seems likely to require more calculation than state-of-the-art algorithms. Again, it would be nice if someone could show how to extract useful information from the formula, but it doesn't look very promising.
Added: I think it worth pointing out that some non-trivial analytic number theory enters into formula (13) in the choice of upper bound on the summation over $k.$ You need to ensure that the $p_n$ lies in the range of summation, but that $p_{2n}$ does not. The reason is that the summand equals $1$ when $k<p_n,$ equals $0$ for $p_n\le k<p_{2n},$ and becomes negative for $k\ge p_{2n}.$ So you need the summation over $k$ to include all $p_n-1$ terms where the summand equals $1$, but none of the terms where it is negative. This will ensure that the sum evaluates to $p_n-1.$
It appears to me that you need something like (3.12) and (3.13) in this paper of Rosser and Schoenfeld to justify the upper bound. Formula (3.12) is Rosser's Theorem which Dietrich Burde's answer to this post suggests requires deep knowledge of zeta function zeros. Interestingly, the author of your formulas (13) and (14) (Sebastián Martín Ruiz) does not rigorously justify the upper bound in his article, giving only a heuristic argument and and stating that it is “very probable” that the necessary inequalities are satisfied.
Perhaps one could play with the constant in front of $\lfloor n\log n\rfloor$ and try to use Chebyshev's bounds (Theorem 3.1 here) instead of Rosser's Theorem. (The derivation of Chebyshev's bounds doesn't require knowledge of zeta function zeros.) But this seems possibly rather delicate, and may not work.
Second addition: Some discussion of the formula, and Ruiz's view of it, can be found here.
Best Answer
OK, so here is a solution to (i) in case we know that when a page is torn out and odd and the following even page number is removed.
Given the book has $n$ pages the sum of the page numbers will be $$ T_n=1+2+...+n=\frac{n(n+1)}{2} $$ known as the $n$'th Triangular Number. So assume that page numbers $x$ and $x+1$ have been torn out where $x$ is odd. Then we can write $$ \frac{105}{4}=\tfrac{1}{n-2}\left(T_n-(2x+1)\right) $$ Plugging in the formula for $T_n$ from above and solving for $x$ then yields $$ x=\frac{1}{4}n^2-\frac{103}{8}n+\frac{103}{4} $$ which is a quadratic expression in $n$ with zeros $n=\frac{103\pm\sqrt{8961}}{4}$ which approximately is $n=2.08$ and $n=49.42$. On the other hand it is obvious that $x<n$ which then in turn yields a quadratic inequality in $n$ that can be solved to see that $n<\frac{111+\sqrt{10673}}{4}\approx 53.58$.
Having a slightly closer look at the expression for $x$ one realizes that $n-2$ must be divisible by $8$ for $x$ to be an integer. So unless we take $n=2$ (which actually works) we must have $49.42<n=50<53.58$ for $n$ to be an integer satisfying all requirements in that interval. Plugging $n=50$ into the expression for $x$ then yields $x=7,x+1=8$, and you can check as I did in the comments that the average of the remaining pages is really $\frac{105}{4}$. With $n=50$ one gets $$ \sum_{i=1}^{10}\left\lfloor\frac{50+r}{r+1}\right\rfloor=105 $$ So option (C) answers the first question correctly.
Part (ii)
Still using $n=50$ we get $\lfloor\frac{50}{3}\rfloor=16$ so that the line is $x+y=16$ or $y=16-x$. Together with the axes $y=0$ and $x=0$ this encloses a triangle with $17$ lattice points on the line $y=16-x$ since you can start from $(0,16)$ and move down right step-by-step to 'visit' 17 lattice points before you hit the x-axis. Similarly you hit $16$ lattice points following the same procedure from $(0,15)$. With this we get the answer to (ii) which is: $$ T_{17}=17+16+...+1=\frac{17\cdot 18}{2}=153 $$ So option (B) answers the second question correctly.