Let $\mathcal A$ be the set of all $n\in \mathbb N$ with a prime factor $p>\sqrt{n}$. First, the number of elements $a\in \mathcal A$ with $a\leq x$ is $\gg x$. Indeed, this large prime factor is unique for any $a$, for a given $p\leq x$ there are $\left[\frac{\min(x,p^2)}{p}\right]$ integers $a\leq x$ with $p>\sqrt{a}$ and $p\mid a$. Therefore,
$$
|\mathcal A\cap [1,x]|=\sum_{p\leq x}\left[\frac{\min(x,p^2)}{p}\right]=\sum_{\sqrt{x}<p\leq x}\frac{x}{p}+\sum_{p\leq \sqrt{x}}p+O(\pi(x))=
$$
$$
=x(\ln 2+o(1))\gg x
$$
On the other hand, for any $\sqrt{x}<p\leq x$ all the numbers between $\sqrt{x}$ and $x$, which are divisible by $p$, lie in $\mathcal A$. Therefore,
$$
r_p(x)=\sum_{p\mid n, n\in \mathcal A} 1-\frac{x(\ln 2+o(1))}{p}\geq
$$
$$
\geq \left[\frac{x}{p}\right]-\left[\frac{\sqrt{x}}{p}\right]-\frac{x(\ln 2+o(1))}{p}=
$$
$$
=\frac{x(1-\ln 2)+o(x)}{p}.
$$
Summing over all $\sqrt{x}\leq y<p\leq x$, we get
$$
\sum_{y<p\leq x}|r_p(x)|\geq x(1-\ln 2+o(1))\sum_{y<p\leq x}\frac{1}{p}.
$$
For $y=x^{1-\varepsilon}$ we would get $\gg x$. Also, a trivial upper bound for $r_p(x)$ is $\frac{x}{p}$, so for $y\geq \sqrt{x}$ there is no hope for an upper bound of non-trivial order.
Let $\langle x\rangle$ denote the fractional part of a real number $x$ (i.e. $\langle x \rangle := x- \lfloor x\rfloor $, where $\lfloor x\rfloor $ is the greatest integer less than or equal to $x$).
Let $\alpha \in \mathbb R$ be irrational and let $S:=\{n\in \mathbb Z: \langle n\alpha \rangle \in (0,1/4)\}$. The upper (and lower) density of $S$ is $1/4$; this is a consequence of Weyl's theorem on uniform distribution. Also, $S-S\subseteq \{n\in \mathbb Z: \langle n\alpha\rangle \in (3/4,1)\cup [0,1/4)\}$.
To see that $S-S$ does not contain an infinite arithmetic progression $\{a+bn:n\in \mathbb N\}$, note that $b\alpha$ is irrational if $b\in \mathbb Z\setminus \{0\}$, so the values $\langle (a+bn)\alpha \rangle$ are dense in $[0,1]$. So if $S-S$ contained an infinite AP, the values $\{\langle n\alpha \rangle:n\in S-S\}$ would be dense in $[0,1]$, but $\langle n\alpha\rangle \in (3/4,1)\cup [0,1/4)$ for $n\in S-S$.
This example $S$ is a Bohr neighborhood in $\mathbb Z$. Generally, if you want an example or counterexample of some structure in $S-S$, where $S$ has positive upper density, it's natural to look among Bohr neighborhoods: Følner ZBL0058.02302 proved that if $S$ has positive upper Banach density, then $S-S$ contains (up to upper Banach density 0) a Bohr neighborhood of $0$. Since every Bohr neighborhood $B$ of $0$ contains a set of the form $B'-B'$, where $B'$ is a Bohr neighborhood, $S-S$ itself is not too far from containing a difference set of a Bohr neighborhood.
Ruzsa's section Sumsets and structure in ZBL1221.11026 and Hegyvári and Ruzsa's article ZBL1333.05042 are both good references on the relationship between Bohr sets and difference sets.
Best Answer
Another construction is to let $n \notin V$ if and only if $n$ begins with at least $\sqrt{\log{n}}$ consecutive '9's when written in decimal. This satisfies the stronger property that there is no non-constant polynomial $f : \mathbb{N} \rightarrow \mathbb{N}$ whose image is contained entirely in $V$.