Let $S=\{(x,y)\in\Bbb R^2\mid x\in\Bbb Q,0<x<1,\text{ and if } x=\frac{p}m,\gcd(p,m)=1,y=\frac{k}m,k=1,\ldots,m-1\}.\tag 1$
Prove that $\displaystyle \int_0^1\int_0^1\chi_S dxdy=0$ while $\displaystyle \int_S\chi_S$ doesn't exist.
My thoughts:
I tried to write down $(1)$ differently, so I fixed some $y\in\Bbb Q.$ Then $y_0=\frac{p_0}{q_0},\gcd(p_0,q_0)=1.$ Then, for every $n\in\Bbb N$ all the points of the form $$\left(y_0,\frac{p}{q_0n}\right),\gcd(p,q_0n)=1,p<q_0n\tag 2$$ are in $S.$ I tried to show that the set of discontinuities of the function $$x\mapsto\begin{cases}1, &(x,y_0)\in S\\0,&(x,y_0)\notin S\end{cases}$$ is of measure zero and exactly the set $S$ and that the function $$y\mapsto\int_0^1\chi_Sdx$$ is non-zero on the set of the Jordan measure $0$. I thought I could prove that, for a given $y_0,$ the points in $(2)$ might form a discrete set,which would have a boundary of the Jordan measure zero, but I failed.
For the second integral, I thought one might argue that, even though, for some $x_0\in\Bbb Q\cap(0,1)$ there are only finitely many points $(x_0,y)\in S,$ since $\Bbb Q\cap(0,1)$ is dense in $[0,1],$if we take a good enough subdivision of $[0,1]\times[0,1],$ letting $m\to\infty,$ there are infinitely many points from $S$ in a rectangle of the subdivision so the lower Darboux sum would equal $0,$ while the upper would be $1$ and therefore, $\displaystyle\int_{[0,1]^2}\chi_S$ wouldn't exist, and neither would the given integral.
I'm not sure if I'm on the right track at all and what I've written seems messy even to me. Does anybody have any advice/hint on what to do?
Best Answer
There is a typo in the question where the double integral should be $dy\;dx$, not $dx\; dy$.
$\int_0^1\chi_S(x,y_0)dx$ does not exist for $y_0\in(0,1)\cap\mathbb{Q}$, because in such a case $S(x,y_0)=1$ if and only if $x$ is rational and the denominator of $x$ in lowest terms is a multiple of the denominator of $y_0$ in lowest terms.
Therefore for $y_0\in(0,1)\cap\mathbb{Q}$, the sets $\{x\in(0,1):\chi_S(x,y_0)=1\}$ and $\{x\in(0,1):\chi_S(x,y_0)=0\}$ are both dense in $(0,1)$.
$\\$
Elaboration, on OP's request, of why the first set is dense:
Let's say $y_0=\frac{n}{m}\in(0,1)\cap\mathbb{Q}$, where $\text{gcd}(n,m)=1$.
For any small interval $I\subset(0,1)$, choose $k$ large enough so that when you look at the points $\frac{j}{m^k}$ for $1\leq j\leq m^k-1$, at least $m$ of these points are in $I$. So the interval $I$ contains $\frac{j}{m^k},\frac{j+1}{m^k},\ldots,\frac{j+m-1}{m^k}$, where at least one of $j,j+1,\ldots,j+m-1$ is $1\,\text{mod}(m)$, which we could call $j'$. Now $\frac{j'}{m^k}$ is in its simplest form, and $m^k$ is a multiple of $m$, so $\frac{j'}{m^k}$ is in $\{x\in(0,1):\chi_S(x,y_0)=1\}$