The second example you give does not work as written. Rather, define $f$ by saying that $f(x)=0$ unless $x=1/n$ for some $n$, in which case $f(x)=f(1/n)=1/n$. (I suspect this is actually what you meant to write.) This function is discontinuous at each $1/n$, having a jump discontinuity there. It is continuous at every other point: This is clear if the point is not $0$, since then there is a neighborhood around the point where $f$ is constantly zero. But $f$ is also continuous at $0$, since $f(1/n)=1/n\to0$. Formally: Given $\epsilon>0$, let $N$ be such that $1/N<\epsilon$. If $0<|x|<1/N$, then either $x$ is not a $1/n$, and $f(x)=0$, or $x=1/n$ for some $n$, which necessarily is strictly larger than $N$, so $f(x)=1/n<1/N<\epsilon$. In either case, $|f(x)-f(0)|<\epsilon$, so the definition of continuity at zero is satisfied.
Let's examine your first example. First of all, your function is discontinuous at each point $x$ of $[0,1]$ (including $0$ and $1$), since arbitrarily close to $x$ there are points $t$ with $f(t)=20$ and points $s$ with $f(s)=10$, so no matter whether $f(x)=10$ or $f(x)=20$, $f$ is discontinuous at $x$. Whether the example works, however, depends the precise meaning of "$f$ is continuous on the complement of $A$". If by this you mean that the restriction of $f$ to the complement of $A$ is continuous, then yes, the function you suggest is continuous there, since it is constant. On the other hand, the common meaning of the term is that $f$ is continuous at $x$ for any $x\notin A$. Under this common meaning, $f$ is continuous on $\mathbb R\setminus[0,1]$, since for each $x$ outside of $[0,1]$, there is a whole neighborhood where $ f$ is constant. On the other hand, $f$ is discontinuous at $0$ and at $1$, and neither point is in $A=(0,1)$.
The example cannot be fixed by changing the values of the constants. We actually need to change the definition a bit. One suggestion is to have $f(x)=10$ for $x$ outside of $(0,1)$, or inside $(0,1)$ and rational, and to define $f(x)$ for $x$ inside $(0,1)$ and irrational in a way that $f(x)$ approaches $10$ both at $0$ and $1$, and yet is discontinuous on $(0,1)$. I suggest you let $f(x)=20x+10$ for $x$ irrational and $0<x<1/2$, and $f(x)=30-20x$ for $x$ irrational and $1/2<x<1$. This is similar to your suggestion, but rather than having the function constant on the irrationals, I picked a value on the irrationals that is away from $10$ (so we still have discontinuity at every point of $(0,1)$, but approaches $10$ at both end-points (thus ensuring continuity at both $0$ and $1$).
This sometimes called the Popcorn Function, since if you look at a picture of the graph, it looks like kernels of popcorn popping. (Also called the Thomae's Function.)
To prove it is discontinuous at any rational point, you could argue in the following way: Since the irrationals that are in $(0,1)$ are dense in $(0,1)$, given any rational number $p/q$ in $(0,1)$, we know $f(p/q) = 1/q$. But let $a_{n}$ be a sequence of irrationals converging to $p/q$ (by the definition of density). Then $\lim \limits_{a_{n} \to (p/q)} f(x) = 0 \neq f(p/q)$. Thus, $f$ fails to be continuous at $x = p/q$.
Now, to prove it is continuous at every irrational point, I recommend you do it in the following way:
Prove that for each irrational number $x \in (0,1)$, given $N \in \Bbb N$, we can find $\delta_{N} > 0$ so that the rational numbers in $(x - \delta_{N}, x + \delta_{N})$ all have denominator larger than $N$.
Once you have the result from above, we can use the $\epsilon-\delta$ definition of continuity to prove $f$ is continuous at each irrational point.
So, first prove 1. (It's not too hard.) Once you've done that, you can accomplish 2. in the following way:
Let $\epsilon > 0$. Let $x \in (0,1)$ be irrational. Choose $N$ so that $\frac{1}{n} \leq \epsilon$ for every $n \geq N$ (by the archimedian property).
By what you proved in 1., find $\delta_{N} > 0$ so that $(x - \delta_{N}, x + \delta_{N})$ contains rational numbers only with denominators larger than $N$.
Then if $y \in (x-\delta_{N}, x + \delta_{N})$, $y$ could either be rational or irrational. If $y$ is irrational, we have $|f(x) - f(y)| = |0 - 0| = 0 < \epsilon$ (since $f(z) = 0$ if $z$ is irrational).
On the other hand, if $y$ is rational, we have $y = p/q$ with $q \geq N$. So $|f(x) - f(y)| = |0 - f(y)| = |f(y)| = |1/q| \leq |1/N| < \epsilon$, and this is true for every $y \in (x-\delta_{N}, x + \delta_{N})$.
Thus, given any $\epsilon > 0$, if $x \in (0,1)$ is irrational, we found $\delta > 0$ (which was actually $\delta_{N}$ in the proof) so that $|x - y| < \delta$ implies $|f(x) - f(y)| < \epsilon$.
Best Answer
This function is known as Thomae's function and is an excellent example of a function that is continuous at uncountably infinite number of points while discontinuous at countably infinite number of points. Note that the reverse result is not possible, i.e. one cannot have a function that is discontinuous on an uncountably infinite dense set and continuous on a countably infinite dense set.
As for the proof , the discontinuity part follows from the fact that irrationals are dense in $ R $ as you rightly thought. The continuity part is a little bit more involved and it would help to employ the Archimedean principle in tandem with the $ \epsilon$-$\delta $ definition.
Crudely, look at $ x=\sqrt3 $ , see that using the non-terminating decimal expansion i.e. $ \sqrt 3=1.732148\ldots $ , one can forge a sequence of rationals $ \ \ 1 , 17/10 ,173/100 ,\ldots \ $ that converges to $ x $, while at the same time your $ f(x_n) $ is $ \ \ 1/10,1/100,1/1000,\ldots \ $ which converges to $ 0 $ which is nothing but $ f(x) $. This is merely an illustration of the continuity and not a rigorous proof.