The second example you give does not work as written. Rather, define $f$ by saying that $f(x)=0$ unless $x=1/n$ for some $n$, in which case $f(x)=f(1/n)=1/n$. (I suspect this is actually what you meant to write.) This function is discontinuous at each $1/n$, having a jump discontinuity there. It is continuous at every other point: This is clear if the point is not $0$, since then there is a neighborhood around the point where $f$ is constantly zero. But $f$ is also continuous at $0$, since $f(1/n)=1/n\to0$. Formally: Given $\epsilon>0$, let $N$ be such that $1/N<\epsilon$. If $0<|x|<1/N$, then either $x$ is not a $1/n$, and $f(x)=0$, or $x=1/n$ for some $n$, which necessarily is strictly larger than $N$, so $f(x)=1/n<1/N<\epsilon$. In either case, $|f(x)-f(0)|<\epsilon$, so the definition of continuity at zero is satisfied.
Let's examine your first example. First of all, your function is discontinuous at each point $x$ of $[0,1]$ (including $0$ and $1$), since arbitrarily close to $x$ there are points $t$ with $f(t)=20$ and points $s$ with $f(s)=10$, so no matter whether $f(x)=10$ or $f(x)=20$, $f$ is discontinuous at $x$. Whether the example works, however, depends the precise meaning of "$f$ is continuous on the complement of $A$". If by this you mean that the restriction of $f$ to the complement of $A$ is continuous, then yes, the function you suggest is continuous there, since it is constant. On the other hand, the common meaning of the term is that $f$ is continuous at $x$ for any $x\notin A$. Under this common meaning, $f$ is continuous on $\mathbb R\setminus[0,1]$, since for each $x$ outside of $[0,1]$, there is a whole neighborhood where $ f$ is constant. On the other hand, $f$ is discontinuous at $0$ and at $1$, and neither point is in $A=(0,1)$.
The example cannot be fixed by changing the values of the constants. We actually need to change the definition a bit. One suggestion is to have $f(x)=10$ for $x$ outside of $(0,1)$, or inside $(0,1)$ and rational, and to define $f(x)$ for $x$ inside $(0,1)$ and irrational in a way that $f(x)$ approaches $10$ both at $0$ and $1$, and yet is discontinuous on $(0,1)$. I suggest you let $f(x)=20x+10$ for $x$ irrational and $0<x<1/2$, and $f(x)=30-20x$ for $x$ irrational and $1/2<x<1$. This is similar to your suggestion, but rather than having the function constant on the irrationals, I picked a value on the irrationals that is away from $10$ (so we still have discontinuity at every point of $(0,1)$, but approaches $10$ at both end-points (thus ensuring continuity at both $0$ and $1$).
You need to show that for any $\epsilon > 0$ and any irrational $a$, there is a neighborhood $I$ of $a$ in which $|f(x)| < \epsilon$. Clearly the troublesome points here are the rational points of the neighborhood $I$. Let $n$ be a positive integer such that $n > 1/\epsilon$. Now consider any rational number of the form $p/n!$. It includes all the rationals whose denominator is from set $\{1, 2, \ldots, n\}$ (and many other rationals too). Now we can choose positive integer $p$ such that $p < n!a < p + 1$. Note that this is possible because $a$ is irrational (if $a$ were rational then one of these inequalities might become an equality). It follows that $$\frac{p}{n!} < a < \frac{p + 1}{n!}$$ Let $I$ be the interval $(p/n!, (p + 1)/n!)$. It should now be obvious that any rational number contained in $I$ must have its denominator greater than $n$ and hence at these rational points the value of $f(x)$ is $1/n < \epsilon$. So we get $|f(x)| < \epsilon$ as desired. Thus $f$ is continuous at $a$.
Best Answer
Take some rational number $r\in\mathbb Q\cap (0,1)$. There exists a representation $r=\frac{p}{q}$ such that $gcd(p,q)=1$. So, there holds $f(r)=\frac1{q}>0$. Now, you can use 1. to get a sequence $(x_n)_n$ in $(\mathbb R\setminus \mathbb Q)\cap (0,1)$ such that $x_n\to r$. But $x_n$ are all irrational, hence $f(x_n)=0$ for all $n$!. You get $$ f(x_n)=0\to 0$$ But $f(r)\neq 0$. You have got a contradition to the continuity at $r$.
Yes you can.
No, here you have to be careful since the situation changed! If you approximate the irrational number by a sequence of rational numbers, you have to to a bit more work to see what the values of the sequence are.
Let us pick an irrational number $x\in(\mathbb R\setminus \mathbb Q)\cap (0,1)$. Then you now, that $f(x)=0$. If you like to get a contradiction using a sequence $(r_n)_n$ of rational numbers in $(0,1)$, then you have to find a sequence such that $f(r_n)\not\to 0$. But in that case, there exists $\varepsilon>0$ such that your sequence has infinitely many elements such that $f(r_n)>\varepsilon$. Therefore, we can choose a subsequence $(r_{n_k})_k$ such that $f(r_{n_k})>\varepsilon$ for all $k$. Next, we choose a $q\in\mathbb N$, such that $\varepsilon>\frac1q$. But $f(r_{n_k})>\frac1q$ will lead to the contradition $r_{n_k}\not\to x$.
If you just have a sequence $\left(\frac{p_n}{q_n}\right)_n$, why should you get $\frac1{q_n}\to 0$? It is because of 1. If $\frac1{q_n}\not\to 0$, then $q_n$ is bounded from above. But then $\left(\frac{p_n}{q_n}\right)_n$ has, because of 1., just finitely many different elements and $\frac{p_n}{q_n}\to x$ can't hold, since the distance from $x$ to finitely many rational points is positive.