The function $f(x)=\frac {x(5-x)} 4$ maps $[0,2]$ into itself and it is increasing there. From this it follows by induction that $x_n \in [0,1]$ for all $n$ and that $x_{n+1} \geq x_n$. Hence $a\equiv \lim x_n$ exists and since $a=\frac {a(5-a)} 4$ we get $a=0$ or $a=1$. Since $(a_n)$ is increasing it follows that $a=0$ if $x_0=0$ and $a=1$ otherwise.
I think I've understood your attempt (as best as I can without being able to read the language) and here are my thoughts:
Your first point is correct: if $a \neq 0$ then we must have $\lim_{n \to \infty} \frac{x_{n+1}}{x_n} = \frac{\lim_{n \to \infty} x_{n + 1}}{\lim_{n \to \infty} x_n} = \frac{a}{a} = 1.$ If you'd like we could show this in the epsilon-delta form like this: if $|x_n - a| \leq \epsilon$ for all $n \geq n_0(\epsilon),$ then
$$|x_{n + 1} - x_n| \leq |x_{n + 1} - a| + |x_n - a| \leq 2\epsilon \ \text{for } n \geq n_0(\epsilon)$$
$$|x_n - a| \leq \epsilon \Rightarrow a - \epsilon \leq x_n \leq a + \epsilon \Rightarrow |x_n| \geq a - \epsilon \text{ for } n \geq n_0(\epsilon), \epsilon < a$$
$$\left|\frac{x_{n + 1}}{x_n} - 1\right| = \frac{|x_{n + 1} - x_n|}{|x_n|} \leq \frac{2\epsilon}{a - \epsilon} \text{ for } n \geq n_0(\epsilon)$$
and with a little algebraic manipulation we get that $\left|\frac{x_{n+1}}{x_n} - 1\right| \leq \epsilon_2$ for all $n \geq n_0\left(\frac{a\epsilon_2}{\epsilon_2 + 2}\right).$
For your second point, I agree with your counterexample for the case where $\frac{x_{n+1}}{x_n}$ need not have a limit when $a = 0$ (and I quite like the counterexample you came up with, good job) but I would justify the lack of existence of the limit a bit differently: rather than taking limits of the terms $\frac{x_{2n}}{x_{2n - 1}}$ and $\frac{x_{2n + 1}}{x_{2n}},$ I would just say that the sequence $b_n = \frac{x_{n + 1}}{x_n}$ oscillates between $1$ and $\frac{1}{p},$ and use that to say there is no limit. In any case you have the right idea, the limit does not exist.
For the third part of your argument, where you argue that if the limit exists in the $a = 0$ case then its magnitude must be no more than $1,$ I think you can avoid a little bit of annoyance with $\lambda$ relying on $n$ by tightening your imposed restriction on $\epsilon$: instead of saying there must be an $\epsilon > 0$ such that $|b| - \epsilon > 1$ when $|b| > 1,$ I would instead say there must be an $\epsilon > 0$ such that $|b| - \epsilon > \frac{|b| + 1}{2},$ which will happen when $0 < \epsilon < \frac{|b| - 1}{2}.$ Now you can proceed by proving that $x_n \geq x_{n_0} \cdot \left(\frac{|b| + 1}{2}\right)^{n - n_0}$ for $n \geq n_0,$ and from there your argument regarding comparing $x_n$ to a divergent exponential function follows.
Overall, good work. It's clear you know what you're doing, there are just a few spots I think things could be tightened up.
Best Answer
Let us notice that $3x_n\leq x_{n+1}\leq 3x_n+2$. So $3^{N-1}\leq x_{N} \leq 3^{N-1}+3^{N-1}-1<2*3^{N-1}$. As $\lim_{n\to\infty} |3^{n}a|^{1/n}=3$ for any constant $a$, so our limit is also equals to $3$.