It follows from your assumption on the conditial expectation that
$$Z_n = X_n - \sum_{i=0}^{n-1} Y_i$$
is a supermartingale. If we define
$$T_k := \inf\left\{n \in \mathbb{N}; \sum_{i=0}^n Y_i \geq k\right\}$$
for fixed $k \in \mathbb{N}$, then $T_k$ is an $F_n$-stopping time. By the optional stopping theorem, the stopped process $(Z_{n \wedge T_k})_{n \in \mathbb{N}}$ is also a supermartingale. Moreover, the definition of $T_k$ and the non-negativity of $X_n$ and $Y_n$ entail that $Z_{n \wedge T_k} \geq -k$ for all $n \in \mathbb{N}$, and therefore
$$\sup_{n \in \mathbb{N}} \mathbb{E}(Z_{n \wedge T_k}^-) < \infty.$$
Applying the standard convergence theorem for supermartingales, we conclude that the limit $\lim_{n \to \infty} Z_{n \wedge T_k}$ exists almost surely, and so
$$\Omega_0 := \bigcap_{k \geq 1} \{\omega \in \Omega; \lim_{n \to \infty} Z_{n \wedge T_k}(\omega) \, \, \text{exists}\}$$
has probability $1$. Now if $\omega \in \Omega_0$, then $\sum_{n} Y_n(\omega)<\infty$ implies that we can choose $k \in \mathbb{N}$ large enough such that $T_k(\omega)=\infty$. As $\omega \in \Omega_0$ we thus know that
$$\lim_{n \to \infty} Z_{n \wedge T_k}(\omega) = \lim_{n \to \infty} Z_n(\omega) = \lim_{n \to \infty} \left( X_n(\omega)- \sum_{i=0}^{n-1} Y_i(\omega) \right)$$
exists. Using once more that $\sum_{i} Y_i(\omega)<\infty$, this shows that $\lim_n X_n(\omega)$ exists.
Perhaps the reference you are consulting has a different definition, but as I understand, to show that $Y_n$ is Martingale, you have to show that:
i) $Y_n \in \mathcal F_n , \quad \forall n\in \mathbb N$;
ii) $E|Y_n|<\infty ,\quad \forall n \in \mathbb N$;
iii)$ E[Y_n \mid \mathcal F_{n-1}] = Y_{n-1}$.
Now, as you pointed out. The two initial properties can be trivially shown. Only the last one remains. Hence,
$$E[Y_n \mid \mathcal F_{n-1} ]= E[Z_n + Y_{n-1} \mid \mathcal F_{n-1}]=
E[Z_n] - Y_{n-1}= Y_{n-1}
$$
Note that above we used the fact that $Z_n$ is independent and that $E[Z_n] = \frac{an}{2n^2} - 0(1-1/n^2) -\frac{an}{2n^2} =0$.
So $Y_n$ is a martingale.
Finally, you asked in the comments to prove that $Y_n$ converges almost surely. We can prove this using Borel-Cantelli as follows:
If for any $\epsilon>0$, we have $\sum^\infty_{n=1}P(|Y_n|>\epsilon)<\infty$, then, $P(\lim_{n\to\infty}Y_n = 0)=1$
So, for any $\epsilon>0$, we note that
$$P(|Y_n| > \epsilon) \leq P(|Y_n| >0) = P(Y_n = a_n) + P(Y_n = -a_n) = \frac{1}{n^2}$$
Therefore,
$$
\sum^\infty_{n=1}P(|Y_n|>\epsilon)<\infty\implies Y_n \rightarrow_{a.s}0
$$
This concludes the proof.
Best Answer
By Jensen's inequality, $b:=E\left(\sqrt{Y_1}\right)<\sqrt{E(Y_1)}=\sqrt{1}=1$. Therefore $E\left(\sqrt{X_n}\right)=b^n\to 0$ as $n\to\infty$. By Fatou's lemma, $E\left(\sqrt{X}\right)=0$.