Big $\mathcal{O}$ notation is a device useful in asymptotic analysis that helps us bound error terms in the approximations that we derive. More specifically, when you see an equation of the form:
$$\text{blah}=\text{stuff}+\mathcal{O}(\text{error})$$
the meaning is that $|\text{blah}-\text{stuff}|\le C\cdot\text{error}$ for some constant $C$. We have to be careful because, as this is asymptotic analysis, we're only claiming this inequality (for some constant) holds in some neighborhood of a region of interest. For example, if our analysis is as $x\to\infty$, we intend the error statement to hold in some neighborhood of infinity, or $(a,\infty)$ for some $a$, or if we look at $\to0^+$ we would mean that the error statement holds in an interval $(0,\epsilon)$ for some $\epsilon>0$.
Take the function $f(x)=1/(1-x)$ for example. This has the following Maclaurin series:
$$\color{Red}{\frac{1}{1-x}}= 1+x+x^2+x^3+\cdots=\color{Blue}{1+x+x^2}+\color{Purple}{x^3}\color{LimeGreen}{\big(1+x+\cdots\big)}.$$
Say we want to do analysis about the point $x=0$. In the interval $(-\frac{1}{2},\frac{1}{2})$ the part in green is no more than $f(1/2)=2$, so we can say that $\big|\color{Red}{f(x)}-\color{Blue}{1+x+x^2}\big|\le \color{LimeGreen}2\color{Purple}{x^3}$, or in Big-$\mathcal{O}$ notation,
$$\frac{1}{1-x}=1+x+x^2+\mathcal{O}(x^3). $$
Keep in mind this analysis applies only sufficiently close to the point $x=0$, so it doesn't apply to the singularity $x=1$ for example. As you hopefully can see, this is a tidy way to convey the asymptotic size of a remainder or leftover term in an approximation, such as a series expansion.
In these easier analyses (exercises with relatively simple and elementary functions), to find the expansion up to $\mathcal{O}(x^n)$ we simply write out the usual series expansion about the appropriate point, but truncate it at the $a_nx^n$ term and replace the deleted terms with a simple $\mathcal{O}(x^n)$.
Mathematicians love to abuse notation when it's convenient, so on top of what I've said so far you need to understand what it means to do operations with Big-$\mathcal{O}$ terms. One mentally helpful interpretation of any $\mathcal{O}(g(x))$ term is "some function bounded by $g$ up to a constant, but what it is specifically isn't important for my purposes or just at the moment." So when you see two $\mathcal{O}$ terms in an equation, you can just imagine the same interpretation for each separately, so there are two unknown error/leftover functions that are independent of each other but at least you're keeping track of how each is bounded in the region under consideration.
Consider, for example, $x^n\mathcal{O}(x^m)$: if $\text{error}(x)\le C x^m$ (our big-$\mathcal{O}$ is a placeholder for $\text{error}$), then we have $x^n\cdot\text{error}(x)\le C x^{n+m}$, which tells us we can just replace $x^n\mathcal{O}(x^m)$ with $\mathcal{O}(x^{n+m})$. Similar reasoning applies to the general term $f(x)\mathcal{O}(g(x))$. Exercise: roughly describe what the product of two $\mathcal{O}$ terms, i.e. $\mathcal{O}(f(x))\mathcal{O}(g(x))$, would mean to us, and find a single $\mathcal{O}$ term for $\mathcal{O}(x^n)\mathcal{O}(x^m)$.
Now if you were able to follow my answer you should be capable of following the explicit derivation given for $\rm5(a)$.
Two notes: in order to evaluate the series expansion of the function composition $f\circ g$, first write the expansion of $f(g)$ in the letter g, i.e. $a_0+a_1g+a_2g^2+\cdots$, and then replacing $g$ with its power series and taking powers where needed, then gathering up all the terms. You do not need to do this with the general, symbolic form of the terms, but rather truncate the expansions of $f$ and $g$ appropriately and work with a finite number of power and $\mathcal{O}$ terms (Exercise: figure out what "appropriately" means here).
Best Answer
The series does not decrease (the partial sums do increase, to infinity); but I reckon what you are trying to say is that its general term $a_n$ is $\Omega\left(\frac{1}{n}\right)$, meaning that there exists a constant $\alpha > 0$ such that asymptotically (for any $n$ "big enough") $a_n \geq \alpha\cdot\frac{1}{n}$.
This is what captures the divergence of the series; saying that the general term is $O\left(\frac{1}{n}\right)$ is not what you want, as it does not imply the series diverges (for instance, $\frac{1}{n^2}$ or even $0$ are also $O\left(\frac{1}{n}\right)$).