In general, if $F(x)=\sum_{n\ge 0} f_nx^n$ is a formal power series, and if $G(x)=\sum_{\color{blue}{n\ge 1}}g_nx^n$ is a power series for which $G(0)=0$, then the composite power series $(F\circ G)(x)$ is defined as
$$
(F\circ G)(x)= f_0+f_1\cdot G(x)+f_2 [G(x)]^2+f_3[G_3(x)]^3+\dots
$$
The condition $G(0)=0$ is necessary for this to be a well-defined power series.
You can apply this with $F(x)=\log(1+x)=\sum_{n\ge 1} (-1)^{n-1}\frac{x^n}n$ to get a power series representation for $\log(1+G(x))$, which gives meaning to take the $\log$ of $1+G(x)$. This will still satisfy the properties of $\log$, like $\log[(1+G)(1+H)]=\log(1+G)+\log(1+H)$.
A formal power series in the ring $R$ is a sequence
$$
(a_0, a_1, \ldots ),
$$
where the $a_i$ lie in $R$, although this is more often
denoted by
$$
\sum_{n=0}^\infty a_nX^n.
$$
Nevertheless, all of the symbols you see above, except for the coefficients $a_i$ themselves, are just superfluous decorations.
The set of all formal power series is denoted by $R[[X]]$ and it has the structure of a ring, where you add two formal
power
series just by adding their respective coefficients, while the multiplication is defined by
$$
\Big (\sum_{n=0}^\infty a_nX^n\Big ) \Big (\sum_{n=0}^\infty b_nX^n\Big )= \Big (\sum_{n=0}^\infty c_nX^n\Big ),
$$
where
$$
c_n:= \sum_{k=0}^n a_k b_{n-k}.
$$
In addition, $R[[X]]$ is a topological ring by identifying it with the product topological space $R^{\mathbb N}$ in the obvious way, where $R$ is
given the discrete topology.
This means that a sequence
$$
\Big\{\sum_{n=0}^\infty a_{n, k}X^n\Big\}_k
$$
converges to an element $\sum_{n=0}^\infty a_nX^n$ in $R[[X]]$, if and only for every $n$, there is a $k_0$, such that
$a_{n,k}= a_n$, for every $k\geq k_0$. In others words, if you focus on the coefficients of a single $X^n$, you'll notice they stop changing after a while.
Whenever you have a topological ring, you can make sense of convergence of "infinite sums", such as, $\sum_{n=0}^\infty P_n$, by simply
interpreting it as the sequence of partial sums
$$
\Big\{\sum_{n=0}^N P_n\Big\}_N.
$$
In the case of $R[[X]]$, one may choose $P_n: = a_nX^n$ as a formal power series in which all coefficients, except for
$a_n$, are taken to be zero. In that case you may ask whether or not the "infinite sum"
$$
\sum_{n=0}^\infty a_nX^n.
$$
converges in $R[[X]]$. Besides observing that the above is NOT to be understood as an element of $R[[X]]$ (it is an
infinite sum of elements of $R[[X]]$), it is not hard to see that it does converge (Exercise: prove it!) to the element
of $R[[X]]$ you might write as $(a_0, a_1, \ldots )$ in its most formal presentation.
This might be expressed by saying that every power series converges in the ring of formal power series, regardless of how big the coefficients are!
Best Answer
As $\mu(n)\in\{0,1,-1\}$ we have one of three cases.
If $\mu(n)=0$ then $$(1-x^n)^{\mu(n)/n}=1$$ so that $H(x)=0$.
If $\mu(n)=1$ then $$(1-x^n)^{\mu(n)/n}=(1-x^n)^{1/n}=1-\frac{x^n}{n}-(n-1)\frac{x^{2n}}{2n^2} -\cdots$$ so that $$H(x)=-\frac{x^n}{n}+\text{higher terms}.$$
If $\mu(n)=-1$ then $$(1-x^n)^{\mu(n)/n}=(1-x^n)^{-1/n}=1+\frac{x^n}{n}+(n+1)\frac{x^{2n}}{2n^2} -\cdots$$ so that $$H(x)=\frac{x^n}{n}+\text{higher terms}.$$