These are essentially the same definitions. The series described by Borcherds is called the "upper central series": it can also be defined as the series
$$\{1\} = Z_0(G)\leq Z_1(G)\leq\cdots\leq Z_n(G)$$
where $Z_k(G)$ is called the $k$-th center of $G$, and is the subgroup of $G$ such that $Z_k(G)/Z_{k-1}(G) = Z(G/Z_{k-1}(G))$. A group $G$ is nilpotent if the upper central series has $Z_k(G)=G$ for some $k$. Note that because $Z(G/Z_{k-1}(G))$ is normal in $G/Z_{k-1}(G)$, it follows that $Z_k(G)$ is normal in $G$, so this is a normal series.
The normal series you describe is said to be a "central series" for $G$:
$$1 = N_0\leq N_1\leq\cdots \leq N_k$$
with $N_i\triangleleft G$ and such that $N_i/N_{i-1}\subseteq Z(G/N_{i-1})$. A group is nilpotent if there is a central series that terminates at $G$, i.e., $N_m=G$ for some $m$.
To verify the two definitions are the same, note that the upper central series is a central series that terminates at $G$: it's just that instead of having $N_i/N_{i-1}\subseteq Z(G/N_{i-1})$, you have equality. So a group that satisfies the upper central series definition also satisfies the definition you knew.
To show that any group satisfying the definition you knew also satisfies the one given by Borcherds, assume that $G$ has a central series
$$ 1 = N_0 \leq N_1\leq\cdots \leq N_m=G$$
with $N_i/N_{i-1}\subseteq Z(G/N_{i-1})$. I claim that $N_k\leq Z_k(G)$ for all $k$. We proceed by induction on $k$.
By construction, $N_1 = (N_1/N_0)\leq Z(G/N_0) = Z(G)= Z_1(G)$, so the result holds for $k=1$.
Assume that $N_k\leq Z_k(G)$; then $G/N_k$ maps onto $G/Z_k(G)$, and therefore the image of $Z(G/N_k)$ maps into $Z_{k+1}(G)/Z_k(G)$ (since the image of the center is contained in the center of the image). Thus, every element of $N_{k+1}$ maps into $Z_{k+1}(G)$, yielding the desired result. Thus, a group having a central series must satisfy the definition given by Borcherds.
Another definition of nilpotent uses the "lower central series", where $G_0=G$, $G_{n+1}=[G_n,G]$, and the group is nilpotent if and only if the lower central series terminates in the trivial group: there exists $k$ such that $G_k=\{e\}$. The reason for the names is that any central series lies, in a sense, between the lower central series and the upper central series. Proving the upper and lower central series definitions are equivalent is a little trickier than the one you are asking about.
cf. 5.1.9 in Derek Robinson's A Course in the Theory of Groups 2nd edition:
Let $1=N_0\leq N_1\leq\cdots\leq N_n = G$ be a central series in a (necessarily nilpotent) group. Then:
- $G_i\leq N_{n-i+1}$ for each $i$, so $G_{n+1}=1$.
- $N_i\leq Z_i(G)$, so that $Z_n(G)=G$.
- The upper and lower central series of $G$ have the same length.
Yes, both hold.
For any family $\{H_i\}_{i\in I}$ of subsets (not necessarily subgroups) of $G$, $\langle H_i\rangle$ is the subgroup $K$ that satisfies the following two conditions:
- $H_i\subseteq K$ for each $i\in I$; and
- if $M$ is any subgroup of $G$ such that $H_i\subseteq M$ for every $i\in I$, then $K\subseteq M$.
This is the "top-down description" of the subgroup generated. You use these properties in the second displayed equation after $(2)$.
We want to show that
$$\bigl\langle G_1\cup G_2\cup G_3\bigr\rangle = \Bigl\langle \bigl\langle G_1\cup G_2\bigr\rangle\cup G_3\Bigr\rangle.$$
(We don't even need to assume these are subgroups).
$\subseteq)$ Note that
$$\begin{align*}
G_1\cup G_2\subseteq \langle G_1\cup G_2\rangle &\subseteq \Bigl\langle \bigl\langle G_1\cup G_2\bigr\rangle\cup G_3\Bigr\rangle\\
G_3\subseteq \langle G_1\cup G_2\rangle\cup G_3 &\subseteq \Bigl\langle \bigl\langle G_1\cup G_2\bigr\rangle\cup G_3\Bigr\rangle
\end{align*}$$
Therefore,
$$(G_1\cup G_2)\cup G_3 = G_1\cup G_2\cup G_3 \subseteq
\Bigl\langle \bigl\langle G_1\cup G_2\bigr\rangle \cup G_3\Bigr\rangle,$$
and since $\langle G_1\cup G_2\cup G_3\rangle$ is the smallest subgroup that contains $G_1\cup G_2\cup G_3$, we have
$$\langle G_1\cup G_2\cup G_3\rangle \subseteq \Bigl\langle\bigl\langle G_1\cup G_2\bigr\rangle\cup G_3\Bigr\rangle.$$
$\supseteq)$ Since $G_1\cup G_2\subseteq G_1\cup G_2\cup G_3\subseteq \langle G_1\cup G_2\cup G_3\rangle$, we have
$\langle G_1\cup G_2\rangle \subseteq \langle G_1\cup G_2\cup G_3\rangle$.
We also have $G_3\subseteq G_1\cup G_2\cup G_3\subseteq \langle G_1\cup G_2\cup G_3\rangle$.
Since both $\langle G_1\cup G_2\rangle$ and $G_3$ are contained in $\langle G_1\cup G_2\cup G_3\rangle$, their union is contained there as well, so
$$\langle G_1\cup G_2\rangle\cup G_3\subseteq \bigl\langle G_1\cup G_2\cup G_3\bigr\rangle,$$
and therefore
$$\Bigl\langle \bigl\langle G_1\cup G_2\bigr\rangle \cup G_3\Bigr\rangle \subseteq \bigl\langle G_1\cup G_2\cup G_3\bigr\rangle,$$
as desired. $\Box$
More generally, for any family $\{B_i\}_{i\in I}$ of subsets of $G$, and any partition $\{I_j\}_{j\in J}$ of $I$ (so $\cup_{j\in J}I_j=I$), we have
$$\left\langle \bigcup_{i\in I}B_i\right\rangle = \left\langle \bigcup_{j\in J}\left\langle \cup_{i\in I_j}B_k\right\rangle\right\rangle.$$
As to whether your description of $(1)$ is accurate, note that in general if $S$ is a subset of $G$, $\langle S\rangle$ is the collection of all finite products of elements of $S$ and their inverses. But if $S$ is closed under inverses, that is, if $S$ satisfies the condition
$$\text{if }s\in S,\text{ then }s^{-1}\in S,$$
then this is exactly the same thing as the collection of all finite products of elements of $S$. (Note the empty product is a finite product and gives the identity element). Since your $\cup\mathcal{H}$ is a union of subgroups, it is certainly closed under inverses; and you can ignore the empty product since the set already contains $e$; so your description of $\langle \cup\mathcal{H}\rangle$ is accurate if the family of subgroups is nonempty.
Best Answer
The proof you refer to applies here in just the same way.
For any element $g$ of the group, consider $g^{-1}Ng=\sum g^{-1}g_ig$. The elements $g^{-1}g_ig$ are (in some order) precisely all the elements of the group and so $g^{-1}Ng=N$.
N.B. Your comment at the end of your proof is correct. This works because of the 'Latin Square' property of groups.