[Tex/LaTex] Nested equations with curly braces on the right

Question

I'm trying to write a nested system of equations with a curly brace on the right of each system, some text and then another curly brace which covers all other systems. However, the rightmost curly brace appears too far and too wide and \vdots goes at the opposite direction of the curly brace. The following code demonstrates the problem:

\documentclass{article}
\usepackage{mathtools}
\newenvironment{rcases}
{\left.\begin{aligned}}
{\end{aligned}\right\rbrace}
\begin{document}
\begin{equation*}
\begin{rcases}
\begin{equation*}
\begin{rcases}
P(x_{10},&x_{20},\ldots ,x_{n0}) \nonumber \ \\ 
&\vdots \\
P(x_{1,d+1},&x_{20},\ldots ,x_{n0})
\end{rcases}
\text{Some text}
\end{equation*}
\vdots \\ 
\begin{equation*}
\begin{rcases}
P(x_{10},&x_{2d},\ldots ,x_{n0}) \nonumber \ \\
&\vdots \\
P(x_{1,d+1},&x_{2d},\ldots ,x_{n0})
\end{rcases}
\text{some text}
\end{equation*}
\end{rcases}
\text{some text}
\end{equation*}
\end{document}

Any ideas will be much appreciated!

Answer 1

I don't know what you are trying to achieve. Probably this?

Code:

\documentclass{article}
\usepackage{mathtools}


\begin{document}
\begin{equation*}
\begin{rcases}
\begin{rcases}
P(x_{10},&x_{20},\ldots ,x_{n0}) \\ 
&\vdots \\
P(x_{1,d+1},&x_{20},\ldots ,x_{n0})
\end{rcases}
\text{Some text}\\
\mskip150mu\vdots \\
\begin{rcases}
P(x_{10},&x_{2d},\ldots ,x_{n0}) \\ 
&\vdots \\
P(x_{1,d+1},&x_{2d},\ldots ,x_{n0})
\end{rcases}
\text{some text}
\end{rcases}
\text{some text}
\end{equation*}
\end{document} 

Note that rcases environment is already defined by mathtools and that you can't nest equation environments.