Here's an approach that, in a sense, combines both methods you were considering. The idea is to inductively embed ordinals into $(X, \le_X)$ until you can't any more, and then deduce that the ordinal you finished up with is the ordinal you seek.
First note that $(0, \in)$ embeds trivially into $(X, \le_X)$.
Suppose there is an embedding $i_{\alpha} : (\alpha, \in) \hookrightarrow (X, \le_X)$, and consider the set $X_{\alpha} = X \setminus i_{\alpha}[\alpha]$.
- If $X_{\alpha}$ is empty, then you can deduce that $i_{\alpha}$ is an isomorphism.
- If $X_{\alpha}$ is inhabited, then it has a $\le_X$-least element $x_{\alpha}$, and then there is an embedding $i_{\alpha+1} : (\alpha + 1, \in) \hookrightarrow (X, \le_X)$ which extends $i_{\alpha}$ and satisfies $i_{\alpha}(\alpha) = x_{\alpha}$.
Now suppose $\beta$ is a limit ordinal and that there are embeddings $i_{\alpha} : (\alpha, \in) \hookrightarrow (X, \le_X)$ for each $\alpha < \beta$ such that $i_{\alpha} \subseteq i_{\alpha'}$ for all $\alpha \le \alpha' < \beta$. Define $i_{\beta} = \bigcup_{\alpha < \beta} i_{\alpha}$ and note that $i_{\beta}$ is an embedding $(\beta, \in) \hookrightarrow (X, \le_X)$.
This process must eventually terminate, since $(\alpha, \in)$ does not embed into $(X, \le_X)$ whenever $|\alpha| > |X|$, and the only way it can terminate is when for some $\alpha$ we have $i_{\alpha} : (\alpha, \in) \hookrightarrow (X, \le_X)$ and $X_{\alpha} = \varnothing$.
There are a couple of gaps in the proof to be filled in, but I'm sure you can flesh out the details.
$\newcommand{\sup}{\operatorname{sup_{i<\xi}}}$
One can do this using induction and basic ordinal arithmetic only.
First lets observe a few rules:
- $\gamma$ is continuous, that is whenever $\alpha$ is the limit of a sequence $(\alpha_i)_{i<\delta}$ then $\gamma(\alpha)=\operatorname{sup}_{i<\delta}\gamma(\alpha_i)$
- $\gamma(\alpha+1)=\gamma(\alpha)+\alpha 2+1$
From 2. it follows by induction that $\gamma(\alpha+ n)=\gamma(\alpha)+\alpha 2n +1$ for infinite $\alpha$ and thus we get
- $\gamma(\alpha+\omega)=\gamma(\alpha)+\alpha\omega$
Let $A(\beta,n,\eta)$ denote
$$\gamma(\omega^\beta n+\eta)=\gamma(\omega^\beta)+\omega^\beta(\omega^\beta(n-1)+\eta)$$
for $\beta\geq 1, 1\leq n<\omega$ and $\eta<\omega^\beta$ not a successor.
In anything what follows, $\beta$ will always be $\geq 1$ and $n$ will be a positive integer and $\eta$ will be an ordinal which is not a successor.
First we will see
If $A(\beta,n, 0)$ holds, then $A(\beta, n, \eta)$ holds for all non-successor $\eta\leq \omega^\beta$.
So assume $A(\beta, n, \eta)$ is true and $\eta<\omega^\beta$. We will show $A(\beta, n, \eta+\omega)$.
We have:
$$\gamma(\omega^\beta n+\eta+\omega)=\gamma(\omega^\beta n+\eta)+(\omega^\beta n+\eta)\omega=\gamma(\omega^\beta)+\omega^\beta(\omega^\beta(n-1)+\eta)+(\omega^\beta n+\eta)\omega$$
Since $\eta<\omega^\beta$, we can simplify the last term $(\omega^\beta n+\eta)\omega$ to $\omega^\beta\omega$. Finally,
$$ \gamma(\omega^\beta)+\omega^\beta(\omega^\beta(n-1)+\eta)+\omega^\beta\omega=\gamma(\omega^\beta)+\omega^\beta(\omega^\beta(n-1)+\eta+\omega)$$
For the limit case, assume $\eta=\sup\eta_i$ for limit ordinals $\eta_i$ so that $A(\beta, n,\eta_i)$ holds for all $i<\xi$. Then by 1.:
$$\gamma(\omega^\beta n+ \eta)=\sup\gamma(\omega^\beta n+ \eta_i)=\sup\gamma(\omega^\beta)+\omega^\beta(\omega^\beta (n-1)+\eta_i)=\gamma(\omega^\beta)+\omega^\beta(\omega^\beta (n-1)+\eta)$$
To verify the last equality, note that $\sup\delta(\mu+\eta_i)=\sup\delta\mu+\delta\eta_i=\delta\mu+\delta\eta=\delta(\mu+\eta)$ for any choice of $\delta, \mu$.
Next up:
If $A(\beta, n, 0)$ holds then $A(\beta, n+1, 0)$ does, too.
We already know that $A(\beta, n, \omega^\beta)$ is true. But this implies
$$\gamma(\omega^\beta (n+1))=\gamma(\omega^\beta n+\omega^\beta)=\gamma(\omega^\beta)+\omega^\beta(\omega^\beta(n-1)+\omega^\beta)=\gamma(\omega^\beta)+\omega^\beta(\omega^\beta n+ 0)$$
which is what we desire.
Note that proving $A(\beta, 1, 0)$ for any $\beta$ is trivial, as this amounts to $\gamma(\omega^\beta)=\gamma(\omega^\beta)$.
All in all, $A(\beta, n, \eta)$ holds for all appropriate $\beta, n, \eta$.
Best Answer
This is actually much simpler than you'd expect, if you pay attention to the right things. In fact, you've almost proved this already.
Since $\Gamma$ is a strictly increasing function, its range cannot be a set of ordinals. So it has to be unbounded. But now simply note that if $\Gamma(\alpha,\beta)=\gamma$ and $\delta<\gamma$, then there is a unique initial segment below $(\alpha,\beta)$ which is isomorphic to $\delta$. And by well-ordering, there is a least $(\alpha',\beta')$ below $(\alpha,\beta)$ such that $\Gamma(\alpha',\beta')=\delta$.
In other words, we show that the image must be downwards closed, and not bounded (i.e., not a set). Therefore it has to be an isomorphism.