Let me add more details to my comment above. Let $S$ be a scheme. Let $\overline{X}$ be a proper $S$-scheme, and let $X\subset \overline{X}$ be a dense Zariski open subscheme.
A closed subset $R\subset X\times_S X$ is an algebraic equivalence relation if it contains the diagonal (i.e., it is reflexive), if it is invariant under the involution $(\text{pr}_2,\text{pr}_1)$ of $X\times_S X$ (i.e., it is symmetric), and if it is transitive, i.e., $R$ contains the image of the following composition, $$R\times_{\text{pr}_2,X,\text{pr}_1} R \hookrightarrow (X\times_S X) \times_{\text{pr}_2,X,\text{pr}_1} (X\times_S X) = X\times_S X\times_S X \xrightarrow{\text{pr}_1,\text{pr}_3} X\times_S X.$$ For instance, for a group scheme $G$ over $S$ with an $S$-action on $X$, the closure $Z$ of the image of the associated map is an algebraic equivalence relation,
$$\Psi:G\times_S X \to X\times_S X, \ (g,x) \mapsto (g\cdot x,x).$$
Denote by $\overline{R}$ the closure of $R$ in $X\times_S \overline{X}$.
For the projection, $\text{pr}_1:\overline{R}\to X$, there is a maximal open subscheme $U$ of $X$ over which the projection is flat. If $X$ is reduced and Noetherian, then $U$ is a dense open subscheme by Grothendieck's generic flatness / generic freeness theorem. Thus, there is an induced $S$-morphism from $U$ to the relative Hilbert scheme, $$f_{\overline{R}}:U \to \operatorname{Hilb}_{\overline{X}/S}.$$ This is the "modern" take on the classical construction of a quotient of $R$ as a "rational map".
In fact, in terms of making $U$ as big as possible, it is usually better to work with the Chow scheme (due to Angeniol in characteristic $0$, and due to David Rydh in positive characteristic and mixed characteristic). The maximal open subscheme $V$ of $X$ over which $\overline{R}$ is a "good algebraic cycle", and thus defines a morphism from $V$ to the relative Chow scheme, always contains $U$ by the existence of the Hilbert–Chow morphism.
As I mentioned in my comment, this construction is in Kollár's book Rational curves on algebraic varieties, specifically in Section 4 of Chapter IV. I do not believe that he explicitly singles out the application to forming group quotients, since that is not his main concern (he wants to construct the maximal quotient by the family of rational curves, not group quotients).
Best Answer
No, this is not true. For instance the symplectic group $$ G = \mathrm{Sp}(V), $$ where $V$ is a symplectic vector space of dimension 6 has two irreducible representations of dimension 14: $$ V_1 = \wedge^2V^\vee / \langle \omega \rangle, \qquad\text{and}\qquad V_2 = \wedge^3V^\vee / (V^\vee \wedge \omega), $$ where $\omega$ is the symplectic form. These representations are not isomorphic, as well as the corresponding closed orbits, which are the isotropic Grassmannians $$ X_1 = \mathrm{IGr}(2,V) \qquad\text{and}\qquad X_2 = \mathrm{IGr}(3,V). $$
By the way, the inverse is also not true unless the line bundles associated with the embeddings of $X \to \mathbb{P}(V)$ and $X' \to \mathbb{P}(V')$ are identified by the isomorphism $X \cong X'$.