Theorem (EGA IV 13.1.3): Let $f \colon X \to Y$ be a morphism of schemes, locally of finite type. Then
$$x \mapsto \dim_x(X_{f(x)})$$
is upper semi-continuous.
Corollary (Chevalley's upper semi-continuous theorem, EGA IV 13.1.5): Let $f \colon X \to Y$ be proper, then:
$$y \mapsto \dim(X_y)$$
is upper semi-continuous.
Corollary (SGA3, ??): Let $X/Y$ be a group scheme, locally of finite type. Then
$$y \mapsto \dim(X_y)$$
is upper semi-continuous.
Proof: The dimension of a group scheme over a field is the same as the dimension at the identity. Thus the function
$$y \mapsto \dim(X_y)$$
is the composition of the continuous function $y \to e(y)$ and the upper semi-continuous function $x \mapsto \dim_x(X_{f(x)})$.
Concerning your application: The fiber dimensions of the stabilizer group scheme Stab/X is upper semi-continuous, but the "diagonal" $G \times X \to X \times X$ does not always have this property (unless it is proper, i.e., "$G$ acts properly").
Note that the question does not make sense until we know that the image of $\rho$ is closed, and such closedness is not obvious since not even construcibility is apparent at the outset (e.g., Chevalley's theorem on constructibility is not relevant here, as $\rho$ is in no sense an algebraic morphism). Also, it isn't said if $G'$ is supposed to be separated, perhaps even affine, but presumably you want $G'$ to be (quasi-)affine?
The only way I know to see the closedness of the image of $\rho$ is by a procedure that answers the entire question. Namely, consider the canonically associated group smoothening $f:\widehat{G} \rightarrow G$ in the sense of section 7.1 of the book "Neron Models". Since $G$ is a flat affine $R$-group of finite type, and the group smoothening is an $R$-homomorphism built from a "dilatation" process that entails forming a certain relatively affine open chart in a blow-up, it is immediate from the construction that the smooth group $\widehat{G}$ is affine, and by design $\widehat{G}(R) = G(R)$ and $f_F$ is an isomorphism. Consequently, since $\widehat{G}(R) \rightarrow \widehat{G}(k)$ is surjective by smoothness of $\widehat{G}$, it follows that $\rho(G(R))$ is the image on $k$-points of the special fiber $k$-homomorphism $f_k:\widehat{G}_k \rightarrow G_k$. Now we can apply our experience with morphisms of algebraic groups over fields to know that $\rho(G(R))$ is closed, since (i) it is just the set of $k$-points of the image of $f_k$ since $k$ is algebraically closed and (ii) the image of $f_k$ is closed (as for any homomorphism between finite type group schemes over a field).
This argument gives more, namely that $\dim H_k \le \dim \widehat{G}_k = \dim \widehat{G}_F = \dim G_F$, whereas you noted that if the $G'$ you seek is to exist then $\dim H_k \ge \dim G_F$, so we would be forced to have $\dim H_k = \dim G_F$, but $\dim G_F = \dim G_k$ by $R$-flatness of $G$, so in such cases $H_k$ would have to contain the identity component $G_k^0$ and then $f_k$ has finite kernel and it has image containing $G_k^0$. So we see that in such cases $H_k$ is a union of some of the connected components of $G_k$, and hence $G'$ can be taken to be the open subscheme of $G$ given by removing the components of $G_k$ not in the image of $f_k$.
The upshot is that the answer is affirmative if and only if $\rho(G(R))$ contains $G_k^0(k)$, and that this is also equivalent to the condition that the special fiber of the group smoothening morphism has finite kernel. (It also makes me wonder if perhaps you would prefer simply to work with $\widehat{G}$ in general and forget about $G'$.)
--
We can say more, by relating the group smoothening to the normalization of $G$. Note that the normalization map $\widetilde{G} \rightarrow G$ is finite since $R$ is excellent, so $\widetilde{G}$ is finite type over $R$ and visibly $R$-flat with the same generic fiber as $G$ over $F$.
Suppose we're in the case with an affirmative answer, so $f_k$ is quasi-finite, and hence $f$ is quasi-finite (as $f_F$ is an isomorphism). Since $f$ is an isomorphism between (possibly disconnected) generic fibers and $\widehat{G}$ is smooth (hence normal), $f$ uniquely factors through the normalization, say via $h:\widehat{G} \rightarrow \widetilde{G}$. But $f$ is quasi-finite, so $h$ is also quasi-finite. Yet $h_F$ is an isomorphism, so by Zariski's Main Theorem and normality considerations we see that $h$ is an open immersion. The quasi-finite $f_k$ must be finite (as for any quasi-finite homomorphism between finite type group schemes over a field), so $h_k$ is finite too. Yet $h_k$ is an open immersion (since $h$ is), so $h_k$ identifies $\widehat{G}_k$ with a union of connected components of $\widetilde{G}_k$ (with their natural open subscheme structure). Since $\widetilde{G}$ is $R$-flat (of finite type), so smooth points in its special fiber are precisely where the Zariski-open $R$-smooth locus of $\widetilde{G}$ meets $\widetilde{G}_k$, and all smooth $k$-points of $\widetilde{G}_k$ lift to $\widetilde{G}(R) = G(R) = \widehat{G}(R)$ (first equality by finiteness of normalization), we conclude that $\widehat{G}$ is exactly the smooth locus in $\widetilde{G}$ in these cases.
But in fact we need much less about the normalization to deduce the same conclusion: suppose that the $R$-flat $\widetilde{G}$ merely has non-empty smooth locus in its special fiber, or in other words that the Zariski-open $R$-smooth locus has non-empty special fiber. That provides a dense open locus of smooth points in some irreducible component $X$ of the special fiber of $\widetilde{G}$, and those all lift to $\widetilde{G}(R)=G(R)$ yet clearly $X$ maps finitely onto an irreducible component $Y$ of $G_k$ (as $G_k$ and $\widetilde{G}_k$ are equidimensional of the same dimension, due to $R$-flatness and equidimensionality of their common generic fiber $G_F$), so a dense open locus in $Y(k)$ lifts to $G(R)$, forcing $\rho(G(R))$ to contain a dense open locus in some irreducible component (= connected component) of $G_k$. Hence, that would force $\dim H_k = \dim G_k$.
To summarize, the following conditions are equivalent: $G'$ exists, $H_k$ contains $G_k^0(k)$, $f_k$ has finite kernel, $\widetilde{G}_k$ has non-empty smooth locus, $\widehat{G}$ is an open subscheme of $\widetilde{G}$. In such cases, $\widehat{G}_k \rightarrow G_k$ is a finite homomorphism with image $H_k$, and $\widehat{G} \rightarrow G$ has open image that we can take as $G'$.
Best Answer
Let $X = (\mathbb{A}^2 \setminus \{x = 0\}) \coprod \mathbb{A}^1$, let $Y = \mathbb{A}^1$, and let $f$ be projection onto the first coordinate on the first component and the identity on the second. Then every point of $X$ lives in a one-dimensional fiber except the origin of the second component.