Let $R$ be a complete discrete valuation ring with maximal ideal
$\mathfrak{p}$ and algebraically closed residue field $k$. Denote
the field of fractions of $R$ by $F$. Let $G$ be an affine flat
group scheme of finite type over $R$, and assume that the generic
fibre $G_{F}$ is smooth. Since $G$ is not necessarily smooth over
$R$ the reduction mod $\mathfrak{p}$ map on the points
\[
\rho:G(R)\longrightarrow G(k)
\]
is not necessarily surjective. Let $H_{k}$ denote the image of $\rho$
provided with its canonical affine reduced structure.
Does there exist a group scheme $G'$ of finite type over $R$ such
that $G'_{F}\cong G_{F}$ (i.e., the generic fibres coincide) and
$G'_{k}\cong H_{k}$ (i.e., the special fibre of $G'$ is $H_{k}$)?
An affirmative answer would give some information about the image
$H_{k}$, for example, we would have $\dim H_{k}\geq\dim G_{F}$ by
Chevalley's upper semicontinuity theorem. Therefore any example where
$\dim H_{k}<\dim G_{F}$ would provide a negative answer.
Added:
A way to see that $H_{k}$ is closed in $G(k)$ is by
using a theorem of Greenberg (in "Rational points in Henselian discrete
valuation rings") which implies that there exist two integers $r\geq s\geq1$
such that the image of any $x\in G(R/\mathfrak{p}^{r})$ in $G(R/\mathfrak{p}^{s})$
has a lift to $G(R)$. This implies that $\mathrm{Im}(G(R)\rightarrow G(R/\mathfrak{p}^{s}))=\mathrm{Im}(G(R/\mathfrak{p}^{r})\rightarrow G(R/\mathfrak{p}^{s}))$.
Using the Greenberg functor, we can consider each $G(R/\mathfrak{p}^{i})$
as (the $k$-points of) a linear algebraic group over $k$ (equipped
with its reduced structure in case $G$ is not smooth over $R$),
and by another theorem of Greenberg ($\S\,5$, Prop. 2 and Coroll. 5
in "Schemata over local rings I") each map $G(R/\mathfrak{p}^{i})\rightarrow G(R/\mathfrak{p}^{j})$
with $i\geq j\geq1$ induces an algebraic homomorphism. Thus $X:=\mathrm{Im}(G(R)\rightarrow G(R/\mathfrak{p}^{s}))$
is a closed subgroup of $G(R/\mathfrak{p}^{s})$ and hence $H_{k}=\mathrm{Im}(X\rightarrow G(k))$
is a closed subgroup of $G(k)$.
Best Answer
Note that the question does not make sense until we know that the image of $\rho$ is closed, and such closedness is not obvious since not even construcibility is apparent at the outset (e.g., Chevalley's theorem on constructibility is not relevant here, as $\rho$ is in no sense an algebraic morphism). Also, it isn't said if $G'$ is supposed to be separated, perhaps even affine, but presumably you want $G'$ to be (quasi-)affine?
The only way I know to see the closedness of the image of $\rho$ is by a procedure that answers the entire question. Namely, consider the canonically associated group smoothening $f:\widehat{G} \rightarrow G$ in the sense of section 7.1 of the book "Neron Models". Since $G$ is a flat affine $R$-group of finite type, and the group smoothening is an $R$-homomorphism built from a "dilatation" process that entails forming a certain relatively affine open chart in a blow-up, it is immediate from the construction that the smooth group $\widehat{G}$ is affine, and by design $\widehat{G}(R) = G(R)$ and $f_F$ is an isomorphism. Consequently, since $\widehat{G}(R) \rightarrow \widehat{G}(k)$ is surjective by smoothness of $\widehat{G}$, it follows that $\rho(G(R))$ is the image on $k$-points of the special fiber $k$-homomorphism $f_k:\widehat{G}_k \rightarrow G_k$. Now we can apply our experience with morphisms of algebraic groups over fields to know that $\rho(G(R))$ is closed, since (i) it is just the set of $k$-points of the image of $f_k$ since $k$ is algebraically closed and (ii) the image of $f_k$ is closed (as for any homomorphism between finite type group schemes over a field).
This argument gives more, namely that $\dim H_k \le \dim \widehat{G}_k = \dim \widehat{G}_F = \dim G_F$, whereas you noted that if the $G'$ you seek is to exist then $\dim H_k \ge \dim G_F$, so we would be forced to have $\dim H_k = \dim G_F$, but $\dim G_F = \dim G_k$ by $R$-flatness of $G$, so in such cases $H_k$ would have to contain the identity component $G_k^0$ and then $f_k$ has finite kernel and it has image containing $G_k^0$. So we see that in such cases $H_k$ is a union of some of the connected components of $G_k$, and hence $G'$ can be taken to be the open subscheme of $G$ given by removing the components of $G_k$ not in the image of $f_k$.
The upshot is that the answer is affirmative if and only if $\rho(G(R))$ contains $G_k^0(k)$, and that this is also equivalent to the condition that the special fiber of the group smoothening morphism has finite kernel. (It also makes me wonder if perhaps you would prefer simply to work with $\widehat{G}$ in general and forget about $G'$.)
--
We can say more, by relating the group smoothening to the normalization of $G$. Note that the normalization map $\widetilde{G} \rightarrow G$ is finite since $R$ is excellent, so $\widetilde{G}$ is finite type over $R$ and visibly $R$-flat with the same generic fiber as $G$ over $F$.
Suppose we're in the case with an affirmative answer, so $f_k$ is quasi-finite, and hence $f$ is quasi-finite (as $f_F$ is an isomorphism). Since $f$ is an isomorphism between (possibly disconnected) generic fibers and $\widehat{G}$ is smooth (hence normal), $f$ uniquely factors through the normalization, say via $h:\widehat{G} \rightarrow \widetilde{G}$. But $f$ is quasi-finite, so $h$ is also quasi-finite. Yet $h_F$ is an isomorphism, so by Zariski's Main Theorem and normality considerations we see that $h$ is an open immersion. The quasi-finite $f_k$ must be finite (as for any quasi-finite homomorphism between finite type group schemes over a field), so $h_k$ is finite too. Yet $h_k$ is an open immersion (since $h$ is), so $h_k$ identifies $\widehat{G}_k$ with a union of connected components of $\widetilde{G}_k$ (with their natural open subscheme structure). Since $\widetilde{G}$ is $R$-flat (of finite type), so smooth points in its special fiber are precisely where the Zariski-open $R$-smooth locus of $\widetilde{G}$ meets $\widetilde{G}_k$, and all smooth $k$-points of $\widetilde{G}_k$ lift to $\widetilde{G}(R) = G(R) = \widehat{G}(R)$ (first equality by finiteness of normalization), we conclude that $\widehat{G}$ is exactly the smooth locus in $\widetilde{G}$ in these cases.
But in fact we need much less about the normalization to deduce the same conclusion: suppose that the $R$-flat $\widetilde{G}$ merely has non-empty smooth locus in its special fiber, or in other words that the Zariski-open $R$-smooth locus has non-empty special fiber. That provides a dense open locus of smooth points in some irreducible component $X$ of the special fiber of $\widetilde{G}$, and those all lift to $\widetilde{G}(R)=G(R)$ yet clearly $X$ maps finitely onto an irreducible component $Y$ of $G_k$ (as $G_k$ and $\widetilde{G}_k$ are equidimensional of the same dimension, due to $R$-flatness and equidimensionality of their common generic fiber $G_F$), so a dense open locus in $Y(k)$ lifts to $G(R)$, forcing $\rho(G(R))$ to contain a dense open locus in some irreducible component (= connected component) of $G_k$. Hence, that would force $\dim H_k = \dim G_k$.
To summarize, the following conditions are equivalent: $G'$ exists, $H_k$ contains $G_k^0(k)$, $f_k$ has finite kernel, $\widetilde{G}_k$ has non-empty smooth locus, $\widehat{G}$ is an open subscheme of $\widetilde{G}$. In such cases, $\widehat{G}_k \rightarrow G_k$ is a finite homomorphism with image $H_k$, and $\widehat{G} \rightarrow G$ has open image that we can take as $G'$.