Suppose $G$ is a $p$-group and $H$ is a $q$-group for different primes $p,q$.
Then $G$ is a characteristic subgroup of $G \rtimes H$ and $H$ is specified up to conjugacy as a Sylow $q$-subgroup.
In particular, for any two semi-direct products $X_i = G \rtimes_{\phi_i} H$ and any isomorphism $f:X_1 \to X_2$ we are guaranteed that $f(G)=G$ and $f(H)$ is $X_2$-conjugate to $H$. In other words, there is an inner automorphism $\nu$ of $X_2$ such that the composition $X_1 \xrightarrow{f} X_2 \xrightarrow{\nu} X_2 = X_1 \xrightarrow{\bar f} X_2$ has the property that $\bar f(G) = G$ and $\bar f(H) = H$.
Consider the centralizer $C_i = C_H(G)$ inside the two groups $X_i$. $$\begin{array}{rl}
C_i
&= \{ (1,h) : (1,h)\cdot_i (g,1) = (g,1) \cdot_i (1,h) \forall g \in G \} \\
&= \{ (1,h) : (\phi_i(h)(g),h) = (g,h) \forall g \in G \} \\
&= \{ (1,h) : \phi_i(h)(g) = g \forall g \in G \} \\
&= \{ (1,h) : \phi_i(h) = 1 \} \\
&= \{ (1,h) : h \in \ker(\phi_i) \} \\
\end{array}
$$
In other words, $C_i$ is the kernel of $\phi_i$. Now $\ker(\phi_1) \neq \ker(\phi_2)$ is not particular relevant. What we care about is whether $\bar f(C_1) = C_2$ (which it must, if $f$ really is an isomorphism). However, $\bar f$ restricted to $H$ is easily seen to be an automorphism of $H$, so what matters is whether there is an automorphism of $H$ that takes $\ker(\phi_1)$ to $\ker(\phi_2)$. If there is not, then there can be no $\bar f$.
On page 187 of Dummit–Foote, exercise 7c, all of these properties are satisfied.
Technical conditions: We needed $G$ characteristic in $X_i$ and $H^1(H,G)=0$. If $G$ is a Hall $\pi$-subgroup of $X$ (for instance a Sylow), then Schur–Zassenhaus guarantees the relevant properties are satisfied. If $G$ is abelian, then we don't need the cohomology condition, just the characteristic.
Counterexample
Without the technical conditions we can have counterexamples. For example let $G=H$ be nonabelian of order 6. Let $\phi_1(h)(g) = g$ and $\phi_2(h)(g) = hgh^{-1}$. Then $\ker(\phi_1) = H$ and $\ker(\phi_2) = 1$ but $X_1 \cong X_2$. In this case $G$ is not characteristic, but we can achieve the isomorphism with an $f$ such that $f(G)=G$. In this case however, there is no $\bar f$ with $\bar f(G) = G$ and $\bar f(H) = H$.
In particular, we have non-isomorphic kernels with isomorphic semi-direct products.
With the external construction you can create new groups you may not have seen before, such as building the nonabelian group of order $12$ of the form $\mathbf Z/(3) \rtimes_{\varphi} \mathbf Z/(4)$ where $\mathbf Z/(4)$ acts on $\mathbf Z/(3)$ by negation: $(a,b)(c,d)= (a + (-1)^bc,b+d)$. That is not $A_4$ or $D_6$, which are the familiar nonabelian groups of order $12$: it has a cyclic $2$-Sylow subgroup, while $A_4$ and $D_6$ have noncyclic $2$-Sylow subgroups.
With the internal construction you can recognize groups you know as having this kind of structure, which is nothing other than having a normal subgroup $N$ that admits a complementary subgroup $H$ ($G = NH$ and $N \cap H = \{1\}$.) For instance, the dihedral group $D_n$ of order $2n$ is abstractly of the form $\mathbf Z/(n) \rtimes \mathbf Z/(2)$, where $\mathbf Z/(2)$ acts on $\mathbf Z/(n)$ by inversion ($(a,b)(c,d) = (a+(-1)^bc,b+d)$).
If you want to classify all groups of a particular order, both the internal and external constructions can be useful: in some cases you can show all the groups of a particular order occur as a semidirect product of nontrivial groups, and then to give concrete models of the different types of groups up to isomorphism you may want to show that concrete groups have the desired type of semidirect product structure. See Sections $4$ and $7$ here. (In that file, the notation for semidirect products is $H \rtimes K$, so $H$ denotes the normal subgroup and $K$ is a complementary subgroup, which does not match the way you use the notation $H$ above.)
Best Answer
Here's an example that might help: Let $X$ have order $7$ and $Y$ have order $3$. These groups are cyclic; write $x$ for a generator of $X$ and $y$ for a generator of $Y$. Let $\phi\colon X \to X$ and $\psi\colon X \to X$ be the automorphisms $\phi(x) = x^2$, and $\psi(x) = x^4$.
Notice that $\phi^3(x) = x^{2^3} = x^8 = x$, and $\psi^3(x) = x^{4^3} = x^{64} = x$, so as automorphisms of $X$, $\phi$ and $\psi$ each have order three.
I claim that $X\rtimes_\phi Y \cong X\rtimes_\psi Y$. Indeed, (abusing notation slightly) $X\rtimes_\phi Y$ and $X\rtimes_\psi Y$ are generated by $x$ and $y$, so to specify a homomorphism, I just need to tell you where $x$ and $y$ go, and then check that the map I've written down indeed defines a homomorphism. I claim that the map $x \mapsto x$, $y \mapsto y^{-1}$ is such a homomorphism and moreover that it is an isomorphism. I'll leave it to you to check this.
So if the answer has to be "sometimes" and not "never," maybe we should reconsider what should be true based on this example. Aut$(X)$ is generated by $x \mapsto x^3$. There is an automorphism of Aut$(X)$ that sends the automorphism $x \mapsto x^3$ to the automorphism $x \mapsto x^5$. Under this automorphism of Aut$(X)$, $\phi$ is mapped to $\psi$.
So some questions to investigate: if we have $\phi$ and $\psi$ in Aut$(X)$, will it be the case that $X\rtimes_\phi Y\cong X\rtimes_\psi Y$ whenever there is some automorphism $f\colon$ Aut$(X) \to $ Aut$(X)$ such that $f(\phi) = \psi$? Is this the only condition?