Suppose $G$ is a $p$-group and $H$ is a $q$-group for different primes $p,q$.
Then $G$ is a characteristic subgroup of $G \rtimes H$ and $H$ is specified up to conjugacy as a Sylow $q$-subgroup.
In particular, for any two semi-direct products $X_i = G \rtimes_{\phi_i} H$ and any isomorphism $f:X_1 \to X_2$ we are guaranteed that $f(G)=G$ and $f(H)$ is $X_2$-conjugate to $H$. In other words, there is an inner automorphism $\nu$ of $X_2$ such that the composition $X_1 \xrightarrow{f} X_2 \xrightarrow{\nu} X_2 = X_1 \xrightarrow{\bar f} X_2$ has the property that $\bar f(G) = G$ and $\bar f(H) = H$.
Consider the centralizer $C_i = C_H(G)$ inside the two groups $X_i$. $$\begin{array}{rl}
C_i
&= \{ (1,h) : (1,h)\cdot_i (g,1) = (g,1) \cdot_i (1,h) \forall g \in G \} \\
&= \{ (1,h) : (\phi_i(h)(g),h) = (g,h) \forall g \in G \} \\
&= \{ (1,h) : \phi_i(h)(g) = g \forall g \in G \} \\
&= \{ (1,h) : \phi_i(h) = 1 \} \\
&= \{ (1,h) : h \in \ker(\phi_i) \} \\
\end{array}
$$
In other words, $C_i$ is the kernel of $\phi_i$. Now $\ker(\phi_1) \neq \ker(\phi_2)$ is not particular relevant. What we care about is whether $\bar f(C_1) = C_2$ (which it must, if $f$ really is an isomorphism). However, $\bar f$ restricted to $H$ is easily seen to be an automorphism of $H$, so what matters is whether there is an automorphism of $H$ that takes $\ker(\phi_1)$ to $\ker(\phi_2)$. If there is not, then there can be no $\bar f$.
On page 187 of Dummit–Foote, exercise 7c, all of these properties are satisfied.
Technical conditions: We needed $G$ characteristic in $X_i$ and $H^1(H,G)=0$. If $G$ is a Hall $\pi$-subgroup of $X$ (for instance a Sylow), then Schur–Zassenhaus guarantees the relevant properties are satisfied. If $G$ is abelian, then we don't need the cohomology condition, just the characteristic.
Counterexample
Without the technical conditions we can have counterexamples. For example let $G=H$ be nonabelian of order 6. Let $\phi_1(h)(g) = g$ and $\phi_2(h)(g) = hgh^{-1}$. Then $\ker(\phi_1) = H$ and $\ker(\phi_2) = 1$ but $X_1 \cong X_2$. In this case $G$ is not characteristic, but we can achieve the isomorphism with an $f$ such that $f(G)=G$. In this case however, there is no $\bar f$ with $\bar f(G) = G$ and $\bar f(H) = H$.
In particular, we have non-isomorphic kernels with isomorphic semi-direct products.
Edit: I didn't realize you were talking about $x$ and $y$ being in the automorphism group, and not the group itself. I've edited below to correct this.
We can take $\alpha$ as a generator of $C_7$, and then $x(\alpha)=\alpha^5$. So $x^2(\alpha) = \alpha^4$ and $x^{-2}(\alpha) = \alpha^2$.
We can take $\beta$ as a generator of $C_{13}$, and then $y(\beta)=\beta^2$, and so $y^4(\beta) = \beta^3$.
Consider your groups as $G=C_{91}\rtimes C_3$ and $H=C_{91}\rtimes C_3$. We can write $z$ as the generator of $C_{91}$, and $g$ and $h$ as the generator of the $C_3$ subgroups.
Then in $G$, we have $g^{-1}zg=z^{-10}$, since $-10$ is $4\pmod{7}$ and $3\pmod{13}$.
In $H$, we have $h^{-1}zh=z^{16}$, since $16$ is $2\pmod{7}$ and $3\pmod{13}$.
If $f:G\rightarrow H$ was an isomorphism, then we would have $f(z)=z^k$ for some integer $k$. We would also have $f(g)$ is some conjugate of $h$ or $h^2$, which thus acts the same as $h$ or $h^2$ on $z$.
So then
\begin{align*}
z^{-10k} &= f(z^{-10})\\
&= f(g^{-1}zg)\\
&= f(g^{-1})z^kf(g)\\
&= z^{16k}\text{ or }z^{74k}
\end{align*}
This implies $z^{26k}=1$ or $z^{84k}=1$, so that $7$ or $13$ divides $k$ respectively, which means $z^k$ is not a generator of $C_{91}$ in $H$. This is the contradiction.
This same idea can be used whenever you have a semidirect product $A\rtimes B$, and $A$ is abelian and $B$ is cyclic. It can be used to differentiate different semidirect products coming from non-conjugate images of $B$ in $\textrm{Aut}(A)$.
Best Answer
For any such $G$, and each $i$, the element $\frac{n}{p_i} \in \mathbb{Z}/n\mathbb{Z}$, together with the generator of $\mathbb{Z}/2\mathbb{Z}$, generates a subgroup $G_i$ of the form $\mathbb{Z}/p_i\mathbb{Z} \rtimes \mathbb{Z}/2\mathbb{Z}$.
I claim that $\varphi_\psi$ can be reconstructed once you know the isomorphism type of each $G_i$.
If you believe my claim, then this is all you need, because there are only two possible semidirect products $\mathbb{Z}/p_i\mathbb{Z} \rtimes \mathbb{Z}/2\mathbb{Z}$: an abelian one and a non-abelian one.
Here's a little push towards seeing why this claim is true: the isomorphism type of $G_i$ is determined precisely by $\varphi_\psi (1) (\frac{n}{p_i})$.