My guess is that the characteristic equation you are departing from is different from mine. Let me proceed in a couple of steps to see whether we agree.
Consider the equation
$$
\lambda^2-\phi_1\lambda-\phi_2=0
$$
If $z$ is a root of the "standard" characteristic equation $1-\phi_1 z-\phi_2 z^2=0$ and setting $z^{-1}=\lambda$, the display obtains from rewriting the standard one as follows:
\begin{eqnarray*}
1-\phi_1 z-\phi_2 z^2&=&0\\
\Rightarrow z^{-2}-\phi_1 z^{-1}-\phi_2 &=&0\\
\Rightarrow \lambda^2-\phi_1\lambda -\phi_2 &=&0
\end{eqnarray*}
Hence, an alternative condition for stability of an $AR(2)$ is that all roots of the first display are inside the unit circle, $|z|>1 \Leftrightarrow |\lambda|=|z^{-1}|<1$.
We use this representation to derive the stationarity triangle of an $AR(2)$ process, that is that an $AR(2)$ is stable if the following three conditions are met:
- $\phi_2<1+\phi_1$
- $\phi_2<1-\phi_1$
- $\phi_2>-1$
Recall that you can write the roots of the first display (if real) as
$$
\lambda_{1,2}=\frac{\phi_1\pm\sqrt{\phi_1^2+4\phi_2}}{2}
$$
to find the first two conditions.
Then, the $AR(2)$ is stationary iff $|\lambda|<1$, hence (if the $\lambda_i$ are real):
\begin{eqnarray*}
-1<\frac{\phi_1\pm\sqrt{\phi_1^2+4\phi_2}}{2}&<&1\\
\Rightarrow -2<\phi_1\pm\sqrt{\phi_1^2+4\phi_2}&<&2
\end{eqnarray*}
The larger of the two $\lambda_i$ is bounded by $\phi_1+\sqrt{\phi_1^2+4\phi_2}<2$, or:
\begin{eqnarray*}
\phi_1+\sqrt{\phi_1^2+4\phi_2}&<&2\\
\Rightarrow \sqrt{\phi_1^2+4\phi_2}&<&2 - \phi_1\\
\Rightarrow \phi_1^2+4\phi_2&<&(2 - \phi_1)^2\\
\Rightarrow \phi_1^2+4\phi_2&<&4 - 4\phi_1+\phi_1^2\\
\Rightarrow \phi_2&<&1 - \phi_1
\end{eqnarray*}
Analogously, we find that $\phi_2<1 + \phi_1$.
If $\lambda_i$ is complex, then $\phi_1^2<-4\phi_2$ and so $$\lambda_{1,2} = \phi_1/2\pm i\sqrt{-(\phi_1^2+4\phi_2)}/2.$$ The squared modulus of a complex number is the square of the real plus the square of the imaginary part. Hence,
$$
\lambda^2 = (\phi_1/2)^2 + \left(\sqrt{-(\phi_1^2+4\phi_2)}/2\right)^2 = \phi_1^2/4-(\phi_1^2+4\phi_2)/4 = -\phi_2.
$$
This is stable if $|\lambda|<1$, hence if $-\phi_2<1$ or $\phi_2>-1$, as was to be shown. (The restriction $\phi_2<1$ resulting from $\phi_2^2<1$ is redundant in view of $\phi_2<1+\phi_1$ and $\phi_2<1-\phi_1$.)
Plotting the stationarity triangle, also indicating the line that separates complex from real roots, we get
Produced in R using
phi1 <- seq(from = -2.5, to = 2.5, length = 51)
plot(phi1,1+phi1,lty="dashed",type="l",xlab="",ylab="",cex.axis=.8,ylim=c(-1.5,1.5))
abline(a = -1, b = 0, lty="dashed")
abline(a = 1, b = -1, lty="dashed")
title(ylab=expression(phi[2]),xlab=expression(phi[1]),cex.lab=.8)
polygon(x = phi1[6:46], y = 1-abs(phi1[6:46]), col="gray")
lines(phi1,-phi1^2/4)
text(0,-.5,expression(phi[2]<phi[1]^2/4),cex=.7)
text(1.2,.5,expression(phi[2]>1-phi[1]),cex=.7)
text(-1.75,.5,expression(phi[2]>1+phi[1]),cex=.7)
Best Answer
The confusion comes from the fact that these conditions (that you state under the label "can be easily proven") pertain to the $Y_t=\varepsilon_t-\theta_1\varepsilon_{t-1}-\theta_2\varepsilon_{t-2}$ formulation. In your case this means $\theta_1=1.3$, but $\theta_2=-0.4$ (not $0.4$!). Substituting these to the conditions, you'll see that all of them is fulfilled.