Time Series – Is White Noise Correlated with ARMA Models?

Question

I know that an ARMA(p, q) refers to the model with p autoregressive terms and q moving-average terms. For example:

$$ X_t = c + \varepsilon_t + \sum_{i=1}^p \varphi_i X_{t-i} + \sum_{i=1}^q \theta_i \varepsilon_{t-i} $$

where $\varepsilon_{t} \sim WN(0,\sigma^2)$. That is, $E(\varepsilon_t) = 0$ and $E(\varepsilon_t \varepsilon_{t-j}) = 0$, for $j\neq 0$. Equivalently, $E(\varepsilon_t | \varepsilon_{t-j}) = 0$.

But in the books I've read, it doesn't tell me anything about the relationship between all the lags of $\varepsilon_t$ and all the lags of $X_t$. For example:

1. Are $X_t$ and $\varepsilon_t$ independent? And how about $X_t$ and $\varepsilon_{t-j}$?
2. If not, are they mean independent? That is: $E[X_t | \varepsilon_{t-j}] = X_t$?
3. Are they not correlated? That is, $cov(X_t,\varepsilon_{t-j})=0$?

Some help?

Answer 1

1. No, $X_t$ and $\varepsilon_t$ are dependent. $X_t$ is a linear function of $\varepsilon_t$, and $X_t$ is positively correlated with $\varepsilon_t$: \begin{aligned} \text{Cov}(X_t,\varepsilon_t) &= \text{Cov}([c + \sum_{i=1}^p \varphi_i X_{t-i} + \sum_{i=1}^q \theta_i \varepsilon_{t-i}] + \varepsilon_t,\varepsilon_t) \\ &= \text{Cov}(\sum_{i=1}^p \varphi_i X_{t-i} + \sum_{i=1}^q \theta_i \varepsilon_{t-i},\varepsilon_t) + \text{Cov}(\varepsilon_t,\varepsilon_t) \\ &= 0 + \sigma^2 \\ &> 0. \end{aligned} $X_t$ and $\varepsilon_{t-j}$ are also dependent. Rewrite ARMA as MA($\infty$), and you will find that $X_t$ is a linear function of $\varepsilon_{t-j}$, and thus an analogous argument to the one above holds also here.
2. No, because of 1.
3. No, because of 1.

If you prefer empirical results to analytical proofs, simulate an ARMA process and check the empirical cross-correlation function of $X_t$ vs. $\varepsilon_t$. Here is how to do that in R:

n=1e6 # length of time series
set.seed(1)
eps=rnorm(n)
x=arima.sim(n=n, list(ar=c(0.8897,-0.4858), ma=c(-0.2279, 0.2488)), sd=1, innov=eps)
ccf(x,eps)