This question follows from my previous question, where Robin answered the question in the case of weak stationary processes. Here, I am asking a similar question for (strong?) stationary processes. I'll define what this means (the definition can also be found here).
Let $X(t)$ be a stochastic process. We say that $X(t)$ is Nth-order stationary if, for every $t_1, t_2, \dots, t_N$ we have that the joint cumulative density functions
$$F_{X(t_1),X(t_2),\dots,X(t_N)} = F_{X(t_1 + \tau),X(t_2 + \tau),\dots,X(t_N + \tau)}$$
for all $\tau$.
This is quite a strong condition, it says that the joint statistics don't change at all as time shifts.
For example, a 1st order stationary process is such that $F_{X(t_1)} = F_{X(t_2)}$ for all $t_1$ and $t_2$. That is, the $X(t)$ are all identically distributed. It is quite easy to see that a 1st order stationary process need not be 2nd order stationary. Simply assign a correlation structure to say $X(t)$, $X(t+1)$, $X(t+2)$ that does not correspond to a (symmetric) Toeplitz matrix. That is, in vector form, the covariance matrix of $[ X(t), X(t+1), X(t+2)]$ could be given as
$$\left[\begin{array}{cc}
\sigma^2 & a & b \newline
a & \sigma^2 & c \newline
b & c& \sigma^2
\end{array}\right]$$
for $a,b,c$ distinct. This is now not 2nd order stationary because $E[X(t)X(t+1)] = a$ and, time shifting by 1 we have $E[X(t+1)X(t+2)] = c \neq a$.
In a similar way (presumably), a process that is 1st and 2nd order stationary need not be 3rd order stationary and this leads to my question:
Does somebody have a nice example of a stochastic process that is both 1st and 2nd order stationary, but not 3rd order stationary?
Best Answer
Here is such an example. I will describe the process in terms of its sample paths.
It's a simple process, it only has four sample paths: $\omega_1$, $\omega_2$, $\omega_3$, $\omega_4$. This means that the only possible outcomes are:
$$ ...,\ X(-1) = \omega_i(-1),\ X(0) = \omega_i(0),\ X(1) = \omega_i(1),\ ... $$
One outcome for every $i \in \{1,2,3,4\}$.
The sample paths are defined as follows, for $t\in \mathbb{Z}$:
$$\omega_1(t):= \begin{cases} 1 & \text{if $t$ is a multiple of $4$,}\\ 0 & \text{else.}\\ \end{cases} $$
$$\omega_2(t):= \begin{cases} 0 & \text{if $t$ is a multiple of $4$,}\\ 1 & \text{else.}\\ \end{cases} $$
$$\omega_3(t):= \omega_1(t-2) $$
$$\omega_4(t):= \omega_2(t-2) $$
So they are:
$$ ...,1,0,0,0,1,0,0,0,1,0,0,0,1,0,0,0,... $$ $$ ...,0,1,1,1,0,1,1,1,0,1,1,1,0,1,1,1,... $$ $$ ...,0,0,1,0,0,0,1,0,0,0,1,0,0,0,1,0,... $$ $$ ...,1,1,0,1,1,1,0,1,1,1,0,1,1,1,0,1,... $$
Here is a plot of the four possible sample paths:
Each sample path is defined to have probability $1/4$. This concludes the definition of the process.
I will now show that it satisfies the conditions required by the OP.
For $t\in\{1,2,3,4\}$ one can see that there are exactly $2$ sample paths such that $X(t)=1$, and exactly $2$ sample paths such that $X(t)=0$. Hence for $t\in\{1,2,3,4\}$:
$$ P[X(t)=1] = P[X(t)=0] = 1/2 .$$
Since the process is periodic this is true for every $t\in \mathbb{Z}$. This means that the distribution of $X(t)$ does not depend on $t$ and therefore the process is 1st order stationary.
Likewise, looking at consecutive pairs, for $t\in\{1,2,3,4\}$ there is exactly $1$ sample path with $X(t)=1$ and $X(t+1)=0$, exactly $1$ sample path with $X(t)=0$ and $X(t+1)=1$, exactly $1$ sample path with $X(t)=0$ and $X(t+1)=0$, and exactly $1$ sample path with $X(t)=1$ and $X(t+1)=1$. So for $t\in\{1,2,3,4\}$:
$$ P[X(t)=1,X(t+1)=0] = P[X(t)=0,X(t+1)=1] =P[X(t)=0,X(t+1)=0] =P[X(t)=1,X(t+1)=1] = 1/4 .$$
Again by periodicity, this is true for every $t\in \mathbb{Z}$ and the process is 2nd order stationary.
However, for instance: $$P[X(0)=0,X(1)=0,X(2)=0] = 0,$$ while $$P[X(1)=0,X(2)=0,X(3)=0] = 1/4.$$
Therefore the process is not 3rd order stationary