Solved – Proving similarities of two time series

arimagranger-causalityrstatsmodelstime series

Let's assume an analytical model predicts an epidemic trend over time, i.e. number of infections over time. We also have a computer simulation results over time to verify the performance of the model. The goal is to prove the simulation results and predicted values of the analytical model (which are both a time series) are statistically close or similar. By similarity I mean the model predicts the values close to what simulation is providing.

Background:
Researching around this topic, I came across the following posts:

  1. https://stackoverflow.com/questions/13835924/similarity-of-trends-in-time-series-analysis

  2. How to statistically compare two time series?

Both discussions suggest three approaches, where I am interested in two of them basically:

(1). Use of ARIMA;
(2). Use of Granger test

For the first suggested solution, this is what has been written there in regards to ARIMA, in (1):

Run ARIMA on both data sets. (The basic idea here is to see if the same set of parameters (which make up the ARIMA model) can describe both your temp time series. If you run auto.arima() in forecast (R), then it will select the parameters p,d,q for your data, a great convenience.

I ran auto.arima on the simulation values and then ran forecast, here are the results:

ARIMA(2,0,0) with zero mean     

Coefficients:
         ar1      ar2
      1.4848  -0.5619
s.e.  0.1876   0.1873

sigma^2 estimated as 121434:  log likelihood=-110.64
AIC=227.27   AICc=229.46   BIC=229.4

I ran auto.arima on predicted model values and then forecast. This is the result of the predicted model:

ARIMA(2,0,0) with non-zero mean 

Coefficients:
         ar1      ar2  intercept
      1.5170  -0.7996  1478.8843
s.e.  0.1329   0.1412   290.4144

sigma^2 estimated as 85627:  log likelihood=-108.11
AIC=224.21   AICc=228.21   BIC=227.05

Question 1 What are the values that need to be compared to prove that the two series are similar especially the trend over time?

Regarding the second suggested option, I have read about it and found that Granger test is usually used to see if the values of series A at time t can predict the values of Series B at time t+1.

Question 2 Basically, in my case I want to compare the values of time series A and B at the same time, how this one is relevant to my case then?

Question 3 Is there any available method can be used to prove that the trend of two time-series over time is similar?

FYI. I saw another method which is using Pearson Correlation Coefficient and I could follow the reasoning there. Moreover, verifying analytical models with simulations has been widely used in the literature. see:

  1. Spatial-Temporal Modeling of Malware Propagation in Networks Modeling
  2. Modeling and Simulation Study of the Propagation and Defense of Internet Email Worm

Best Answer

Here's what I understand the situation is. You have one model, which you call your simulation, that you are confident generates a set of data that accurately represents what will actually happen in the epidemic. For some reason (presumably because it's expensive or slow to build and run, or there's theoretical interest in a simple equation that generates similar results to the complex model), you have an alternative model (the one you call a model) which can also generate a set of data, and you want to check if the version generated by this model is close to the version generated by the known-good model.

I'm also presuming that each time either of the models generates data, it generates a similar and pretty regular trend to other times. Otherwise (for example, if there's a random "take off" moment where the series suddenly breaks) there's another big complication.

First, the method of comparing parameters from an auto-fit ARIMA is a bad one (I'm guessing the reason that answer you linked to survived is that it is on Stack Overflow rather than Cross-Validated, where the statistical problems would have been picked up probably). The reason is that the same time series can get good fits with quite different combinations of auto-regressive and moving average values. There's no obvious way to look at the "similarity" of two different ARIMAs - ones that look very different may in fact be similar. As @IrishStat says in his answer to the second question you linked to, you could construct an F-test of a common set of parameters for both models, but that would require something quite a bit more complex than auto.arima(). And even then you might find that they don't have common parameters, but deliver similar predictions of the trend which is what you are actually interested in, rather than the details of the ARMA process that is generating some of the random noise around the trend.

So what would I recommend instead? It sounds like you aren't worried about the small fluctuations but only the overall trend. I would compare a smoothed version of the trend of each dataset, and start by making a visual comparison. In the case you've got, this shows that they're definitely not the same time series; one of them hovers around 1478, the other around zero, and that's good enough for me. But if there were some ambiguity, I would probably sum the squares or absolute values of the difference between the two smoothed series and determine if that was close enough, for some arbitrarily chosen meaning of "close enough" which in the end will have to depend on your domain, and the costs of being wrong. Definitely I'd start with the graphic.

If you want a more objective benchmark, I would try running both simulations multiple times and seeing how much difference (sum of squares or absolute differences) there is between different instances of the same simulation, and comparing that to the inter-simulation differences. If they're the same, that shows that you can't tell which model produced the simulation. If they're different, you still have to make a judgement call about how different is too much, but you'll have some numbers to help you.

enter image description here

While fitting ARIMA models is a bad idea for identifying similarity in trends, it's a good way to let me generate some data, so pasted below is how I did that. I'm guessing something's wrong with the data - maybe you fit the ARIMA model to a transformed or differenced version of the data, in which case you might want to go the next step of quantifying the difference between the two trends. 

library(forecast)
library(ggplot2)
library(tidyr)
library(dplyr)

# generate some data
good_model <- arima.sim(model = list(ar = c(1.4848, -0.5619)), n = 1000)
test_model <- arima.sim(model = list(ar = c(1.5170, -0.7996)), n = 1000) + 1478


combined <- data.frame(good = good_model, test = test_model, time = 1:1000)  %>%
 gather(variable, value, -time) %>%
 mutate(value = as.numeric(value))

 ggplot(combined, aes(x = time, colour = variable, y = value)) +
    geom_line(alpha = 0.5) +
    geom_smooth(se = FALSE, size = 2) +
    theme_minimal()

Edit

I blogged about this at http://ellisp.github.io/blog/2015/09/20/timeseries-differences , basically just exploring how you might use simulation brute force to determine if two models are similar. However, I reach the conclusion that you still need a (probably) subjective decision on a cost function - obviously you're two methods will be different, but how different are you prepared to put up with?

Related Question