Solved – Using information on both sides of a ‘gap’ in time series data for imputation

data-imputationforecastingrtime series

As with my previous question, I'm looking at ways to impute missing data in a hierarchical time series data.

With al my other procedures, including the experimentation of imputation packages (Amelia, HoltWinters from Forecast and MICE imputation) I've only been able to use the time series data prior to the missing gap.

     Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2001 220 194 238 190 217 244 242 225 242 259 267 244
2002 212 246 250 236 261 286 265 269 226 267 234 246
2003 202 199 297 272 236 266 235 226 260 183 226 265
2004 211 215 219 213 240 236 273 266 262 244 241 235
2005 212 198 233 251 259 282 305 267 241 264 222 269
2006 182 220 250 287 279 281 286 332 300 272 221 233
2007  NA  NA  NA  NA  NA  NA  NA  NA  NA  NA  NA  NA
2008 193 215 235 242 246 315 326 280 279 239 236 258
2009 246 189 257 241 268 223 260 288 234 260 216 195

I'm trying to do simple imputation procedure that uses forecasting and backcasting estimates from the time series model. Forecasting using prior data to predict the future and backcasting using the later data to “predict” the past.

I would then like to combine the forecast and backcast value to use as imputation. After which I will look at the fit etc.

How do I go about this in coding?

For example, I'm able to determine what SARIMA model exist for the first period 2001-end2006. But not the full period (because my basic functions I know from R does not support the NA values.)

This is only for the period 2001-end2006:

ARIMA(2,0,2)(1,0,1)[12] with non-zero mean 

Call: auto.arima(x = ts.datt) 

Coefficients:
         ar1      ar2      ma1     ma2    sar1     sma1  intercept
      1.3610  -0.8258  -1.2407  0.9191  0.8982  -0.7560   244.8374
s.e.  0.0884   0.0960   0.0878  0.1127  0.2190   0.3335     6.1894

sigma^2 estimated as 605.9:  log likelihood = -335.01
AIC = 686.02   AICc = 688.3   BIC = 704.23

Should I just model the first period, predict by forecast; model then the last period separately and then backcast? How will I do this backcasting (ie. 'predicting' the past)?

EDIT:
What I'm asking:
1) How do I use the data from years 2008 & 2009 to BACKCAST? I already know how to use 2001-2006 to forecast.

2) How do I determine the SARIMA model for the whole period? (2001-2009) ie.

Best Answer

Try using na.StructTS in the zoo package. It has methods for zoo and ts series. e.g. using the built in USAccDeaths insert some NAs and then interpolate them:

library(zoo)
window(USAccDeaths, 1975, c(1975, 12)) <- NA
na.StructTS(USAccDeaths)

See ?na.StructTS for more.

Related Solutions

Solved – Multiple imputation for missing count data in a time series from a panel study

You can use the Amelia package to impute the data (full disclosure: I am one of the authors of Amelia). The package vignette has an extended example of how to use it to impute missing data.

It seems as though you have units which are district-gender-ageGroup observed at the monthly level. First you create a factor variable for each type of unit (that is, one level for each district-gender-ageGroup). Let's call this group. Then, you would need a variable for time, which is probably the number of months since January 2003. Thus, this variable would be 13 in January of 2004. Call this variable time. Amelia will allow you to impute based on the time trends with the following commands:

library(Amelia)
a.out <- amelia(my.data, ts = "time", cs = "group", splinetime = 2, intercs = TRUE)

The ts and cs arguments simply denote the time and unit variables. The splinetime argument sets how flexible should time be used to impute the missing data. Here, a 2 means that the imputation will use a quadratic function of time, but higher values will be more flexible. The intercs argument here tells Amelia to use a separate time trend for each district-gender-ageGroup. This adds many parameters to the model, so if you run into trouble, you can set this to FALSE to try to debug.

In any event, this will get you imputations using the time information in your data. Since the missing data is bounded at zero, you can use the bounds argument to force imputations into those logical bounds.

EDIT: How to create group/time variables

The time variable might be the easiest to create, because you just need to count from 2002 (assuming that is the lowest year in your data):

my.data$time <- my.data$Month + 12 * (my.data$Year - 2002)

The group variable is slightly harder but a quick way to do it is using the paste command:

my.data$group <- with(my.data, 
                      as.factor(paste(District, Gender, AgeGroup, sep = ".")))

With these variables created, you want to remove the original variables from the imputation. To do that you can use the idvars argument:

a.out <- amelia(my.data, ts = "time", cs = "group", splinetime = 2, intercs = TRUE,
                idvars = c("District", "Gender", "Month", "Year", "AgeGroup"))

Solved – Imputation methods for time series data

Your approach sounds very theoretical. Did you analyze the imputations of the packages you mentioned?

Often imputation packages have requirements (e.g. MCAR data), but will still do a reasonable good job on data not fulfilling these conditions.

Only a actual test and comparison of algorithms will show you which one is best suited for your data.

The testing procedure can look like this:

Find a interval with no (or very few) missing data
Artificially add missing data in this interval. (these should resemble the NA patterns in the rest of the data)
Apply different imputation methods to this dataset. (e.g. methods from imputeTS, mtsdi, AMELIA)
Since you have the real values for your artificially deleted NA values, you can now compare how good alle the algorithms did on your data

Additional info:

The Amelia package also has some options to support the imputation of multivariate time series (see in the manual under 4.6)
Also other packages like mice could be tried

In general if you have multivariate time series, this means you have correlations between your different variables plus you have correlations of each variable in the time axis. (here is a talk from useR! 2017 conference which among other things explains this)

In theory it sounds like it would make most sense if you try to use both of the correlations. But if the correlations in time is for example very strong, univariate time series imputation methods from imputeTS might even work best.

On the other hand, if the correlation between your variables is very strong, non time series imputation packages could work best. (like mice, VIM, missMDA and others)

Best Answer

Related Solutions

Solved – Multiple imputation for missing count data in a time series from a panel study

Solved – Imputation methods for time series data

Related Question