Solved – Missing data in repeated measures design

missing datamultiple-imputationrepeated measures

I have data from a a simple 2(voice system, manual system) x 2(easy, hard) within subjects experiment. Multiple DVs were collected.

Due to issues with both the systems tested, data for one system condition or another is corrupt (so badly that it can be considered missing) for many subjects (ss). Out of data for 34ss, only 14 have data for both system conditions. The remaining 20 have data for the voice system (9) or the manual system (11).

To list it out:

14 x data for both
11 x data for manual system condition, but missing voice
09 x data for voice system condition, but missing manual

I am considering my options:

I can use a repeated measures analysis with listwise deletion of missing data and run my analysis on only the 14 with full data.
I can use multiple imputation to attempt to fill in for 20 missing sets of data, however, that is a lot of data to impute. I'm wondering if that is wise.
I can redesign the experiment into a between/within ss experiment, randomly discarding data from the 14 who have both systems dara and arriving at a balanced 22n in each newly coined 'between' group. The difficulty group would still be analyzed as within.
I can begin collecting again. Sadly, in repairing the facility it was also 'upgraded', meaning enough changes that I'll have to start from scratch.

Any opinions on which is the best path forward? Any paths I have not considered?

Best Answer

With missing data, deciding what to do is (almost) all about deciding what assumptions you can validly make about the missingness.

The three main options are:

Missing Completely at Random (MCAR) - i.e. there are no possible predictors of missingness. In this case, you could validly drop those individuals with missing data and work with the complete observations only (this is called a complete case analysis). However, this is generally an unrealistic assumption in my experience. If you do a complete case analysis and the MCAR assumption does not hold, then any conclusions you want to draw from your results will only be valid for that specific set of people who have complete data. It is invalid to make inference about the whole population (or even whole sample) if MCAR does not hold, when using complete case analysis.
Missing at Random (MAR) - in this case, you know why some individuals have missing data and others do not (there are some variables which predict missingness) but within strata of those variables the data is truly randomly missing. In this case, you can use multiple imputation or other methods such as inverse probability weighting for non-missingness (IPW). Under MAR, these methods allow you to make inference about the entire sample, and therefore about the population from which your samples arose. You are right that you have a lot of data to impute, which may cause a problem. I don't have enough experience with multiple imputation to advise how best to proceed.
Not Missing at Random (NMAR) - there is a definite pattern to which data is missing and there are no variables which can be used to create strata within which the data is missing at random. If this is your situation, I believe you will need to redo the experiment as there is no valid solution I am aware of for analyzing the data as is (assuming that you want to make inference about some larger population).

To be honest, given the very high percentage of missing data, I would probably suggest starting over if that's not too costly in terms of time and money. I would try to do multiple imputation or IPW if you have variables to predict missingness, but you may not have much power with so few subjects and so much missingness. You will also need to be think about whether missingness on one measure affects missingness on the other measure - this would make the analysis more complicated.

Related Solutions

Missing Data Methods – Should Missing Values be Handled by Imputation or Deletion?

It depends on

Amount of missing data (what percentage of data is missing)
Type of missing data (MAR, MCAR, NMAR)

According to this nice article (Tsikriktsis: A review of techniques for treating missing data in OM survey research, 2005), if more than 10% data is missing, the best solution is

Maximum likelihood imputation if data are NMAR (non-missing at random)
Maximum likelihood and hot-deck if data are MAR (missing at random)
Pairwise deletion, hot-deck or regression if data are MCAR (missing completely at random)

Solved – Cronbach’s Alpha with missing data

If data are MCAR, one would like to find an unbiased estimated of alpha. This could possibly be done via multiple imputation or listwise deletion. However, the latter might lead to severe loss of data. A third way is something like pairwise deletion which is implemented via an na.rm option in alpha() of the ltm package and in cronbach.alpha() of the psych package.

At least IMHO, the former estimate of unstandardized alpha with missing data is biased (see below). This is due to the calculation of the total variance $\sigma^2_x$ via var(rowSums(dat, na.rm = TRUE)). If the data are centered around 0, positive and negative values cancel each other out in the calculation of rowSums. With missing data, this leads to a bias of rowSums towards 0 and therefore to an underestimation of $\sigma^2_x$ (and alpha, in turn). Contrarily, if the data are mostly positive (or negative), missings will lead to a bias of rowSums towards zero this time resulting in an overestimation of $\sigma^2_x$ (and alpha, in turn).

require("MASS"); require("ltm"); require("psych")
n <- 10000
it <- 20
V <- matrix(.4, ncol = it, nrow = it)
diag(V) <- 1
dat <- mvrnorm(n, rep(0, it), V)  # mean of 0!!!
p <- c(0, .1, .2, .3)
names(p) <- paste("% miss=", p, sep="")
cols <- c("alpha.ltm", "var.tot.ltm", "alpha.psych", "var.tot.psych")
names(cols) <- cols
res <- matrix(nrow = length(p), ncol = length(cols), dimnames = list(names(p), names(cols)))
for(i in 1:length(p)){
  m1 <- matrix(rbinom(n * it, 1, p[i]), nrow = n, ncol = it)
  dat1 <- dat
  dat1[m1 == 1] <- NA
  res[i, 1] <- cronbach.alpha(dat1, standardized = FALSE, na.rm = TRUE)$alpha
      res[i, 2] <- var(rowSums(dat1, na.rm = TRUE))
      res[i, 3] <- alpha(as.data.frame(dat1), na.rm = TRUE)$total[[1]]
  res[i, 4] <- sum(cov(dat1, use = "pairwise"))
}
round(res, 2)
##            alpha.ltm var.tot.ltm alpha.psych var.tot.psych
## % miss=0        0.93      168.35        0.93        168.35
## % miss=0.1      0.90      138.21        0.93        168.32
## % miss=0.2      0.86      110.34        0.93        167.88
## % miss=0.3      0.81       86.26        0.93        167.41
dat <- mvrnorm(n, rep(10, it), V)  # this time, mean of 10!!!
for(i in 1:length(p)){
  m1 <- matrix(rbinom(n * it, 1, p[i]), nrow = n, ncol = it)
  dat1 <- dat
  dat1[m1 == 1] <- NA
  res[i, 1] <- cronbach.alpha(dat1, standardized = FALSE, na.rm = TRUE)$alpha
      res[i, 2] <- var(rowSums(dat1, na.rm = TRUE))
      res[i, 3] <- alpha(as.data.frame(dat1), na.rm = TRUE)$total[[1]]
  res[i, 4] <- sum(cov(dat1, use = "pairwise"))
}
round(res, 2)
##            alpha.ltm var.tot.ltm alpha.psych var.tot.psych
## % miss=0        0.93      168.31        0.93        168.31
## % miss=0.1      0.99      316.27        0.93        168.60
## % miss=0.2      1.00      430.78        0.93        167.61
## % miss=0.3      1.01      511.30        0.93        167.43

Best Answer

Related Solutions

Missing Data Methods – Should Missing Values be Handled by Imputation or Deletion?

Solved – Cronbach’s Alpha with missing data

Related Question