Solved – Fast missing data imputation in R for big data that is more sophisticated than simply imputing the means

data-imputationlarge datar

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of course working fast, but more complicated packages which impute using regression or PCA take too long for a high number of missing values. I tried missMDA and missForest, but as I said, they look like taking forever. There is a package named FastImputation, but I could not figure out how to use it when I have no patterns from some training data. Any suggestions of packages which would impute fast?

Best Answer

I used mice (multiple imputation by chained equation). It's fairly fast, and quite simple. I used it on 3000 obs. for c.a. 10 variables. Done in 10min on an old computer. Further, I believe it is one of the best multiple-imputation packages out there. It can use regression to impute, among other methods.

You need to create a dataframe with the variable you want to impute, and include every variable that might predict values of that variable (so every var. in your model + possibly other var. as well). The mice package will impute every missing value in that dataframe.

Simplest way of imputing. Gives you a dataframe Datimp that has five imputed data + the original data. 

library(mice)
#m=5 number of multiple imputations
#maxit=10 number of iterations. 10-20 is sufficient.
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE) 
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
            sep="\t", dec=",", row.names=FALSE)

A better way to do this is: 

library(mice)
Dat1 <- subset(Dat, select=c(id, faculty, gender, age, job, salary)) #create subset
#of variables you would like to either impute or use as predictors for imputation.
ini <- mice(Dat1, maxit=0, pri=F)
pred <- ini$pred
    pred[,c("id", "faculty")] <- 0 #variables you do not want to use as predictors (but
    #want to have in the dataset, can't add them later.
    meth <- ini$meth
meth[c("id", "faculty", "gender", "age", "job")] <- "" #choose a prediction method
#for imputing your variables. Here I don't want these variables to be imputed, so I
#choose "" (empty, no mehod).
imp <- mice(Dat1, m=5, maxit=10, printFlag=TRUE, pred=pred, meth=meth, seed=2345) 
Datimp <- complete(imp, "long", include=TRUE)
write.table(Datimp, "C:/.../impute1.txt",
            sep="\t", dec=",", row.names=FALSE)

See if your imputations were any good:

library(lattice)
com <- complete(imp, "long", inc=T)
col <- rep(c("blue","red")[1+as.numeric(is.na(imp$salary))],6)
stripplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20,
xlab="Imputation number",cex=0.25) 
densityplot(salary~.imp, data=com, jit=TRUE, fac=0.8, col=col, pch=20,
xlab="Imputation number",cex=0.25) 

long <- complete(imp,"long")
levels(long$.imp) <- paste("Imputation",1:22)
    long <- cbind(long, salary.na=is.na(imp$data$salary))
densityplot(~salary|.imp, data=long, group=salary, plot.points=FALSE, ref=TRUE, 
xlab="Salary",scales=list(y=list(draw=F)),
par.settings=simpleTheme(col.line=rep(c("blue","red"))), auto.key =
list(columns=2,text=c("Observed","Imputed")))

Finally, and importantly. You can't just save your new dataset and use your imputed values as normal observed values. You use pooled regression or pooled lmer ...So the uncertainty of the imputed values is taken into account.

fit1 <- with(imp, lm(salary ~ gender, na.action=na.omit))
summary(est <- pool(fit1))
pool.r.squared(fit1,adjusted=FALSE)
Related Question