Solved – PreProcess from Caret doesn’t work with a smaller dataset

caretdata preprocessing

I am trying to use Caret to train some prediction models. As part of this, I would like to use 'PreProcess'. I have, however, come to the conclusion that PreProcess requires a certain number of columns in the data set. Can this be true? If I take the first 179 columns of this dataset, “preProcess(dataset, method = c("center", "scale", "YeoJohnson"))” works without problems. If, however, I take only the first 178 columns, I get the error message:

Warning in pre_process_options(method, column_types) :
The following pre-processing methods were eliminated: 'center', 'scale', 'YeoJohnson'

Could you please explain this? And is it somehow possible to preprocess my data even if I have fewer columns?

I am using RStudio Version 0.99.491 with R version 3.2.3 on a PC with windows 7.

Thanks in advance for any help!

Best Answer

I had a similar problem before and discovered it related to my inadvertently using a matrix without columns names. preProcess calls the column names within the function so if these are null it will fail. If your subsetting method is losing this information this could be a cause of your error.

Related Solutions

Solved – Pre-processing (center, scale, impute) among training sets (different forms) and the test set – what is a good approach

If I understand your question, one of your ideas is to calculate a normalization (center and scale) across all of your data: both test and training.

Imagine that you take all of your data and calculate a centering and a scaling once, then use this on both the training set and test set. Your training set represents the data you have now, your test set represents future data that you don't have when you are training. But you somehow magically calculated centering and scaling values that included this future data. A Leak From The Future (tm), which is bad.

Solved – caret preProcess knnImpute error more nearest neighbours than there are points

The problem you run into is that knnImpute requires at least as many samples in your data without missing values as you have specified with the k parameter for the k-nearest-neighbours. As you use prodNA, you distribute NA randomly - which with a noNA=0.95 being pretty high, just very likely turns out to not have sufficient samples without NA values for your k:

table(apply(iris.mis, 1, function(r) all(!(is.na(r)))))
# FALSE 
#   150

What would work:

# slightly reduce amount of NA
iris.mis[, c(1,3)] <- prodNA(iris[, c(1,3)], 0.8)
table(apply(iris.mis, 1, function(r) all(!(is.na(r)))))
# FALSE  TRUE 
#   147     3 

# use at max the amount of samples without NA as k
myK = sum(apply(iris.mis, 1, function(r) all(!is.na(r))))

# impute as done in the question
preProcValues <- preProcess(iris.mis[, -4], method = c("knnImpute"), k = myK)
t_imp <- predict(preProcValues, iris.mis[, -4])

So, bottom line, knnImpute does work with many values missing - but if you want to use it with such few samples will depend on your problem and goal.

One more thing: keep in mind that if you would (be able to) use samples for imptutation that have certain features set to NA themselves this boils down to looking at samples in a subspace of your features. For example, in the extreme case of looking at and imputing 1 feature at a time only, you would not use any other information about a sample that has NA vaues to impute those. This would therefore not be classic imputation anymore which e.g. knnImpute is designed for.

Best Answer

Related Solutions

Solved – Pre-processing (center, scale, impute) among training sets (different forms) and the test set – what is a good approach

Solved – caret preProcess knnImpute error more nearest neighbours than there are points

Related Question