I have also been crawling these threads on this topic.
Step 1: Split the data into 80,000 training and 20,000 test sets.
Step 2: Using cross validation, train and evaluate the performance of
each model on the 80,000 training set (e.g. using 10 fold cv I would
be training on 72,000 and testing against 8,000 10 times).
Ok up to this point!
Step 3: Use the 20,000 test set to see how well the models generalize
to unseen data, and pick a winner (say ridge).
Either do this on a portion of the training set that was not used to tune parameters, or implement nested cross validation in your training set (e.g., use 3/4 of each fold to train and 1/4 to select among RF, logistic regression, etc).
Step 4: Go back to the 80,000 training data and use cross validation
to re-train the model and tune the ridge alpha level.
Step 5: Test the tuned model on the 20,000 test set.
This would not be a valid estimate of the error as you've already used this data to choose one of the three RF, LR, etc..
Step 6: Train tuned model on full dataset before putting into
production.
Tuning the model should be considered a step in the training process.
Say you have 2 models: RF with param NE = 100, 200; LR with param C = 0.1, 0.2.
You have 2 options (you can mix and match them as long as you adhere to the basic principle: if you use data to make a decision, don't use that same data to evaluate):
A
- Step 1. Split all data into
train_validate and test. Put test in a vault.
- Step 2. Split
train_validate into train and validate.
- Step 3. Train 2 RF on
train with param NE = 100 and 200. Train 2 LR on train with param C=0.1 and 0.2. Try all four models on the validate. Choose the model model_se with the smallest error. This is your "modeling process".
- Step 4. Unlock the vault and test
model_se (as is) on test to get some error. This error (one number) will be expected error on unseen data.
(It appears you have many observations. There is no hard rule for this that I know of, but if your classes are balanced A might be most reasonable).
B
Convert step 1 into an (outer) loop. If you use 7 fold you will have
7 train_validates and 7 tests.
Convert step 2 and 4 into an (inner) loop. If you use 5 fold you will
then 5 times create a train on which you will test the 4 models and
then 5 times see which is best on validate. Take the model
model_ba with best average performance over folds.
Test model_ba on the test set (in the outer fold) each time (each
one will be a different model). Since within each outer loop you
have an estimate of error, you will have 7 estimates errors. The
average of these errors is E and the variance V.
Rerun the modeling process Steps 2 and 3 from scratch on the entire
dataset. Eg, take 100% of the data and run step 2 to 4 (use the same
train:validate split ratios or 5 fold CV that you used there).
You will return some model M. You can expect performance E from
model M on unseen data. The variance V unfortunately
cannot be used to construct a 95% confidence interval (Bengio, 2004).
B is also known as 'nested cross validation,' but it is actually just plain cross validation of an entire modeling process (that involves tuning both parameters and hyperparameters or considering hyperparameters to be parameters and just tuning parameters (see here)). If you choose, B, it is worth running multiple iterations of B to see the variance of the entire process.
Other methods such as bootstrap may be preferable to cross validation. I have not had time to work out the details of why this is true.
Best Answer
You can use the defaults. Of course, it's possible to overfit to the test set by trying lots of different hyperparameter values and seeing what performance on the test set they lead to, but if all you're trying is two options, the default values and tuned values, and the default values themselves weren't set with the test set somehow, this kind of overfitting is not a substantial danger.
Note that when tuning hyperparameters worsens test-set performance but improves training-set performance, that's a hint that the tuning procedure is overfitting on the training set. Using regularization or switching to a simpler model may help.