Consider a prediction setting in which we are interested in training a regression or classification function $f$ with inputs $X \in \mathbb{R}^k$ and target $Y$, and assessing its expected generalization performance on new data. For example, in a regression setting, we might want to estimate what our model's mean absolute error (MAE) or root mean square error (RMSE) will be on unseen data, and we typically do that by evaluating our model's predictions on a held-out test set.
Many standard textbooks (for example, Chapter 7, Model Assessment and Selection, of The Elements of Statistical Learning) discuss data partitioning and cross-validation in a context where we are dealing with i.i.d. samples from a joint distribution $F\left(X, Y\right)$. In this case, constructing the test set is straightforward: we take a simple random sample and call it our test set.
Are there any standard references that discuss how model assessment strategies should be modified when data are not i.i.d.? I've seen discussions that are specific to particular settings (for example, Data partitioning for spatial data is about spatial data), but I was wondering whether there is a general reference covering multiple settings.
Examples include:
- Spatial data: the model may generalize more easily to points that are geographically close to the training set, and if we are interested in estimating the model's ability to generalize to points that are far from the training set, we'd need to account for that in our data partitioning / test set construction
- Data with a natural discrete group or hierarchical structure: for example, if we are dealing with patients and hospitals, the question "how does my model generalize to new patients in a hospital that was included in the training set" differs from "how does my model generalize to new hospitals", and we should construct our test set(s) accordingly; we may even want to answer both questions, which we could do with two different test sets (one on held-out patients and another on held-out hospitals)
- Similarly, if dealing with panel data (such as observing individuals over time), the question "How does my model do when predicting on an individual who was already observed $K$ times?" might have a different answer than "How does my model do on its first prediction for an individual who was never observed before?"
- Time series data: in some time series contexts (see https://stats.stackexchange.com/a/195438/9330 for an exception), we want to construct our test sets such that they cover a period that is entirely "in the future" relative to the training set
- More complex real-world examples might involve multiple "complications" relative to the simple i.i.d. case: we might be dealing with spatiotemporal data, for example, and it might also have a hierarchical or group structure
Does any textbook or paper discuss cross validation or test set construction in real-world problems, in a way that is general enough to cover all of the examples above (and possibly more)?
Related:
- How are cross validation and i.i.d. assumption of of a dataset related?
- Statistical learning when observations are not iid
- Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?
- Data partitioning for spatial data
- Cross-validation techniques for time series data
- Using k-fold cross-validation for time-series model selection
- Cross-validation in multi-level model
- https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data
- https://www.fast.ai/2017/11/13/validation-sets/ (thank you Bjorn for adding this in a comment)
Best Answer
It really all boils down to two rules of thumb:
Some relevant papers:
https://onlinelibrary.wiley.com/doi/abs/10.1111/ecog.02881
https://www.sciencedirect.com/science/article/pii/S0020025511006773
https://www.tandfonline.com/doi/full/10.1080/00949655.2020.1783262
You can also check out the Sperrorest R package for this.