Expected Value – Use in Machine Learning Models

machine learningpredictive-modelsrandom forestregressionshapley-value

I see that we have a concept called expected value being used in machine learning (ML) models. For example, SHAP has a concept called Expected value. It means when all input features are 0, we can consider model to output the expected value (baseline value).

Similarly, in linear regression, we have a term called intercept. Intercept also means the same, which is expected model output when all input features are zero.

While, I understand that in real world we may not encounter this scenario of all input features being zero.

Why do we consider this base performance? Is it because, we assume that all models will have some inherent power to predict output (on random). So, we have that base value?

Like a student who doesn't prepare for exams, can still get some marks (>0). Is that the same understanding here?

Best Answer

It is the simplest (or amongst the simplest) possible model(s). If your explanatory data (feature list) is $\mathcal D$ and the target variable is $Y$, what models usually try is predicting the conditional expectation, i.e. $\mathbb E[Y|\mathcal D]$. But, in the absence of data/features, you're left with unconditional expectation, i.e. $\mathbb E[Y]$. It is the simplest you can do without looking at the data/features.

Related Solutions

Solved – Bias parameter in machine learning linear regression

That seems like really confusing terminology, but what it means is, irrespective of the input $x$, the data will tend to be centered around $b$. If $x=0$ for all observations, the output of the regression would be $b$ in each case.

Bias here refers to a global offset not explained by the predictor variable. Consider the equation of a line:

$$ y = mx + c $$ Here $m$ is slope and $c$ is the intercept. If we omit the constant intercept $c$, $m$ as well as explaining the relationship between $x$ and $y$, must also account for the overall difference in scale irrespective of the value of $x$.

To demonstrate, if we have a really simple linear model in R with a constant difference between the variables (a difference in scale), then ignoring the intercept causes us to incorrectly estimate the relationship between $x$ and $y$ (the slope).

x <- rnorm(100)
y <- (3*x) + 100
lm(y ~ x)
#> 
#> Call:
#> lm(formula = y ~ x)
#> 
#> Coefficients:
#> (Intercept)            x  
#>         100            3
lm(y ~ 0 + x)
#> 
#> Call:
#> lm(formula = y ~ 0 + x)
#> 
#> Coefficients:
#>      x  
#> -5.505

Machine Learning – How to Improve Stacking and Meta-Learning Performance

There's a lot of useful material and practical recommendations for stacking from the Kaggle community. These include:

Often simple ensembling models (e.g. simple average of predicted probabilities, weighted average of predicted probabilities or logistic regression of logits - possibly with regularization towards a simple average) perform a lot better than trying to fit fancier models on the second level. This is especially the case when there's little data (esp. with time series data it's hard to know how much data 10,000 rows really is, because of the dependency of data over time and across different observations at the same time). XGBoost would probably not be my first thought for a stacking model, unless there's a lot of data.
Giving the ensembling model access to base features can be valuable when there is a lot of data and depending on some features the predictions of one model should be trusted more than the predictions of another model (of course, the latter is hard to know up-front). However, one often has to worry about this making it too easy to overfit and not adding value, so I would consider not doing that.
Ensembling models also need good hyperparameter choices and often need to be heavily regularized to avoid overfitting. How much so? Hard to say without a good validation set-up. If you do not have a good validation set-up for them, yet, this needs thought. A form of time-wise-splits like you seem to be using would be a sensible approach - e.g. you could use what you show in red as a the validation data on which you fit your ensembling models and then validate based on going even further into the future (perhaps you are already doing that?).
Have a look at the chapter on this in Kaggle GM Abishek Thakur's book (page 272 onwards) or - if you have easy access - there's excellent sections on ensembling and the validation schemes for it in the "How to Win a Data Science Competition" Coursera course, as well as what various Kagglers have written on ensembling (e.g. here or simply by looking at the forum discussions and winning solutions posted for a Kaggle competition that resembles your particular problem).

Why am I emphasizing Kaggle (and similar data science competition settings) so much? Firstly, because a lot of ensembling stacking gets done there. Secondly, because a lot of good incentives exist there to ensure that ensembling is not overfit and performs well on the unseen test data (while pracitioners sometimes fool themselves into believing in overfit results that are evaluated in an unreliable manner). Of course, there's also incentives to do things that would not work in practice, but might work well in the competition (like exploiting target leaks).

70-80% agreement between models that perform comparably (although a difference in accuracy of 65 vs. 70% seems large) actually sounds like a promising scenario for ensembling in the sense that this is on the low side for models trained on the same data. This reflects the reasonable diversity of models you chose (I'd expect a lot more similar results if you e.g. used XGBoost and LightGBM). Having models that are too similar in nature e.g. just multiple XGBoosts with slightly different hyperparameter values is usually much less valuable. Perhaps even more diversity could be achieved by having more models e.g. kNN classifier, logistic regression (both might require some good feature engineering) and depending on the details (that will determine whether there's any hope to do this - e.g. high cardinality categorical features, some inputs are text/images, being able to feed the time series nicely into a LSTM etc.) neural networks of some form (e.g. LSTM type).

Best Answer

Related Solutions

Solved – Bias parameter in machine learning linear regression

Machine Learning – How to Improve Stacking and Meta-Learning Performance

Related Question